Microsoft Principal Applied Research Scientist
Stanford University Ph. D., EES (now MS&E)
Current Time 0:00
/
Duration Time -:-
Progress: NaN%

How did you get to where you are today? What is your story? What incidents and experiences shaped your career path?

Summarized By: Jeff Musk on Thu Feb 06 2020
Let me kind of work backward. I've been here for five years with Microsoft, started a division of data scientists that were outward-facing, on one hand, to work with enterprise customers and get them up to speed on what we call the digital transformation. At the same time, we're working closely with our product group and as you may know if you're working with software in the field, the cloud product we have. It's called Azure. It has quite a wide range of machine learning tools and features that come along with it. That was interesting. We did quite a few projects with customers as a group of PhDs that I ran, maybe about half a dozen of varying degrees of success. That group has now been reorganized and the data scientists who were in that team have been spread widely through Microsoft. Let me go back a little bit before Microsoft. Before Microsoft, I had worked with Toyota in a user experience role where we were looking at user data in the cabin. What it was like for the driver and what we could do to change their experience. Then, before that, I worked quite a few years with Intel research. Going back to some of the early work that they did on analyzing data from fabrication plants. In the midst of those major roles, I have worked with several startups, which is typical of what happens in the Bay Area in Tech. My original degree was out of Stanford School of Engineering. At the time, we had a department called Engineering-Economic Systems. It's now part of management, science, and engineering. I think that perhaps gives a good sense of the academic area I came out of. At the time when I did my degree, we were starting work on the research program in Bayesian that was coming up and spinning up in the AI field, and I was part of that community and today continue to consider myself part of that community. It has somewhat been eclipsed by a lot of the recent advances in machine learning, but I think there's still some good lessons from that area of study that should not be forgotten. So that's kind of in a nutshell what my career path has been. I want to underline a few things. I can't say I've come up through a data science curriculum, right? When I came out of school and as we developed the analytics that I have progressively engaged with, Data Science was coming to life. Data science saw its first origins in I think in about 2006 when Patil at LinkedIn coined the term. I think the other side of it is that I think the preparation that we had in different analysis methods in the management science and engineering program and maybe it's worth going into some of that in detail about excellent preparation for what we're doing today. So I guess that's kind of a nutshell there. I think maybe worth delving into a few of the experiences along the way. 

What are the responsibilities and decisions that you handle at work? Discuss weekly hours you spend in the office, for work travel, and working from home.

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
I think it might be worth speaking broadly about Microsoft and picking up some of those items that I mentioned in the previous answer. As an applied research scientist, I think of it as a role that encompasses data science but as a broader reach and some broader aspects than what more people maybe think of today's conventional data science. And I think being on the cusp between the research community at the product teams is a very interesting place to be. And I think that is really what an applied scientist and a data scientist, as one of those kinds of applied scientists, find themselves in. Now, Microsoft has a very extensive research division of about 1000 people. It's perhaps if you put it alongside many university computer science programs, it's probably one of the top 10 in the country in terms of its productivity and the kind of work that's done there. The challenges of how do we make that relevant and bring it into use and see its effects on the day to day product is an important role that I see for an applied research scientist here at Microsoft and that again, there's a lot to say about that. It's quite an extensive discussion we have internally, and it's also an evolving discussion. I think for myself coming out of a research background originally going back to the work when I was part of the research at Intel research, gives me some good insights and some good road feel for what it takes to do that kind of work. So to go down to responsibilities and decisions. Well, I've done two things here. So in the previous position here at Microsoft, we realized that positions are pretty fluid. They turn over regularly. This kind of tech has quite a bit of evolution in the organization. This organization tends to move quickly and reform as necessary. The original works as a researcher, as a data science manager, I had a team of five PhDs. Our job was to go into a situation internally or with an external client, and we'd be the first person on the scene in terms of analysis. We'd sit down with them at a whiteboard and say, "What's your problem? What are the issues? What's relevant to stuff we could do with our toolset." And we thought that widely, we didn't just think of your conventional machine learning tools. We thought of it as using all the possible analytic toolset that we could apply probably to a problem either from our proprietary products set and actually even from open source. We'll be the people who would design the original approach to the problem. I think that's a really interesting area. Now I'm doing something similar, but I'm entirely in a product group. So I had the same ability to initiate work on a problem to define a problem, figure out how it can be approached if it's worth approaching. And, convert what maybe started off as w somewhat poorly defined set of issues has something that we can frame as a problem with a solution and figure out within the resources and data that we have, whether it is a feasible problem to solve. My hours are quite flexible. In fact, I have nobody who pays much attention or gives much regard as long as I am speaking to the right people at the right times in the meetings that I've agreed to come to either by telecommuting or by being here in person or by traveling to one of our offices, Redmond being very commonly the destination. Since that's what we call our mother ship, where most of the work goes on. I have pretty much responsibility to work where I can and when I can. And there's a responsibility about what's the joke about 90% of life is actually just being there is actually being present, and it turns out that if you think of what that freedom might entail, I'm often here close to eight in the morning and I'll work. I'll try to get home for dinners may be the way to say it, and often part of my weekends is involved with reading papers or reviewing papers or tidying up some e-mail or software that I've been working on. I say I'm fascinated by the work. I'm motivated strongly by it, and, it inter-leaves with my life. Both in the office. I have a good work-life balance in the office. I like the people I'm working with. We go out, we have fun. And also out of the office, where often bring a lot of it with me and flavors and colors. Some of my outside activities, such as the local meetups I go to in the field, and, actually, things like this.

What tools (software programs, frameworks, models, algorithms, languages) do you use at work? Do you prefer certain tools more than the others? Why?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
I think Microsoft has one of the best R offerings is in the cloud. I work closely with a group of folks who came from a startup called Revolution Analytics that brought in our product here into Microsoft. A lot of my work right now is in Python. I think the main interesting point here is working in a product group, the tools we used in data science are actually somewhat distinct from the tool train that a software engineer uses. And one should not confuse the fact that a data scientist such as myself or the folks I've worked with and over for maybe quite proficient in the software, in getting things done, in terms of solving the problems in software being self-sufficient to build and run and generate experiments. But that's a different task than running a product, working with the repositories of the actual code that is in the product and the toolsets that are there. And, I'm very cognizant of what a software engineer does, but I would never want to present myself fork or claim to be that particular set of talents that a software engineer brings. I think there's a vice versa there also. So, yes, much of the work I do is writing software. It tends to be often taking a certain mathematical approach is and customizing them to the problem at hand may be different than in some other areas of data science, which are much more an application of existing software tools. But I think it is clear. And I think, it's really necessary for me to work side by side with the actual engineers who are using the toolchains and that test procedure and the deployment procedures that are part of the actual product development rollout.

What things do you like about your job? Were there any pleasant surprises?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
Oh, I like my job. To be honest, I think everybody has to realize in life there are ups and downs so I can't really put just a rosy picture on and say it's all been a walk in a rose garden. What's particularly pleasant about this is that it's high energy people. I'm working with their high energy of this, quite a diversity in both age backgrounds, talents in the group. It's a very open, collaborative environment. I have a lot of freedom to approach my work schedule and my problems in the way I see fit. I think that if I can put a pleasant surprise in a single phrase, it would be that I didn't expect Microsoft to be as collaborative as it is compared to, for example, some of the other companies I've worked with and that's nice. That's very nice.

What are the job titles of people you routinely work with inside and outside of your organization? What approaches do you find to be effective in working with them?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
I've never been very attentive to people's titles. To be honest, I think in terms of people, skills and backgrounds and areas, maybe that's a more perceptive way to address this. But I'm working inside the organization right now. I think if you think about Microsoft, it may sound obvious, but this is a software company. Of course, there's hardware. We have a substantial role in that. But the parts I know are software and it's very mature and it's a very aggressive software company in the sense of having quite a good space in the market and pushing on it hard. So the vast part of the organization that I see is the software development and engineering pieces from the bench level engineers, interns that I work with, recent college graduates all way up the chain to the VPs. I have visibility up to maybe the first or second VP level in terms of my work and what I'm commonly able to perceive the organization. When I'm working with people outside. I'm not working very much in this role with the third parties. When we worked with enterprise customers, I think there was, on the one hand, a good interface when we would meet with VP level folks in the enterprises who often were the people who owned the problems that we were working on. I think that it was really important to have that dialogue. As the work evolved, I think the best part, and I'm answering more the question of, how do you be effective in that role? I think the best part was when we met a group in enterprise partners where they had the sophistication in terms of understanding data science and having some data scientists on board.so we can talk a data scientist to data scientist or think of it more is working at the same level of mathematical maturity that we brought to the problem on their side. I think then one could go into a large discussion about how do you successfully run an analysis project, actually bring it into deployment, and what are the pitfalls there? But that's a long discussion. That would be another long discussion.

What major challenges do you face in your job and how do you handle them? Can you discuss a few accomplishments?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
I'm going to frame this as the role of a senior data scientist in Microsoft. And maybe this is specific. Maybe there's a lesson learned here. Maybe it says something about the field where it's going right now. Data science is still even at Microsoft, not pervasive, and it's still under evolution. And how it fits into more traditional groups is still very much in the area of intense scrutiny and study. And so I think the interesting thing here is there's a certain amount of community and camaraderie among people who both have come in as Data Scientists and similarly, people who have played a significant research role before now find themselves in more applied jobs. And I would frame that in terms of the larger question of How do you take the knowledge or the expertise that comes out of researcher, or the particular contribution that a research person offers and make something useful out of it in the terms of direct deadlines that people work within a product group. So I think our accomplishments here are to see, maybe this is speaking very, very broadly is to see this pervasive influence of data science and to see the level of analysis, see people looking at problems in more systematic ways throughout the product groups and seeing some boats being seen the kind of the level of all boats being raised somewhat in their specific accomplishments. Well, I am very proud of a few third parties, a few companies where we've worked with and we come back after a year. They started often with very scant ideas of what they could do with machine learning. And now they're running applications at scale, in forecasting, in pricing and similar areas in their business. That's a very rewarding feeling to be an enabler and educator and contributor terms of having often worked side by side, which we would speak in those cases of developed with rather than developed for. The accomplishments in the product groups? it's a little early to tell at this point, but I think there's a very strong sense of optimism in the groups about the innovation of better integration of data science.

What are the recent developments in the field? How significant are these improvements over past work? What are their implications for future research & industry applications, if any?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
Data science has evolved quite a bit. I think if you look at the past five years, how people use the term and I think the term is in need of some discussion. What actually constitutes data science maybe is also a good question there. I would have seen recently that the field is a kind of at a higher level of integration with software approaches. We have a product called Auto ML. The idea is you write a few lines of code and you leave the various machine learning steps that you might previously have had to manually go through in terms of featurization, selection of a model, designing a test sequence to choose the model, figuring out how to put the model in a deployable form, it's now kind of a push-button thing. Now, I'm not sure it's there, and I wouldn't recommend it in all cases, but the field has been moving in that direction to integrate forward and to essentially apply automation to automation. To make the task of building automation more automated. A part of that is trying to bridge the gap between software development techniques, Dev-ops and find out what Dev-ops for a machine learning would be. And there's progress in that. What else is going on? Well, what's in the research world right now? What's exciting is the recent work in reinforcement learning where you are and that covers a huge span of possible things. But in short, it causes the analyst to look at the problem not only from the sense of the kind of conventional, how predictive is my model, but also in what is the actual value function? What are the rewards? What is the value function that often brings you into a better insight into the business value that you're bringing because it's now modeled? It's part of the model. Reinforcement learning also brings in a dynamic aspect. You have state, and you have a model that can be evolving forward, in both senses, both in terms of carrying state that's relevant to your current decision, but also in terms of working for problems that are non-stationary. Stationarity means, and this is true in most business problems, that the world is changing at the same time that you're modeling it. And when you release a model, you have to be cognizant that it's gonna be used in a different circumstance. The probability distributions, the variables are likely to be evolving underneath you as you deploy the model. Reinforcement Learning has the benefit of approaches to handling that and also the other good thing about reinforcement learning and this maybe gets into some interesting technical errors that I think are really exciting is you have an experimentation aspect called exploration. So the model is actually going out and running a kind of efficient test sequence. If it's done right on the world. Well, if you're testing, then you're actually able to explore certain causal aspects that are not evident if you're doing a statistical approach on existing data. That's a very interesting area, and again, we'll probably have much more involved technical discussion. But the work currently being done in what does it mean to find causal conclusions from a statistical model? And can we do that to some degree without either observational or with approaches like reinforcement learning, I think, is a really, really fascinating area. I would put Active Learning under Reinforcement Learning and I think active learning, on one hand, I think it's underutilized. Our experiments with it have been very encouraging. It is interesting because on the one hand, it kind of breaks a bunch of things that you would expect from normal supervised learning. And that often you're learning curves don't evolve monotonically in the same way. It's hard to figure out sometimes what's going on. There isn't in my mind an elegant or final theory of active learning. I see its umbrella for a range of different approaches. I think the combination of active learning in terms of using an incremental approach, an online approach where you use the online aspect to make decisions about what data to collect, together with the idea of semi-supervised learning where essentially you're looking at, "Can I learn something? ". For example, in a neighborhood from unlabeled points around the points that have been labeled by making some perhaps smoothness assumptions, I think those go very nicely together. I think it's really a good point, and I think I'm glad you brought it up that there are these frontiers and they do address things that are very relevant to the evolution of the field. Supervised learning has always had the labeling bottleneck and what we can do with active learning, more efficient experiments and making assumptions that let us extract information from unlabeled points in addition to a small label that these were all I think, areas which deserved to be pushed on. No, I'm gonna pivot. I will not give you another opinion, which probably you are going to bring up, which is, what about deep learning? I have to say, I don't play very much in that area. For a couple of reasons, one is that it's not applicable to many of the areas where I'm working, where I'm working with either small data sets such as time series and small here, maybe in only a few 100 megabytes, right? I'm working with small data by deep learning standards and also most of the techniques and tools that I've learned coming out of the base nets. The basing statistics, probability modeling, dynamics, and value-driven kinds of models that optimize against a particular value function are really a different sub-discipline, then deep learning. Now deep learning, of course, should not be discounted for what it's been able to do what I call perceptive problems comes where you have a data feed. That is very much like an animal's sensory or perceptive source. But let us not take that and assume that it's going to replace the areas that you've mentioned that I mentioned that I think are in danger of being overshadowed.

What was the hiring process like for your job? What were the roles of people who interviewed you? What kind of questions were asked?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
There's now become kind of a standard data science interview format, which starts with a technical interview and often includes a problem, often from statistics and often an exercise in starting from a poorly framed problem to see how a person could frame it better. I do that kind of interview. I think it should be distinguished from a pure computer science interview, and I think we make that distinction clear when we do interview people for Data Science.

What qualities does your team look for while hiring? How does your team interview candidates?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
On the other hand, I expect a person to have a reasonable facility, not necessarily a deep mathematical ability. I'm not looking for a person to prove theorems in abstract algebra or in stochastic processes, but they should have on their fingertips a very good understanding of the use of probability working in high dimensional spaces how you capture or do things like an SPD decomposition or Eigenvalue decomposition or what it means to have Eigenvalues or when different agencies have different Eigenvalues? Why that's relevant to solving a lot of problems in high dimensions. Microsoft to pivot does have a particular interview process. And I think for people who are coming out of the University, it's important for them to explore that and perhaps way beyond what I can do in an interview here to both talk to recruiters who will be very open about what they're expecting in the interview process and talk with recent hires. But typically, we have a full day after we've done initial qualifications. After the person had a phone interview and reached a first-level technical qualification, we will bring a person in for 4 to 6 interviews of people. Some will be purely technical. Some will be, for example, with another data scientist. Another interview would typically be to understand, your particular contribution. Do you understand something about that domain? How do you fit in? And then there's always an interview of how do you fit in the group? There's a clear understanding that interviews have to be on the level that we don't discriminate and that we not only follow the law, but we have a clear understanding of the value of diversity. Both from people's backgrounds and also intellectual diversity. I think that's really important. In my particular sense, I would ask three questions that need to be covered in the interview process. One is, does the person have the talent? Can they solve the problems that we're approaching? But that's only one of the three. The other is do they want to work? What's their motivation? Are they strongly motivated to do this? And I think that could come across clearly positively or negatively, even if they are qualified. It's not a question of giving a person an award when you give him a job, and then the last one, I think, is really important. It is can they work with the team? There's a very strong emphasis here in your annual evaluation of your ability to contribute to a team in both ways. In both do you contribute to others? Do you let others take credit for things that you have done that helped them out, and vice versa. Do you also borrow from other people? Do you know how to exploit the talents of others to the benefit of the larger project in a way where everybody feels that they've been rewarded? So I think my experience is interesting, I think, coming into hiring and by virtue of having interviewed with many startups and actually I would admit in Silicon Valley you're almost always in that feeling, for one reason or another. I think a company's interview style, how they approach prospects often is very revealing about the company and its culture as a whole, and I think for a company that is very well organized and can interview well and puts the interviewer in the best light speaks very strongly about the company.

What are different entry-level jobs and subsequent job pathways that can lead students to a position such as yours?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
Remember I didn't come through a conventional hire through the company's career path and I have to say that data science, it's hard to say exactly the career path is still not so clear. I think, practically if a person is interested in this kind of work or growing in this kind of work and Microsoft, we bring in about 6000 people a year as recent college grads and again realizing this is a company that's largely building software or building hardware. It's building software and hardware that a good, solid familiarity with computer science and software. Even if a person is not a computer science grad, which I don't think we need the majority of our hires to be, just realize that's the nature of the company. If you're interested in working here. Like many high tech companies, we recruit widely. A lot of our hires are recent college grads, done through the internship pipeline that we, recruit strongly for. If someone is interested in joining Microsoft, I think the first thing they really have to do is while they're in college, look at the internship opportunities during the summer, or possibly even during the year. Those are available. When we bring in an intern. It's a pretty in-depth thing. We put you right in the middle of work. We typically get some really, really amazing results from our interns. And when we find an intern who is a good match on and has a successful internship, we will often let them walk out of the building after the internship with a perspective offer in their hands. So, the practicality of getting hired as a recent college graduate, often goes back to your experience, through an internship program. For mid-level applications, it's important to be part of a professional community and again, just the practicality is it knowing people in the company, I laugh because if you think back to my experience, just having lunch with somebody, and the discussion was around LinkedIn. Well, LinkedIn is actually part of Microsoft. But I don't think there's a job I've gotten in the last 10 years where LinkedIn was not a crucial part of the process of both discovery and it's exploring the possible people to talk to and actually on the mechanics of the hire. I think in high tech that's now much the case.