Microsoft Principal Applied Research Scientist
Stanford University Ph. D., EES (now MS&E)
Current Time 0:00
/
Duration Time -:-
Progress: NaN%

How did you get to where you are today? What is your story? What incidents and experiences shaped your career path?

Summarized By: Jeff Musk on Thu Feb 06 2020
enough material there, I think to fill a full hour so they see what I can do that is targeted toward our topic of data science and and, uh ah, career as it now stands, uh, let me kind of work backwards. I've been here at Microsoft for five years with Microsoft, started a division of data scientists that were outward facing on the one hand toe work with enterprise customers and, uh, get them up to speed on what we call the digital transformation. At the same time, we working closely with our product group. And as you may know, if you were working with software in the field, the cloud product we have a sewer has quite a wide range of machine learning tools and ah, features that come along with it. Um, that was an interesting we did quite a few projects with customers as a group of PhDs that Iran maybe about half a dozen of varying degrees of success. Ah, that group has now been, um, reorganized, and the data scientists were in that team have been spread widely through Microsoft. Let me go back a little bit. Ah, Before Microsoft for Microsoft, I had worked with, um, Toyota in a user experience role where we were looking at user data in the cabin. Know what what it was like, uh, for the driver and what we could do to change their experience. Then, before that, I worked quite a few years with Intel research, Um, going back to some of the early work that they did on analyzing data from fabrication plants. Um, in the midst of those major roles, I have worked with several startups, which is typical of what happens in the Bay Area in Tech. My original degree was out of Stanford School of Engineering. At the time, we had a department called Engineering Economic Systems. It's now part of management, science and engineering. I think that perhaps gives a good sense of the academic area I came out of at the time when I did my degree, we were starting work of the research program in Beijing and networks. Was was coming up with spinning up in the eye field, and I was part of that community and today continue to consider myself part of that community. Ah, it has somewhat been eclipsed by ah lot of the recent advances in machine learning, but I think there's still some good lessons there from that area of study that should not be forgotten. Um, so that's kind of in a nut show what my career path has been, I think just him maybe underline a few things I never I can't say. I've take come up through a data science curriculum, right? When I came out of school and as we developed the analytics, uh, that I have progressively engaged with, um, data science was was coming to life. Data science was was, um uh saw its first origins, I think about 2006 when, uh, Potala linked in coined the term. Um, I think the other side of it is that I think the preparation that we had in different analysis methods in the er management, science and engineering program and maybe it's worth going into some of that in detail were an excellent Annette excellent preparation for what we're doing today. So I guess that's kind of a nut shell there. Uh, I think maybe worth delving into a few of the experiences along the way, but, um, why don't, uh, when we go on from there

What are the responsibilities and decisions that you handle at work? Discuss weekly hours you spend in the office, for work travel, and working from home.

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
I think, uh, it might be worth speaking broadly about Microsoft and picking up some of those, um, items that I mentioned previously in the previous. The answer? Um, yeah, as they applied research scientist. I think of it as a role that encompasses data science but as a broader, uh, reach and some broader of aspects, then what more people maybe think of today's conventional data science. And I think being on the cusp between the research community at the product teams is a very interesting place to be. And I think that is really what an applied scientist and a data scientist, as one of those kinds of applied scientists finds themselves in. Now, Microsoft has a very extensive research, a division of about over 1000 people. It's perhaps, if you put it alongside many university computer science programs, it's probably one of the top 10 in the country in terms of its productivity and the kind of work that's done there. The challenges of how do we make that relevant and bring it into use and see its effects on the day to day product is a an important role that I see for an applied research scientist here at Microsoft and that again, there's a lot to say about that. It's quite a extensive discussion we have internally, and it's also an evolving discussion. I think for myself coming out of a research background originally going back to the work when I was part of research and Intel research gives me some good insights and some good, um, Road feel for what it takes to do that kind of work. So to go down to responsibilities and decisions. Well, I've done two things here. So in the previous position here at Microsoft, he realized that positions are pretty fluid. They turn over wreck regularly. This, uh, kind of tech has quite a bit of, uh, evolution in the organization. This organization tends to move quickly and reform as necessary. The original works as a researcher as, ah, data science manager. I had a team of five PhDs. Our job was to go into a situation internally or with an external client, and we'd be the first person on the scene in terms of analysis. We'd sit down with them at a white board and say, What's your problem? What are the issues? What's relevant to stuff we could do with our tool set. And we thought that widely, we didn't just think of your conventional machine learning tools. We thought of it as using all the possible analytic tool set. Uh ah. Sets that we could apply probably to a problem either from our proprietary products said. And actually even from open source and we be the people who would design the original approach to the problem. I think that's a really interesting area. Now I'm doing something similar, but I'm entirely in a product group. So I had the same ability to initiate Ah, work on a problem to define a problem. Figure out, uh, how it can be approached if it can be approached if it's worth approaching. And, um uh, ah uh, convert What would have been for convert? What maybe started office? A somewhat poorly defined set of issues has something that we can frame as a problem with a solution and figure out within the resource is and ah, data that we have, whether it is a feasible problem to solve. Um, my hours air quite flexible. In fact, I have nobody who uh pays much attention or gives much regard as long as I am, um, speaking to the right people at the right times in the meetings that I've agreed to come to either buy telecommuting or by being here in person or by traveling to one of our offices, Redman, to being very commonly the destination. Since that's what we call our mother ship, where most of the work goes on. I have, ah, pretty much responsibility to work where I can when I can. And there's a responsibility there of being ah, um, what's the joke? About 90% of life is actually just being there is actually being present, and it turns out that if you think of what that freedom might entail, um, I'm often here close to eight in the morning and ah, I'll work. Oh, I'll try to get home for dinners may be the way to say it, and often part of my weekends is involved with reading papers or reviewing papers or tidying up some Ah, um e e mail or software, um, that I've been working on. So I say I'm fascinated by the work. I'm motivated strongly by it, and, uh, it inter leaves with my life. Um both in the office. You know, I have, ah, think a good work life balance in the office. Oh, I like the people I'm working with. We go out, we have fun. And also out of the office, where often bring a lot of it with me and, ah, flavors and colors. Some of my outside activities, such as the local meet ups I go to in the field, and, Ah, actually, things like this.

What tools (software programs, frameworks, models, algorithms, languages) do you use at work? Do you prefer certain tools more than the others? Why?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
um, I think Microsoft has one of the best are offering is in the cloud. I work closely with a group of folks who came from a startup called Revolution Analytics that brought in our product here into Microsoft. Ah, lot of my work right now is in Python. I think the main interesting point here is working in a product group. The tools he used in data science are actually somewhat distinct from the tool train that a software engineer uses. And one should not confuse the fact that a data scientist such as myself or the folks I've worked with and over for me maybe quite proficient in the software in getting things done in terms of solving the problems in software being self sufficient to, um, build and run and generate experiments. But that's a different task vent running a product, working with the repositories, ah, of the actual code that is in the product and the tool sets that are there. And, um, I'm very cognizant of what a software engineer does, but I would never want to, um ah present myself fork or claim to be that particular set of talents that a A software engineer brings. I think there's a vice versa there also. So, yes. Um, much of the work I do is writing software. It tends to be often taking what's certain. Um, mathematical approach is ah. And, uh, customizing them to the problem at hand may be different than in some other areas of data science, which are much more an application of existing software tools. Um, but, uh, I think it is clear. And I think, um, it's really necessary for me to work side by side with the actual engineers who are using the tool chains and that test procedures and the deployment procedures that are part of the actual product development rollout.

What things do you like about your job? Were there any pleasant surprises?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
Oh, I like my job. Um and you know, to be honest, I think everybody has to realize in life there are ups and downs s so I can't really put, you know, just a rosy picture on and say it's all been, um, a walk in a rose garden. Um, what? What's particularly pleasant about this is that it's high energy people. I'm working with their high energy of this, quite a diversity in both age backgrounds, talents in the group. It's a very open, collaborative environment. I have a lot of freedom to approach my work schedule and my problems in the way I see fit. Ah, I think that if I can put a pleasant surprise in a single phrase, it's I didn't expect Microsoft to be as collaborative as it is compared to, for example, some of the other cos I've worked with, um and that's nice. That's very nice.

What are the job titles of people you routinely work with inside and outside of your organization? What approaches do you find to be effective in working with them?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
I've never been very attentive to people's titles. To be honest, I think in terms of people, skills and backgrounds in areas, maybe that's, ah more perceptive way to address this. But I'm working inside the organization right now. I think if you think about Microsoft, it may sound obvious, but this is a software company. Software. Of course, there's hardware. We have a substantial role in that. But the parts I know our software and it's a very mature and it's a very what should I say? Aggressive software company in the sense of having quite a good space in the market and pushing on it hard. Um, so the vast part of the organization that I see are the software, Um uh, development and engineering pieces from the bench level engineers, interns that I work with recent college graduates all way up the chain to the V. P. S. I have visibility up to Oh, maybe, uh, the first or second VP level in terms of my work and, uh, what I, uh ah, commonly are able to perceive about of the organization when I'm working with people outside. I'm not working very much in this role with dirt as we call them the third parties. When we worked with enterprise customers, I think there was on the one hand, um ah, good, Um, interface. When we would meet with VP level folks in the enterprises who often were the people who owned the problems that we were working on. I think that was really important to have that dialogue as the work evolved. I think the best part, and I'm I'm answering more. The question of, um, how do you be effective in that role? I think the best part was when we met a group in Enterprise Partner where they had the sophistication in terms of understanding data science and having some data signs on board So we can talk about a scientist to data scientist or think of it more is working at the same level of mathematical maturity that we brought to the problem on their side. Um, I think then one could go into a large discussion about how do you actually successfully, um, run a analysis project, actually bring it into deployment, and what are the pitfalls there? But that's a long discussion. That would be another long discussion

What major challenges do you face in your job and how do you handle them? Can you discuss a few accomplishments?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
challenges. And again here I'm going to frame this as, um, the role of a senior down a scientist in Microsoft. And maybe this is specific. And but maybe there's a lesson learned here. Maybe it says something about the field where it's going right now. Data science is still even at Microsoft, not pervasive, and it's still under evolution. And how it fits into more traditional groups is still very much, um, I, uh, on area of, I think, intense scrutiny and study. And and so I think the interesting thing here is there's a certain amount of, um, community and camaraderie among people who, um, both have come in his data. Scientists and similarly, people who have played a significant research roll before now find themselves in more applied jobs. And I would frame that in terms of the larger question of again. How do you take the knowledge or the expertise that comes out of researcher, or the the particular contribution that a research person offers and make something useful out of it in the, um, in the term in the at the, um within the I could put its directly deadlines that people work with in a product group. Um so I think our accomplishments here O r to see on maybe this is speaking very, very broadly is to see this pervasive influence of data science, but and to see the level of analysis, see people looking at problems in more systematic ways throughout the product groups and seeing some boats being seen the kind of the level of all boats being raised somewhat in there specific accomplishments. Well, I am very proud of a few third party of you cos where we've worked with and we come back after a year. They started often with a very scant ideas of what they could do with machine learning. And now they're running, um um applications at scale, in forecasting, in pricing and similar areas in their business. That's a very rewarding feeling to be a nen, abler and educator and ah ah, contributor terms of having often work side by side, which we would speak in those cases of developed with rather than developed four. The accomplishments in the product groups. It's a little early to tell at this point, but I I think there's a very strong sense of optimism in the group's about the innovation of better integration of data science

What are the recent developments in the field? How significant are these improvements over past work? What are their implications for future research & industry applications, if any?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
That's a great That's a great question. That's really good. Um, data science has evolved quite a bit. I think if you look at the past five years or what, people, how people use the term and I think the term has, uh is in need of some, uh, this discussion. Oh, um, what actually constitutes data science maybe is it is also a good question there I would have seen recently. The field is a kind of ah, higher level of integration with software approaches. We have a product called Auto ML. The idea is you right, a few lines of code and you leave the various machine learning steps then you might previously have had to manually go through in terms of computerization. Um, selection of a model. Um uh, designing a test sequence. Two. Choose the model. Figuring out how to put the model in a deployable form, it's now kind of a push button thing. Now, I'm not sure it's there, and I wouldn't recommend it in all cases, but the field has been moving in that direction. To integrate forward and to essentially apply automation to automation, make the task of building automation more automated part of that is trying to bridge the gap between software development techniques, Dev ops and find out what Dev ups for a machine learning would be. And there's progress in that, Uh, what else is going on? Well, what's in the research world right now? That's for me. Exciting is the recent work in reinforcement learning where you Are, and that covers a huge span of possible things. But in short, it causes theano list. Look at the problem not only from the sense of the kind of conventional how predictive is my model, but who brings I also in what is the actual value function? What are the rewards? What is the value function that often brings you into a better insight into the business value that you're bringing because it's now modeled? It's part of the model reinforcement. Learning all also brings in a dynamic aspect. You have state, and you have a model that can be evolving forward, um, in both senses, both in terms of carrying state that's relevant to the your current decision, but also in terms of working for problems that are non stationary stationery. He means that and this is true in most business problems that the world is changing at the same time that you're modeling it. And when you release a model, you have to be cognizant that it's gonna be used in a different circumstance. The the, um, probability distributions. The variables are likely to be evolving. Um, underneath you as you deploy the model reinforcement. Learning has the benefit of approaches to handling that, um, and also the other cute thing about reinforcement learning And this may be gets into some interesting technical errors that I think are really exciting is you have an experimentation, um, aspect called exploration. So the model is actually going out and running. Ah, kind of efficient test sequence. If it's done right on the world. Well, if you're testing, then you're actually ableto too. Ah, explorers certain causal aspects that are not evident if you're simply doing simply, if you're doing a a statistical approach on existing data, Um, and that's a very interesting area, and again, we'll probably have much more involved technical discussion. But the work currently being done in what does it mean to find causal conclusions from a statistical model? And can we do that to some degree without either observational or with approaches like reinforcement? Learning, I think, is a really, really fascinating areawell, I would put active learning under reinforcement, learning and I think active learning. On the one hand, I think it's underutilized. Ah, our experiments with it have been Ah, very encouraging. Uh, it is interesting, because on the one hand, it kind of breaks a bunch of things that you would expect from normal supervised learning. And that often you're learning curves don't involve Montagne tickly in the same way. It's hard to figure out sometimes what's going on. They're really, really isn't a in my mind a huh? An elegant or a final theory of of active learning. I see it umbrella for ah range of different approaches. Um, I think the combination of active learning in terms of, um using a kn incremental approach on online approach where you used the online aspect to make decisions about what data to collect, together with the idea of semi supervised learning where essentially you're looking at, Can I learn something? For example, in a neighborhood from unlabeled points around the points that have been labeled by making some perhaps smoothness assumptions, I think those go very nicely together. Um, I think it's really a good point, and I think you glad you brought it up That there are these frontiers and they do address things that are very relevant to the evolution of the field. Supervised learning has always had the labeling bottleneck and what we can do with active learning, more efficient experiments and making assumptions that let us get extract information from unlabeled points in addition to a small labels that these were all I think, areas which deserved to be pushed on. No, I'm gonna pivot. I now give you another opinion, which probably you are going to bring up, which is, But what about deep learning? Um, I have to say, I don't play very much in that area. Um, for a couple of reasons, one is that it's not applicable into many of the areas where I'm working, where I'm working with either small data sets such as time serious and small here, maybe in only a few 100 megabytes, right? I'm working with small data by deep learning standards and also most of the techniques and tools that I've learned coming out of the the bays nets. The basing statistics, probability modeling dynamics and value driven kinds of models which optimize against a particular value function are really a different sub discipline, then deep learning now deep learning, of course, should not be discounted for what it's been able to do. What I call perceptive problems comes where you have a data feed. Um, that is very much like a kn animal. Sensory or perceptive A source. But let us not. Okay, take that and assume that it's going to, uh um uh, replace what? The areas that you've mentioned that I mentioned that I I think, um, are in danger being overshadowed.

What was the hiring process like for your job? What were the roles of people who interviewed you? What kind of questions were asked?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
you know, there's no become kind of a standard data science interview format, which starts with a technical interview and often includes, uh, a problem, often from statistics and, uh, off in the oh a, um, exercise in starting from a poorly framed problem. See how a person could frame it Better. I do that kind of interview. I, uh I think it should be distinguished from a pure computer science interview, and I think we make that that distinction clear when we when we do interview people for Daddy.

What qualities does your team look for while hiring? How does your team interview candidates?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
another hand, I expect a person toe have a reasonable facility, not necessarily a deep mathematical ability. I mean, I'm not looking for a person to prove theorems in abstract algebra or in, um, uh, stochastic processes or something like that, but they should have on their fingertips a very good understanding of the use of probability, uh, working in high dimensional spaces how you, uh, capture or, um, do things like an SPD decomposition or Eigen value decomposition or what it means to have Aryan values or when different agencies have different kinds of values? Why that's relevant to solving a lot of problems in high dimensions. Um, Microsoft to pivot does have a particular, uh, interview process. And I think for people who are coming out of the university, it's important for them to explore that and perhaps beyond way beyond what I can do in an interview here to both talk to recruiters who will be very open about what they're expecting in the interview process and talk with recent hires. Ah, but, um, typically, we have a full day after we've done initial qualifications. After prisons had a phone interview and reached a first level technical qualification will bring a person in for 4 to 6 interviews of people. Um, some will be purely technical. Some will be, for example, with another data scientist. Another interview would typically be to understand, um, you're your particular contribution. Do you understand something about that domain? How do you fit in? And then there's always any interview of, um, how do you fit in the group? No, Uh, of course, there's a clear understanding that there's a you know, that interviews have to be on the level that we don't discriminate, Um, and that Ah, we not only follow the law, but we have, ah, clear understanding of the value of diversity. Both, um, from people's backgrounds and also intellectual diversity. I think that's really important. Um, in my particular sense, I would ask three questions that need to be covered in the interview process. One is, Does the person have the talent? Can they solve the problems that we were approaching? But that's only one of the three. The other is, um, do they want to work? What's their motivation? Are they strongly motivated to do this? And I think that, you know, that could come across clearly um, positively or negatively, even if they are qualified, you know, it's it's not a question of giving a person an award when you give him a job, and then the last one, I think, is really important. It's, um can they work with the team? There's a very strong emphasis here in your annual evaluation on your ability to contribute to a team in both ways. In both do you contribute to others? It's to do you let others take credit for things that you have done that helped them out, and vice versa. Do you also borrow from other people? Do you know how to exploit, uh, the talents of others to the benefit of the larger project in a way where everybody feels that they've been rewarded? Uh, so I think my experience is interesting, I think, coming into ah hiring and by virtue of having interviewed with many startups and actually I would admit in silicon value are almost always in her feeling, for one reason or another. I think a company's interview style, how they approach prospects often is very revealing about the company and its culture as a whole, and I think for a company that is very well organized and can interview well and puts the interviewer in the best light. Uh, speaks very strongly about the company.

What are different entry-level jobs and subsequent job pathways that can lead students to a position such as yours?

Based on experience at: Principal Applied Research Scientist, Microsoft
Summarized By: Jeff Musk on Thu Feb 06 2020
Well, remember I didn't come through a conventional, um, higher through companies, Career Path and Mike and I have to say that data science, I It's hard to say exactly the career path. They're still not so clear, I think, practically if a person is interested in in this kind of work or growing in this kind of work and Microsoft, we bring in about 6000 people a year for as recent college grads and again, you know, realizing this is a company that's largely building software or his building hardware. It's building software and hardware that a a good, solid familiarity with computer science and software. Even if a person is not a computer science grad, which I don't think we need the majority of our hires to be, just realize that's the nature of of the company. If you're interested in working care, Um, like many high tech companies, we recruit widely. Ah ah, lot of our hires. A recent college grads are done through the internship pipeline that we, um, recruits strongly for if someone is interested in joining Microsoft, I think the first thing they really have to do is while they're in college, Look at the internship opportunities during the summer, or or possibly even during the year. And those are available when we bring in an intern. Ah, it's not a ah a, um, it's a pretty in depth thing. We put you right in the middle of work. We typically get some really, really amazing results from our interns. And when we find an intern who is Ah, good match on and has a successful internship, we will often let them walk out of the building after the rain shirt internship with perspective offer in their hands. So, uh, the practicality of getting huh? Hired as a recent college graduate, um, often goes back to your, um um experience, um, through an internship program for mid level applications. Um, again, I think there is. It's it's important to be part of a professional community and, ah, again, just the practicality. Is it knowing people in the company, uh um, I laugh because, you know, if you think back to my experience, I've just having lunch with somebody, and, uh, the discussion was around linked in well, lengthen, you know, is actually part of Microsoft. But I don't think there's a job I've gotten in the last 10 years where Lincoln was not a crucial part of the process of both discovery. It's exploring Ah, uh, the possible people to talk to and and actually on the mechanics of the higher, Um, I think in high tech that that's now much the case.