Apple Machine Learning Research Manager
University of Cambridge Doctor of Philosophy (Ph.D.), Mathematics
Current Time 0:00
/
Duration Time -:-
Progress: NaN%

How did you get to where you are today? What is your story? What incidents and experiences shaped your career path?

Summarized By: Jeff Musk on Tue Jan 07 2020
I began with a career in academia. I was a mathematician, did a PhD in Maths. I started at Cambridge University. And then I did some postdocs, Cambridge and Southampton then back in Cambridge again. So I started out doing Stochastic network modeling, then branched out into Rotation Systems. I also did some kind of game series and sort of stuff, but after I've done three postdocs, I didn't quite make it to the next level of academic career and become, like so in the UK you would become a lecturer, but in the U. S. you will be sort of like Assistant Professor. I realized I was getting to that age where I should be moving on and I think I hadn't ever been because I moved my focus around a little bit I never ever really got fully established in an area on, but also it's quite hard. It's hard to make that transition from being a Post Doc to becoming a faculty staff member. So as I kind of saw this looming up at that time period of my career. So starting my last postdoc, I realized it might be my last postdoc. As that was coming up, I spent some time by diversifying my skills and understanding. I taught myself more about coding, programming. I looked into what it would take to become like a software engineer. Due to the technical work, I had previously worked for six months as an engineer. It's a company called Cambridge Consultants. There was some sort of theoretical work. We mainly programmed that Matt lab and sort of prototyping things. I kind of tried to learn a bit more about programming in general, become an expert. After I finished my work, I was looking for a position. I was excited about working machine learning on and there was a spinout company from the Cambridge University engineering departments that was a member during the postdoc. They were looking for people and I interviewed and I joined a start-up quite early on. It turns out that I actually suited startup life. In programming and then in later management quite well on, because it is a very small startup environment. If you take the initiative and you simply listen to what needs to be done and do it, then you become in charge of that thing. That allows you to expand your responsibilities and get experienced in things that you might not get to do in another company because they're much less restricted. It's very much a single swim kind of environment. It's a good way if you come from academia like I did maths. I had the mental ability to do a lot of things, but not sort of the technical knowledge on the job experience. So the start-up went well. We got acquired by Apple and now I'm managing people inside Siri. It was a huge change in my career trajectory and truely the big break was doing the startup. Before the startup, it would have been quite hard to get a similar position. A PhD is gonna be useful if it's extremely relevant. If I had a machine learning PhD might've been easy to get a machine learning job. But even here within this company, when we're interviewing people, a massive PhD is not taking quite the same way if he hasn't got industry experiences. But what I found was that a lot of the things you need to know to be a good software engineer, they're not that hard to pick up if you've got a very good technical mind and if you pick up things quickly. It does help to actually do some of the research beforehand. If you want to make that transition into industry, do some of the groundwork. These days, the environment, the job market is good for loads of internships and lots of companies. That's something that's really valuable to do. You can do internships with a professional body, you can get experience. It looks great on the CV and you'll probably get contacts. People will probably wanna hire you. I hire people, speaking with my experience, If I work with someone during an internship and I know they're good, then I'll hire that person. 

What are the responsibilities and decisions that you handle at work? Discuss weekly hours you spend in the office, for work travel, and working from home.

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
My job has transitioned from one where I was I was coding and doing machine learning research and product development. So my team is responsible for machine learning components, a couple of other side projects within Siri. We do some research to try and develop the techniques, and then we privatize and deploy them. So my role of transition from a machine learning scientist and engineer, developing and coding, create production code to the one where I saw managing on doing that. So I was managing the projects and also I do line management as well and then over time into transition, where my teams got bigger, bigger, I'm actually doing more with the management responsibilities and now I don't really do any programming. But I do technical leadership still. So I'm involved in discussions where we'll talk about problems and possible solutions, possible algorithms, ways to approach things, ways to investigate things and my experience with the research in the past means they're going to have a reason to get intuitions about what things are gonna work or things are not gonna work, I guess. A guide for junior people, especially machine learning. You know what kind of things to try, generally tend to find that people get really excited about possible solutions on devices should always be. Have a look at the data and try and understand what's actually going on first. Maybe the problem's very easy. Or maybe the problem's very hard, but you must understand the area. So in my day to day, I spent a lot of time communicating with other people, having discussions about what work should be done in the future. Currently, we're in a planning phase. So a lot of interactions with other teams to try and come up with a road map for how to make Siri better. And how our component should evolve, how it should interact with other components. That's a big part of my responsibilities. Certainly networking with the By the rest of the time, I spend on assigning work for people and detailing planning solutions, having discussions, line management. You have to organize people. And as a line manager, one thing I take very seriously is like personal developments for my team. We have to find out where do they want to go. What can you do to help get them to carry in that direction? So it's a little case of kind of finding out what kind of things we want to work on, what experience I want to get and make sure you line things up currently. Make sure they're doing things that are interesting to keep them motivated. Motivation is more complicated than just what you're working on. So I live in San Francisco, but I work in the UK fairly frequently. So I do travel kind of a few times a year at least. So when I'm in the UK It's pretty much a 9 to 5 job. When I'm in the U. S. there's quite a long commute from San Francisco to Cupertino. I tend to work on the bus, so it's a bit more. I spend a bit longer working, but I don't tend to do it. It's not too crazy. Actually, it's quite reasonable. Things were a little bit crazy during the startup. Not too bad but I think we were quite lucky. The general policy that my managers had kept was we allow one day a week for working from home. Well, they're flexible when people want to do a couple of extra days because maybe they need to look after their child or something.

What tools (software programs, frameworks, models, algorithms, languages) do you use at work? Do you prefer certain tools more than the others? Why?

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
I use a lot of different languages. So we've got programs in Python, JavaScript, c++, Java, Kotlin, Spark, Big Data, stuff like that. In terms of frameworks or machine learning frameworks, I think I've mostly used TensorFlow. My team quite like Pytorch. I think that TensorFlow kinda hits the good spots of flexibility, usability, and extensibility and also there's a lot of support for it in the community. Models and algorithms? So I mean I don't think there's any I can think of that's unusual that we've been using which is outside of the usual computer science type. We need to study computer science. You learn classic algorithms. With models, generally, I feel like the right way to operate a machine learning problems. The first thing to do is sort of investigate. See what's going on there. Then you need to start out with a very simple, structured model. See, what if the simplest thing works. So the easiest way? Well, actually, the idea is to start the heuristic and the kind of cut your heuristic up the features and feed into the linear model that's like pretty simple. And then a nice thing to do after that is to kind of try out a mix of different approaches. You won't find that kind of a good bit of future engineering. I found that in a linear model with pulling all the future combinations can be expressive enough to kind of do just as well as the recently used a small neural network. And you really don't need like a big, deep learning network until you get on the next stage of really big data. It depends on the problem that you're looking at so obviously in some areas, Deep Learning's the industry standard. It gets huge benefits. Generally, with a new problem, a good way to start off with the simple thing first and build up your complication. 

What things do you like about your job? Were there any pleasant surprises?

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
What I like the most is the team I have. I kind of enjoy really working with the people I like. That's I think the most important thing. So, generally, it's often said actually that for most people, the most supportive thing is to have a good manager, and that's what they call it the best work. I think if you end up being a good manager, that good as well because you can make friends with people that work with you. You can also help them to have a good job that they enjoy. That's quite nice really. Pleasant surprises? I think this job is a successful job, in the sense that when I fly for work in the UK and back, I get to fly in business class. The salary is quite a lot compared to what I was doing in academia. It's kind of rewarding doing something which isn't hard. I think this is easier than being an academic researcher.

What are the job titles of people you routinely work with inside and outside of your organization? What approaches do you find to be effective in working with them?

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
I work with software engineers and software engineers managers, machine learning engineers and machine learning managers and scientists I work with designers and their managers and I have meetings with executives as well. And Quality engineerings and quality insurance. They have people who will be of use to the test and also test engineers as well. So I think dealing with these different people mostly isn't that interesting. I think so. Sort of machine learning people, we talk a common language. It is important to collaborate and have good communication to talk about things with people. So when you are a researcher you have an idea that you can attribute yourself and get credit for it. But in Apple people are very collaborative. It's good to sort of come and talk to you about different ideas and get their ideas as well and try and generate something together. That is something I really enjoy. Software Engineers, there's obviously some lack of skills and language. Sometimes I think people don't understand what machine learning can do. So there's sort of two different problems, one of which people think, think machine learning is magic. And you have to kind of dampen their enthusiasm and other people who have experienced something like interactive machine learning on projects before they think machine learning is useless because some people came in are far too ambitious and didn't really know what they're doing and then they come back saying "I've heard these promises before in the same way last time, it didn't work." But this time we're trying to do a better job in terms that because I think just being great, where there's a desperate level of understanding about approaches, I think you're being very clear about what you believe you could do. Not kind of what you think might be possible. And also how much work is gonna be. Getting good at estimating how much work it will be is a really vital skill. You're in corporate environment. That one probably is useful in research as well. Academics that I know. You tend to be really bad time in time management because you think, "Oh, I'll just whip this up and it takes a week. One of the interesting groups of people I work with the designers. That's interesting because designers want to design how Siri behaves, what those interactions should look like. Once he's built models that learn how to best behave, how to control the behavior of Siri. So this sort of kind of a conflict between the two problems. And you have that typical kind of problem issues when you explain what you want the system to do and then the designer is like okay you make the model do this when this happens and do that when that happens. how do you feel? You're thinking in programming terms, how does it work? A flow that I kind of find quite useful and effective with designers, I probably use these for clients as well, if you work with them, because it is the same with every client, is you have an idea about how something you can do; you need to go and explain it to them in terms of the rough behavior. And then ask them to figure out how they want the fine details to work And if they explain to you what their vision of how it should work, you can then turn things like descriptions about in their mind when X happens, why it should trigger Y in the system. X could be something that could become a label for a greater, it could be a particular so you know something that they think, when this has happened, could be a request. So make that a designer understands it could be something that a creator can understand as well. There's something we can listen to and they can like to help design the great guidelines and then X triggers Y, which is very easy after the model has determined X and Y

What major challenges do you face in your job and how do you handle them? Can you discuss a few accomplishments?

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
I can't really discuss things that I've worked on. Uh, I think I need to You. I talked to the apple, our communications office first, Major challenges?. I think in a corporate environment, many times the challenges you have are that it could be slow to get things done. There are a few reasons for that. I guess this is often the biggest challenge you face if you're working in a large team. Things could slow down and hold. The reason for this could be a lot of them. It could be sort of technical death in your codebase, things that could be kind of medicated by having good coding practices. We try to practice a lot especially my team. I encourage that strongly, so often, especially people who work just for themselves. Who code in, if they got it, want to experiment for a paper. I'm just coding for myself. Sit down. You know, like a big page and you're very real school and it looks like it is completely unreadable to anyone else. But you know what's going on there. And if he would have that kind of idea about how to code and you know you don't work, impression environment, you get tolls. Now call your variables Long names and add comments and do this and do that. Anything on this is stupid. It's obvious. Well, okay, just look at the code? That's no still the best way to do things. You inherited some code from someone else that looks like that and you try to go through, it's like a huge cognitive load at the end of the day you don't understand. Maybe it's a good idea to name your variables, things that are readable and have functions describe the functioning. The best practices recording to read it and, also other things like processes like using Git for source management. Think of these as good ways to prevent people from really like clashing when they worked together bigger organizations on a single project. I found people called kind of argue about these things and well, sometimes you have to work quickly, but I personally find that the biggest slowdown could be like 50%. So you could slow down like you could take me twice as long to do something in a really quick way versus during the slow way. But what I do to slow way other people can use it and work with it. So it's kind of twice as valuable more than twice as valuable, especially if you got more than two people in that team because everyone's gonna take huge time to read and understand that, then they have to change. Organizational challenges are much tougher. See the way technical that arises is when you have sort of obscurity dependency inside your code. You get a similar thing happening inside, like an organization where you got obscurity and dependency. So there were times when you know you don't understand how to get something done and another problem would be in order to get something done, you require someone else to do something hands and you don't have a good way of understanding what they're doing. Knowing when they've done it, asking them to do is making sure they get them to kind of work with you. Those kinds of organizational challenges. There's not really much you can do it in each of those other than identify them, raise them and try and slowly asked for a sort of more senior people to get involved and change the processes involves. You fix code by changing your coding style for changing your process, you change management and comments and then in organizations, you fix things by, changing the processes you have, you having documentation and regular communication. But then the other problem, you might end up going to too many meetings.You can bring your laptop and kind of work when people don't say anything useful

What are the recent developments in the field? How significant are these improvements over past work? What are their implications for future research & industry applications, if any?

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
At NeurIPS, there was some predominately illustrated work in reinforcement learning. All over interesting work on combining reinforcement learning with planning systems. That's the stuff that kind of caught my eye. I think Reinforcement learning had a huge kind of boosted interest since the DQN work by DeepMind, proving that deep reinforcement learning could be done was a huge step forward. People rushed to build on that work and find, you know, different things you could do there and it has really ramped up the interest. You know, I went to see ICML and the Reinforcement Learning room was a tiny room with a very small number of people, and it was quite theoretical ideas about algorithms. And then the DQN came out in the and in the next year, next conference, there was a deep reinforcement agreement. It was massive, was packed with thousands of people. I feel like within the field of RL, the development was kind of being sort of progressing sort of fairly steadily notes. DQN was interesting on arguably quite a significant step forward. But then, it required a lot of computing. There's a lot of interest in hierarchical RL and a sort of like to speed up exploration, try and get better laced Atari games that but the detail was bad at, like what does? It was revenge. And I think also maybe things about talks transfer to make transfers condensation as well, like, you know, learning in one in one domain. That's kind of abstract CS and then having that be useful in no time. That's also kind of motivation for hierarchal RL now he has also have any rate is a meta-learning, which is also kind of a growing area there as well of interest. I think I think that's steady progress. I think it'll be a few more years before it really gets a good system, which is kind of hierarchical. And it could learn the hierarchy as it goes on, do sort of lifelong learning and keep building these abstractions. I think that's the sort of thing we kind of need to have very good RL, possibly. But I think we can get sort of reasonably good, I think kind of the implications in the future. I think the future of RL is very strong. I think like robotics it's gonna continue to get better and better. There's exciting proof of concept where I was really interested to read anything that comes from the Berkeley AI. We'll see. I think it's an area. It will be a steady improvement but if we do crack very proper hierarchical learning and abstracting will be very exciting.

What qualities does your team look for while hiring? How does your team interview candidates?

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
The first thing we look at is the CV. The next thing we look at is do you have a Ph.D.? Do you have a Master's?Many people with masters in Machine Learning apply, much smaller numbers of a PhD apply that's currently in States now. I can't guarantee that's going to be the case in three years' time. So things as hard and fast rule that if you do a PhD, we'll give you the edge up that you need. It may not be the case, but certainly that as it is, when I'm hiring any CV with the Phd and I will definitely read thoroughly. Some with masters I'll skim through quicker and I'll also be looking for industry experience, internships are good. If you did an internship in Apple, that's beneficial for me because I can look at the work or I can contact the people who worked with you that's kind of handy. I think it will be similar if you do an internship within the company that you want to work with, even if it's not on the type of thing you want to work on. That's kind of beneficial. People outside the company talk candidly about how it was working with you. Other things I don't for there is also a general interview of themselves. So, I think once you got past, the work experience and the PhD, mostly other stuff in the CV doesn't really make a huge difference. Maybe that's just me. But I find that an impassioned covering letter, I'll read it, but it doesn't necessarily gonna change my opinion. I'm more impressed by where did you go to school? What kind of grades did you get? That's the kind of thing that gives me more solid evidence about who you're gonna be. Then if someone looks good, I do a phone interview with them. I try to understand. There's sort of 2 aspects that phone interview here. So one of them will be trying to understand where are you in your career and what do you kind of want and does that align with what I'm looking for? So that's kind of important to clear that out. The neat thing is the kind of get some rough idea after you appointed like how do you think about work? What do you like about work? Not personal. A little bit of entering into your abilities. Another hiring manager I know, people really feel strongly that he's looking for someone who can kind of tell a story about something they've done. So they have to be able to sort of not just like solving a problem, but then I can come and tell you about when they solved that problem and explain the story of why they did it. What evolved? I've also done a bit of coding interview. So they need to go through a couple of very small coding exercises and just see if they could do it. If they look good, we go into, a full day's interview. That's my process for hiring in Cupertino. You know, in the UK, we have actually, in place of the code of that interview. We have a coding exercise. So it's like a problem. You go where the work of a few hours on and bring back solution. It's quite a bit of work to ask someone to do, but it kind of gives you the opportunity to show your best work. You can actually just sort of hand in some very well designed, well-structured code with comments and documentation. Solving the problem with code about anything. It's very much like on the spot. Can you code on the minute with it? With the coding test, it's much more like how do you work as a professional? Well, how would your work leave an impression? That tells us a lot more information, but it is also your kind of demanding as well to ask people to do that in America. So during the full day-long interview, we have several people who will quiz on different topics. So it depends on you know, what we're hiring for? It is a software engineer. Then we might test algorithms and design. If it's a machine learning engineer, it might be a sort of coding test on that Machine Learning tests and quiz. People on that knowledge talk about what they're interested in. It's a really good question to ask someone in an interview. What was the last paper you read that really excited you? What did you like about it? What would you do to continue that research if you work on the area? Do you have any expansions? That's the kind of question you might get. If you're passionate about the topic, it's not hard to answer that question. I am generally looking for is someone who's good with communication skills as well as being smart, conscientious and decent worker. I think communication skills are an important, really important thing. An interview is to listen to what the interviewer says and don't talk over them and shut up when they try to talk. That's very important. If I were having into someone and I had this a bunch of times and I've done it in an interview, I've been to the one who won't shut up. And I've been talked over. If they took over, you just think I don't want to work with this guy or woman. No, I don't wanna work with them. But it's quite easy to get stuck in that trap where you go it's not actually that you're a bad communicator. It's just that you're nervous and you feel like you didn't answer the question properly. Pay attention to the interviewer looks if they look like they're happy that you've said enough to answer that question, they wanted to go on, It's because they want different things about you they don't really care about that question. Just see where they want to take it. 

What are different entry-level jobs and subsequent job pathways that can lead students to a position such as yours?

Based on experience at: Machine Learning Research Manager, Apple
Summarized By: Jeff Musk on Tue Jan 07 2020
My job's specific when I was hired, was a machine learning engineer and then I gained seniority. I started management. If you become a senior engineer, you might want to try and get a promotion every few years. You might want to skip from one company to another, trying to get promoted. You could try to work in a smaller company as a start-up, this is a good way to get an opportunity. But if you join a team, if you're looking for a good career path, it's great to join a team in an area that's growing. So definitely, that's a good idea. If you doing a startup, that's probably gonna grow unless it fails. If you're joining a big company, then you want to look for an area. Do you want a rapid career growth? Look for an area where that team is expanding because, after a couple of years, they'll get bigger and everything. They want to be looking for people to take on management roles. They'll be looking for people to mentor new people to help, to do interviews to try and deal with some of the line management responsibilities. If you let them delegate that onto you, you get the position of the more senior member of the team. The difficult progression is, for a software engineer as well, it's a sort of an engineer, then become a senior engineer. At that point you can keep growing the career path, becoming more and more senior. That can be harder in some companies than others. And at Apple, it's quite hard to go. There's a level of seniority as equivalent to mine or one below me that you can plateau one if you're just an unusual contributor. It's a bit easier. She just switched the management and grow up that way. And then when you become a manager as your team grows or you deliver more and more impact, you can get more seniority that way

How did the program prepare you for your career? Think about faculty, resources, alumni, exposure & networking. What were the best parts?

Based on experience at: Doctor of Philosophy (Ph.D.), Mathematics, University of Cambridge
Summarized By: Jeff Musk on Tue Jan 07 2020
I think the way that the Maths PhD prepared me. Not just the PhD actually, I suppose the whole sort of degree and a master's and a PhD, I think it prepared me for thinking technically and abstractly and for learning many things. I think it's sort of It's challenging to work and do the math. It's quite difficult subjects at times and yet think very abstractly. And it meant this, computer science concepts made a lot of sense given the in terms of why the skills I think maybe perverse. If I'd done some industry experience before doing the PhD I probably would have a better PhD. So I think that's actually kind of something where industry could be quite good where the pressure isn't. You're under so much pressure to kind of like perform and do publications have worked on your own. You could work more collaboratively. People could be helpful. I'd get more guidance on what's a good idea and what isn't? So with research, you have to come up with something novel. In the industry, you just want something that works. I guess by the time you have something that works, you then have an idea about somewhat you doublethink could actually improve it. So that's gonna be quite nice that the pace off whereas into academia, you have to try to think of something new. Otherwise, usually, you're not delivering research, so that's a quite high-pressure thing. Certainly, networking helped my career hugely. My big break in the startup came because I didn't actually personally founders. I was talking to my department secretary, as I was leaving. She was hiding some paperwork while leaving. And talking to me, asking me what I was gonna do, and I said I didn't have anything lined up yet and she pointed me in the direction of these two families of the Castano. So certainly, networking is very important. I think, other than that, I feel just technical knowledge or something. That technical skill kind of helped me, The best parts of my PhD, I think I enjoyed working with the people there, I or just socializing with them as well. It's great to be with Cambridge University has a mass department, has very smart people. While I was there, Stephen Hawking was there too sometimes. I never met him personally but it was kind of people whose caliber was very high.