Boeing Artificial Intelligence Chief Technologist
Oregon State University Ph.D., Computer Science / Machine Learning
Current Time 0:00
/
Duration Time -:-
Progress: NaN%

How did you get to where you are today? What is your story? What incidents and experiences shaped your career path?

Summarized By: Jeff Musk on Tue Jan 28 2020
A friend of mine gave me Pat Winston's AI textbook when I finished high school. And although, as you know, that's mostly about symbolic AI, I said I wanted to do AI. After that, I did my undergrad in Eastern Europe, in Romania. There was one AI faculty was an expert in expert systems. I tried to stay informed about advances in AI. I tried to follow as much as possible. This was pre World Wide Web. I tried to buy books in AI and around my fourth year of undergrad, there are five years of undergrad in Romania, around my fourth year, I started learning about neural nets, and I implemented Backprop myself single AI neural net. I also implemented some recurring neural nets myself. Really simple with one-two nodes. but I continued work on expert systems because my professor was working in expert systems. My dream in AI was to build a system that is able to solve geometry problems. And as you know, it is still, an open question. So I'm talking here about the synthetic geometry problems, which typically have a logical answer. This was my work. I didn't do any research at that time. I was an undergrad. And then I got in touch with Tom Dietterich, who ended up being my Ph.D. advisor. I contacted him. I said I wanted to do work in machine learning, and I worked with him at Oregon State. So, he was very influential in my career. He was also a person in the mentor who exposed me to all advances in AI and also computer science. I'd like to emphasize that in AI We are computer scientists. Above all applied computer scientists. And, it helps a lot, although, at some points, it may not seem important. It helps a lot to understand the fundamentals of the theory of complexity to have a good understanding of how operating systems work, how computers are architected because then it's gonna be easier for us to build good applied solutions. That's the story of how I got involved in AI. Incidents and experiences. As I said somebody buying me Pat Winston's book in a time where there was no Amazon to search for AI books. Also, I think it was everyone who I mean if you have a passion in an area, you as an undergrad or is an advanced high school student or a junior graduate student should be able to pursue it and start to do a lot of work on your own. As I said I was in Eastern Europe, but I bought myself some neural nets books and some AI books with very little money I got from my scholarship. Basically, it's gonna be a lot of work. Our field is computer science and the product of computer science is code. So remember we tend to see now a lot of people trying to talk about the AI but forgetting that in our product is code. So I recommend that everyone embracing any field of applied computer science to have good coding skills and good software engineering skills. I tried to understand them as much as possible. I'm not seeing myself as a hacker. I'm not in computer science because I really enjoyed the code. I like to code because I want to see a solution in my applied field. That's in short my story.

What are the responsibilities and decisions that you handle at work? Discuss weekly hours you spend in the office, for work travel, and working from home.

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
As everybody will learn as you advance in your career that you don't have the luxury that you have in graduate school to just work on a project or a small set of projects. First, I have to set an AI strategy for Boeing. And, as you know, and as everybody knows, probably in 10 years, it's gonna be clear what the frontiers of AI are and what are the problems in AI. Right now I work a lot in educating others who are not AI people, both engineers, but also managers and executives on what's possible and what's not. Nowadays, a lot of our work and a lot of our results are taken and interpreted and discussed outside by people who are not AI people. Also, it's very easy for somebody who doesn't work in the field to interpret a result in a way so that he or she thinks that some other solution is easy. For example, the steps from detecting objects and images to understand interactions between the object to really understand images. Some people think it's a trivial step. No, we all understand that. We need a totally different set of approaches for that. So you know, just that as an example, I have to explain it to different people, for example, who have problems in computer vision. Also for example, if you have enough data on detecting objects and images, you and I and everybody know that it's a totally different problem than let's say detecting anomalies in images in which you really do not have a lot or maybe you don't have at all images with some classes or object. So, AI Strategy, trying to plan projects and to discuss and to estimate what kind of effort is needed for a project that's quite a bit of my roles. And also, technically, I always keep one or two projects for me to work for hands-on. Now it's somewhere between 20 and 40% of my time. I try to make it as large as possible. I have two projects right now on which I worked myself on and then here's another rule that we researchers in machine learning and AI should have is the problem formulation, which translated in technical terms is actually the problem abstraction. People in the real world don't have machine learning problems. They have optimization problems or predict which airplane will be produced on time or not, that has to be mapped into a machine learning problem or into an optimization problem that we can address. Sometimes people don't care that we have enough data or a lot of data or a little bit. It's up to us to understand how much data we can collect, how much data and what kind of features and what kind of noise they're affected by and how non- stationary the data is and so on. So that problem formulation, I think every student, every senior graduate student and every AI person who steps in the career should be able to sit down with somebody who has minimal knowledge in optimization, machine learning and so on to formulate a viable plan to solve the problem. The reality is that virtually no practical problem is actually just one classifier or one step or one policy that can be learned through reinforcement learning. Our solutions again it's code. We have to architect our solutions. Most of the time now we have in many places, we have learned models and then we have some optimization problems, and then we have a lot of code that do standard things. So this architecting falls into our hands, and that's sometimes if you do not have a good understanding of the tools that are available, that can make or break there. That's the difference between a good group and they're not so good group in addressing problems. We have had people internally and externally, people groups of six trying to address a problem for half a year, and they weren't enabled to. And then once you had somebody who understood the nature of the problem and the nature of the uncertainty in the data and the nature of uncertainty in the real world to address the problem, one person in three months. So, that's my role. And, I have to make this happen. And then we all know that we have to make it happen with the people that we have around. 

What tools (software programs, frameworks, models, algorithms, languages) do you use at work? Do you prefer certain tools more than the others? Why?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
The preference is driven by technical advantages. Nowadays, if somebody asks you what tools, probably they're thinking we are using PyTorch or Tensorflow or something. And obviously I and my team, we use all these tools. We also write, quite a bit of code and algorithms ourselves. But you know, for deep learning PyTorch lately quite a bit of PyTorch, TensorFlow. I want to point out one thing in AI we use and I'm looking more and more at probabilistic programming. In TensorFlow, you have TensorFlow probability. So if you have an idea on how to model the uncertainty in the task and you can exploit that for example, you know how to model the certainly, but you want the actual probabilities, you can learn those from data. It's actually pretty useful. We started to use probabilistic programming tools also for selecting a hypothesis. So let's say you have two hypotheses learned by the same algorithms and they do the same on your test data. Which one would you prefer for the real world? Here's one thing that I care more and more as you know we teach that. We have training data and test data. 

What things do you like about your job? Were there any pleasant surprises?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
I joined Boeing after 9/11 and first, there were less pleasant surprises because there was a shrinking of the economy and virtually all companies around the country headed downturn. What I like is that I am exposed to extremely hard problems almost on a daily basis. And that may be a blessing or also, maybe a disadvantage if you're the type of person who likes to produce a solution every day, I have very few problems on which I have the luxury of going to some tool and produce a solution. Just to give you an example. We're looking now at using machine learning for design which is extremely hard. Large scale beijing optimization problem. We have all sorts of normal detection problems that have their intricacies, and we don't have the advantage of or we don't have the luxury of allowing lots of false positives. So the practical problems that I'm exposed to in a company like Boeing are unique, and I'd like to invite everyone to think about joining a company like this or like our competitors in this arena because these are super hard problems. I can list at least 10 hard problems in which we all machine learning people could sit down and think about for weeks and weeks. So that's both the pleasant surprise and the big challenge that I like it here.

What are the job titles of people you routinely work with inside and outside of your organization? What approaches do you find to be effective in working with them?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jyotsana Gupta on Tue Jan 28 2020
So inside Boeing, I work a lot with mathematicians. Boeing has had an applied mathematics group for years. Now, this is a group of people who have a Ph.D. in mathematics, so experienced people in statistics, probability, optimization, and geometry. So I have worked close to them from the moment I joined Boeing. I love them. I learned a lot from them, and I think virtually all solutions that we have produced in Boeing in machine learning and related areas were done with my colleagues in the mathematics school, which is a sister group. Outside, I work a lot with universities. We have projects with different universities all over the world. The job titles are professors and graduate students. So, we have projects on fundamental research questions on applied research questions mostly these projects fall in between two areas. So, for example, we work in applied machine learning applied to aerospace with colleagues at MIT the same for example, at Stanford, we work on an interesting project on modeling hardware and applying machine learning for modeling, unique hardware that needs to be designed then, the third category of people are people in the industry. My colleagues', peers of mine in different companies. So, for example, now I'm involved in putting the first steps into and looking at the challenges for certifying algorithms that are based on our current algorithms developed in the AI world. And here we work with colleagues at Airbus and at car companies together to set standards and to just lay the groundwork for the certification of these algorithms. Approaches, as I said, I learned a lot from Tom Dietrich and what I learned is so the applied world is vast. I learned a lot from my colleagues. Not only mathematicians but physicists about a different kind of phenomenon. So you have to be open to listen and to be willing to understand complex phenomena that sit in a different world than yours. So, in applied AI virtually everyone will be exposed to the phenomenon in physics in aerospace, in geology. I have had the project where data came from, it was on rocks and sand and so on. So, the approach is to read a lot and to try to understand we always are Applied Computer Scientists. I think we have to go towards the problem and to try to understand the nature of the problem. Let's not forget that we do not have our problems. Machine learning is not a problem, it is set of tools that solve practical solutions. So we have to understand the real problems which sit in the world of mechanics, aerospace, chemistry, geology and so on. So I think it's helpful if somebody takes a job in this field to understand the fundamentals of the sciences that are studied by the company. I learned a lot about aerospace and about how airplanes fly when I joined Boeing, I knew, I had my classes in control, but I had to learn way more than that.

What major challenges do you face in your job and how do you handle them? Can you discuss a few accomplishments?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
First accomplishments. The biggest satisfaction for us, computer scientists, is to have a piece of code deployed and to see that our algorithms run on an application. So one of the first things that I worked on when I joined Boeing and now Boeing has an entire division working on that is predicting falls of airplanes when the airplane gets to the gate. It's connected to a computer, which takes the data from the previous flight and other types of data and tries to make suggestions on what things to replace, what parts to replace and what maintenance actions to do. Some of those models were worked by myself and my colleagues, and now they're deployed. So that's a satisfaction. We also deployed models on the defense side, and recently we have worked on autonomy and we have had deployed algorithms and on building an autonomous airplane. I cannot talk more about that. But it's a major satisfaction. So the satisfaction comes from seeing the algorithms deploy and work the way the people who interact with them expect them to work otherwise, they're not useful, and I and my team have had the number of methods deployed. Major challenges? I think the major challenges stem from this hard interaction between our set of tools and the problems in the world. These are extremely challenging. At the beginning of a project to formulate the problem right. And sometimes it can be frustrating. Sometimes it may end up in nothing. I've had beginnings of projects which ended up not in a project, simply because either myself or people in my team didn't understand well, the practical problem or on the practical side, there was not a willingness to understand our challenges. So one challenge that we have in machine learning is we love to run experiments right? We love to generate data and to try out. In some cases, it's risky and costly, Right? As we know autonomous cars or autonomous airplanes. You really need to deploy the algorithms on a platform and to run experiments and collect data and see and tune the algorithms and so on. Now this requires money and investment, and you have to have an understanding of that from the stakeholders the people and the customers in the organizations that want your solutions and that's a challenge. We have to explain this ourselves. I think this is the major challenge to explain ourselves why and how costs, sometimes high costs are warranted for a solution. Let's be frank, in machine learning successes come in steps, right? So you and I may work on a solution and we see that it's very promising initial data. But you know, people outside may not see it until the solution is deployed. So I think that's a challenge. We always have to convince it to build this trust with whoever the customer, the user, the consumer of our solution. This is a major challenge.

What are the recent developments in the field? How significant are these improvements over past work? What are their implications for future research & industry applications, if any?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
As I said, for me and my team probabilistic programming and ways of representing knowledge in a way so that we can both exploit data and domain knowledge, which gives us a representation off dependencies between observations and phenomena. It's not that it's a recent development. It was always desired of AI to bring together the symbolic learning, but now I think we're closer than before. The major thing about deep learning is that we learned the abstractions, right? Can we exploit this in, if we have minimal or high-level representations of our domain? So I think that's the recent developments and results in this field. All the work in learning causal influence in Lawrence structure or doing causal inference in Lawrence structure. I think that's major, and I'm trying to follow the literature in that. They're quite a few super interesting papers. As I said, I have two projects on my own. One is about robust machine learning. I think at this point we are able to formulate the research questions that lead not only to high accuracy for our classifies but to robust classifieds. That means classifies that perform in a way that they can be trusted in the real world. And you know your third question on implications. I think it's a major. Doing inference is our ultimate desire, right? So predictions are good but inferences, we always see inference as, you know, one of the ultimate things, one of the ultimate desires for our intelligence systems. So we have to be able to do inference with our models somehow. Robustness the implication is major because I think all this work in robustness will start seeing trackability into deploying our approaches into high stakes applications, high-risk applications. Our colleagues working on autonomous cars and our colleagues working on a high stake, all of them have some research on robust machine learning and AI. I think the implications, are that we will be able to deploy easier and with more trust and with fewer costs. So those are the implications. 

What was the hiring process like for your job? What were the roles of people who interviewed you? What kind of questions were asked?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
So I came to a research organization. Our organization was the first industrial AI organization. It started in the seventies. Boeing has had an organization of symbolic AI people. Those people were gone by the time I interviewed. So I have to say that I did two internships during graduate school in my organization. So I knew some of the people. I was interviewed not only by AI people. I was interviewed by neural nets people, I was interviewed by electrical engineers who were doing it that time work in fuzzy systems. But I was interviewed by mathematicians and I don't remember the questions that were asked. I had to give a technical presentation. It's a research organization, a technical talk, very much for academic jobs. I was interviewing also for academic positions, and I kind of gave the same talk, a little bit focused on the application importance. And managers there was a structure basically like today. I was interviewed by the director, by the second-level manager and the manager. So it was a kind of a standard research interview. I don't remember the questions. 

What qualities does your team look for while hiring? How does your team interview candidates?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
The interview process can be about exploring what the candidate knows, and that's what we want to have. We all want our candidates to have a basic knowledge of the field. I'm looking personally at candidates who really are interested in learning, and, the candidates who showed interest in learning and curiosity, who are eager and hungry to learn more. Those are also candidates who also have proven to perform well. Right now in our field, I said, apart from being applied computer scientists, we are a core computer scientists. So candidates have to have a core computer science knowledge, and also to have a minimal mathematics background. So linear algebra, calculus, and also probabilities. We typically have questions on probabilities and on inference. So these are areas in which we try to see how our candidates do and the interview process even now, candidates in AI are also interviewed by our colleagues and Mathematicians. We work very close together. It would be very hard to come into Boeing if you only have a hammer. So if you're just an expert in one narrow domain and have little or no expertise in anything else. So you have to have some sort of broadness. Now we know that we researchers, we tend to be more focused on AI. However, there's some breadth that you need to have to come into Boeing. However, as I said myself and others, you don't have to have an aerospace background or, to have aerospace knowledge to come into Boeing. So we have people who have learned a lot in this domain after joining our organization.

What are different entry-level jobs and subsequent job pathways that can lead students to a position such as yours?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
Most people in our team are of a Ph.D. level. Or we have programmers who are mostly software engineers of masters-level and so on. We have jobs in all areas of applied math and AI and Boeing very much like other companies aside from having people advance in the managerial arena, we have the technical fellowship which people advance up to a level of corporate VP now in the technical fellowship. And, the technical fellowship in Boeing, is structured very much like the technical fellowship in IBM and in other companies encouraging people to stay technical to build depth and also breadth in their field. And become the experts of their field in their arena. So that's where my learning of basic concepts in aerospace and how airplanes are designed and how they're operated comes into play. As I said, probably the most important skill is the capability and willingness to learn. We in research in any field anywhere, right? We all know that learning doesn't stop the moment you get done with your Ph.D. and we all carve ourselves through different paths in the careers. We all know AI people who are very successful. And then they carved themselves successful careers in totally different areas. I have mathematician colleagues who now work in machine learning, I have machine learning colleagues who now work actually in aerospace which is completely different. So if you're willing to learn, you can carve yourself a path. But passion and ideas in any company are appreciated.

How did the program prepare you for your career? Think about faculty, resources, alumni, exposure & networking. What were the best parts?

Based on experience at: Ph.D., Computer Science / Machine Learning, Oregon State University
Summarized By: Jeff Musk on Tue Jan 28 2020
As I said, my P.hD advisor was amazing. Also, the department at Oregon State is a small department. It has 20-25 faculties. So it's not a big department but here are some computational complexities and never realized the importance. So I had to take classes. As we all know we all take classes during our Ph.D. The classes in computational complexity have shaped my thinking. And although I'm not an expert in computational complexity, that knowledge gives you good fundamentals. Also, Tom Dietterich had a graduate class, not in machine learning. I don't remember what was called, but it was focused on applied computer science. So the class was structured like this, and I recommend to every school to have a class like this. So it was on how to write the grant proposal, how to write a paper, how to review a paper, how to run experiments, how to analyze data. So this was a graduate-level class which was extremely useful. Then Mike Quinn, who is now a faculty of the different universities here. He's the dean of Seattle University. He had a very good class on not the technical parts of distributed computing but perspectives on distributed computing, which was extremely useful. So I think for you as a faculty, we in the industry right now, I'm involved also with other universities are shaping classes. We have the core computer science classes. Then we have the core applied classes writing AI. We have to think about classes, not many, that students take to prepare them for when they end up applying there and becoming researchers, either in academia or in industry. We all have to write papers. We all have to write grant proposals. Even if you don't write grant proposals, if you work in a company that has enough money, you write a proposal to your manager or something. We have to be able to explain and justify asking money for a solution in AI So that was important. Then I was exposed early. As I said in Oregon State, there's a small department, but I visited CMU and I directed with Herb Simon, for example, I visited MIT during graduate school. I was a part of some grants in which I interacted with students and other schools obviously participating in ICML and NIPS. And, all the AI conferences were very useful. I went to a doctoral consortium at AAAI. It was very influential for me. One of my mentors was Niels Nielson. And, it was in AAAI 1999 in Orlando, where Niels Nielson spent time with me talking about not so much about details on AI, but exactly on the types of questions that you tried to address here on, having a broad view on all approaches in AI. And who better than Nels Neilson who has worked all fields of AI. He has worked on neural nets, Statistical AI and Symbolic AI and so on. So that was very influential. I recommend all the students in AAAI to consider this doctoral consortium, which is part of AAAI during the workshop or similar events. Each guy has them and I think ICML tried to build one of those so that that helps, especially if you're part of a smaller department rather than the department, such as the one at CMU.

Do you have any parting advice for students and professionals starting out in your field? What three mistakes they should avoid? What three things would help them the most?

Based on experience at: Artificial Intelligence Chief Technologist, Boeing
Summarized By: Jeff Musk on Tue Jan 28 2020
Have a passion and an interest that is yours. Not because somebody else is successful. If you're interested in deep learning architecture, I'd prefer you to be interested in that for good technical reasons rather than oh, now it's cool to work in deep learning. So that's very important. That's a big do. Listen to others and see good mentors. Try to seek a good mentor, not necessarily in your field, but also outside of your field, in math or statistics or applied field. Let's say if you work in, Marine Sciences, try to have a mentor in that area. Also, try to have a good understanding of previous work. So I go back to AI papers from the seventies and eighties if you can afford that time was I think it's important. Now, these are also don't. Don't follow others just because you know they're successful. Now try to carve your way. Always a good technical reason and a good motivation for why you are interested to work on the problem that you work on, rather than financial interest or, to be famous and so on. I think these are my advice for students and professionals. And right now in AI let's stay away from the hype. We all know the goods in our field. Let's not oversell it. Let's not undersell it. So I'm kind of frustrated by some people who try to oversell. We all know the limits and I think we're part of a fairly honored community, and I hope to keep it like this.