Zappos Family of Companies Head of ML/AI Research and Platforms
Emory University Master of Science (M.Sc.), Computer Science with a concentration in Computational Life Sciences
Current Time 0:00
/
Duration Time -:-
Progress: NaN%

How did you get to where you are today? What is your story? What incidents and experiences shaped your career path?

Summarized By: Jeff Musk on Fri Sep 11 2020
so I think it was a lot of lot coupled with being curious and excited about interesting technologies. I started off by coming Thio study in the United States from India and, uh, I studied biology, initially moved into computer. Someone's got interested in robotics is what kind of let my interest in machine learning. And after that it was always just need thio legal fit when appropriate. I dropped out of high school to stop all the early and start by bhd and dropping about cto start example where I broke up. Organization. Exactly. It's been a fantastic journey for been support from my peers, and there is also ensuring that I can capitalize on the opportunity. Thio, Giant cage, whatever organization on the bottom as just kind of proved to be very helpful, move things along.

What responsibilities and decisions does one handle in a job like yours? Tell us about weekly work hours, including the time spent on work travel and working from home.

Summarized By: Jeff Musk on Fri Sep 11 2020
the responsibilities in my particular job right now is any it? Broadly, as I put it, it is deriving information inside from data and making it available for consumption across the organization. So I always think of the field of data analytics machine learning, artificial intelligence as a very kind of service oriented Fiona. When I think service oriented I mean, it's only as powerful as the consumers that I wouldn't do. Leverage later, we provide and, uh, make organization brings with it. So my adaptability examples are search algorithm for the website. Demand Forecasting predicts sizing for customers in shoe price elasticity, profit, intelligent personalization, anything that requires me oh, intelligence. Do made a intelligent piece of information available thio consumers across the order whether it's marketing a traditional experience doesn't matter. Uh, in terms of my literally walk out with, uh, I think it really changes from Thio. Want current fire is and the company. There's always some emergency that we delivered sends our both ends of a regular 9 to 5 work there, but predominantly my kind of spent trying to keep the opposition on being focused on executable and balls and accountability. You may have organization and, uh, what? That's kind of how we it's kind of you

What are the challenges in a job like yours? What approaches are effective in dealing with these challenges? Discussing examples will help students learn better.

Summarized By: Jeff Musk on Fri Sep 11 2020
in a job like mine, it's still an emerging technology. Uh, fundamentally speaking. There's not a lot of broad understanding how machine learning and artificial intelligence could be used effectively in a business setting. So a challenge you have to be a very regular basis is being able to translate the mathematical solutions that we come up with into the business problem or that we're trying to tackle with algorithm. For example, examples we don't rate was two way were asked. Is that a exclusion for that? And we kind of dug into it. Identified that there was a problem with side related with customers buying the thing true, returning it and then, you know, buying the same shoe again in different size. And that's, you know, millions of dollars with lines so way both an algorithm to predict the constable styles in every shoe. How does that manifest is an experience on the website? How does that affect the actual return rate? How are we gonna build an environment around it to determine what the actual impact waas? All this becomes very important ingredient to communicate and whether it's in retail in the business setting, I study. I did machine learning. You know, clinical setting is well at Emory, so you know, it always maps back to problem. You're always gonna have a stakeholder that is expected to solve a problem where data And you're the particular that date that data and transported into inside trying to say that into an actionable solution that's gonna solve that problem each step of the way, you're gonna get a translator. I think that's critical.

What tools (software programs, frameworks, models, algorithms, languages) are typically used in a role like yours?

Summarized By: Jeff Musk on Fri Sep 11 2020
get up all of the research and in platform teams for all of that pose. So I am very rarely doing, you know, specific, obviously work. But I do get my hands dirty every now and then. But there's an assortment of different job functions that would go into a machine learning or machine intelligence organization in a company. And, uh, you could be a soft engineer. You could be a machine learning scientists, a research scientist, a data scientist, statistician, cloud architect of solutions, architect. And then there's a variety of different roles that go and do so exactly building an ML product, right? It's not just someone making an album. We need thio. Force the data we need Thio determine the actual impact off the algorithm in a customer setting. So in some product management, we need to deploy the outer than the sale on a Web site. The size examples with a lot of traffic. So there's a lot of nuances there, Uh, in general, you know, from a framework perspective, machine learning scientists will be using, you know, care us a pytorch or tensorflow or like very traditional neural network development frame books for Cyclone Apache ml spark ML for development in Like Jonah, a scholar for some of them or none. Deep learning approaches and models of the leverage from algorithmic perspective. I'm a strong believer on using the simplest model that gives you the solution you need. Right from linear regressions up to weigh have used generative adversarial neural networks to some of our problems. But there's no shame in where the spectrum your model YouTube lies. I think it's the matter. It doesn't get the job done. Uh, from a languages perspective, you know our statisticians that usually are our machine learning scientists panel and python scholar, our platform engineers, they usually working in Java and from, you know, a p I deployment perspective. Microsoft Safety out free food application models for live inference that deployed in AWS sentences like Sage Maker. It's a really broad, uh, interesting material, expanding a lot of teams, so you know, we could get into specific details if you'd like, but I think that's a broad overview. There

What are the job titles of people who someone in your role routinely works with, within and outside of the organization? What approaches are effective in working with them?

Summarized By: Jeff Musk on Fri Sep 11 2020
That's a really interesting and it's a great question because someone who's leading machine learning team, uh, you know, deals with the team that they are. The reports into them, which is in a sort of researchers, engineers and analysts usually and your engineers have broken up a bit more. Application developers and platform architect and data scientists and research scientist for model development on the research side, statisticians of data analyst for experimentation, framework, your A B testing and things like that. So when dealing with them, you realize that they all speak different deck language. They're not all the same type of technologists. So there's our translation going on across those job functions as well. There's a lot of recognition of everyone. There is a T shaped knowledge. They kind of know what everyone else is doing. But they're really good at their thing, and they're gonna want their thing when their methodology is to drive a lot of decisions, you need to balance that very efficiently. Uh, pushing out of the organization you'll deal with. You know, the CEO, your CMO, your CFO, just every one of the executive on their lieutenant directors, bps, etcetera, whoever it is that you're interfacing with for executing and delivery of a project very often in this kind of role. You know, you see a lot more invented structures where there's a lot of, uh, lateral integration with other teams where, AH, marketing directors welcome directly with an engineering Li like deploy solution, and it's it's good to break those silos because this is a service oriented industry. Uh, you're providing two kids, two people executing their ideas. I don't believe in the machine only product as a whole, so it's a very collaborative space.

How would you describe your management style? How has it evolved over the years? Can you tell about experiences or books that influenced your management style?

Summarized By: Jeff Musk on Fri Sep 11 2020
My management style has been something I've developed at Zappos predominantly, and I think is Apple's. Very early on. Got rid of managers were process called democracy, and we made it a hierarchy of purpose and rather than hierarchy of people. And I think that is really informed management style because, yes, I have the lead of the organization. But I see myself as a calling you compare and a friend and I, you know, I I focused very heavily on one on ones, and I think that I've learned over the years there is nothing more critical than a productive one on one in which your teammate you feel personally connected to you and be able thio talk to you about what's concerning them, what they like about their work or they don't like about the work, their gratification framework, what makes them look right. And if you can't write what he says on each 20 interact reports, I think you're not a good manager, and I think that is what inborn someone, everything totally, just like tell me everything about everyone in your team. My only question is gonna be well, how much time do you have? Because we're gonna take a while, right? And I think maintaining those complex relations with each member of the team is critical. And I think fostering a environment comfort do where they can, openly talking about what is troubling them and what is exciting. Them being able thio create a trust relationship or sometimes they're gonna be able to work on what's exciting to Sometimes you have to do what's right for the organization, but always we're collectively delivering on a shared vision that we all believe in that we all understand eyes, I think, what makes the team take examples? Without that, I think we would have founded a long time ago because it za top fields an emerging field. We need to have a strong belief system and what we're building so that we can sell it. The others Uh, yeah,

How do you manage conflicts within and across teams? How do you promote trust, openness and a healthy work culture? Sharing stories will greatly help.

Summarized By: Jeff Musk on Fri Sep 11 2020
So you know, this is something I've been very lucky with is that Zappos itself is the organization with a very rich community in it's rooted in our core values. Off open and honest communication is one of our core values. Building a healthy and family spirit is one of our core values, right? So it's bred into us. We hire and fire by those core values. So it's bred into us. Do you know, maintain the open relationships so that being said, conflict does arise. And I think the the best way to do that deal with it and in my opinion is as a leader, making sure that you completely remove yourself from having kind of opinion in a way and getting the right people in the room, getting them to have a conversation, understanding each side of the story, and and kind of being able to speak intelligently than what the point of conflict is on over here. I'm talking purely united technical contracts, right? You're dealing with job functions in this space that a statistician is gonna wanna do One thing an analyst is gonna do one thing a deep learning fashion or do one thing and you need to be an expert in all areas to be able to communicate with them in a way where they believe that you know what you're talking about. So I think faking it is probably the biggest mistake you could make as a leader while trying to manage a conflict. If you don't know, you don't know, learn with them, and the conflict often result bearing open conversation.

How can one get better recognition of work from one's boss and higher management? What mistakes should one avoid? Stories or examples will be quite helpful.

Summarized By: Jeff Musk on Fri Sep 11 2020
So I think recognition is very free game right at in corporations, and it very often becomes the motivating factor for why you do look, and I think that's the beginning of the end, right when you feel like, Oh, the CEO needs to know I would have done that for every one of the companies should know I would have done that or when somebody mentioned my project but doesn't mention my name. That's a slight right. I feel like that is the start of the end of not really getting the good type of recognition, A strong belief that I have and something that helped me examples a lot was I never tried to take credit for the work I did. I never tried to put my name and a tactical projects that we worked on. It kind of happened organically. We reached the point. We have anything data related happen with. Someone would assume I had something to do for it. There's a danger there where the good and the bad getting community, whether you're doing the work or not. But that's fine because you own a principal. You'll be your own area, and I think that trickles down to the beam, right? If it's the search algorithm, it doesn't matter who worked on. You know, I have account made a person accountable for owning the search algorithm. And, you know, if you're if you're not that person, that they've come to you for help do the work, knowing that we're going to get the credit because it's their accountability. They came to you for help because they knew you were the right person for the job and our leader of the circle. And she's a actually brilliant machine learning scientist. You know, one of my men was somewhat I learned a lot from, uh, and you know, she will never let me know whether you work in al algorithm or someone that looked in the algorithm. It doesn't matter because it's a function of it's a hierarchy of book hierarchy of purpose. If the purpose is being met, the recognition will follow is and I think the biggest mistake that you could make is letting your gratification framework or whether you are happy or not in a job, begged on how many times your name is mentioned alongside a project. I think that set you up for. We're not failure. Some people succeed that way, and that's fine. It does set you up for a lot of disappointment, and it's unnecessary disappoint.

What indicators are used to track performance in a job like yours? Think of the indicators such as key performance indicators (KPIs), objectives & key results (OKRs), or so on.

Summarized By: Jeff Musk on Fri Sep 11 2020
I'll speak of this from the perspective of my job rather than the people on my team is we have, ah, concept of P zero metrics effectively, which are the core. Yeah, you need to move right. So we do a lot of baby testing a lot of experimentation. And that's a total for us to understand. What is, uh, moving and what is was docked right? And there's this concept of a star goal that we use very often of my team, which is also another star. A smart goal, a smart goal. Yeah, a smart goal is one is, you know, a smart goal. But also it's an acronym that tells you it's a goal that is consumable right, that other people can understand what we're working towards. It's always advisable, not really going to something that's very abstract, something that only you and the experts understand, because then it's hard. Thio shows success so smart. Goal is specific. It's it's measurable. It's achievable, relevant and dime bound, right? So what that gives you is something that is I'm going to increase this number by this much in this much shine, and it is beautifully achievable. No one thinks you're just blowing smoke and then you work towards that. You quarrel. A court approved that you're moving in forward by the amount that you thought you would. And I like setting expectations high and missing my a little bit. And that's fine. You know, my my my leader now understands that that's my method. It allows us to push towards the end of the quarter on, uh, I am, like, four of those goals of those goals that I'm accountable for three of them or customer experience facing three are more infrastructure and innovation specific and, you know, the most important part of my job. I think they're sitting down and making show that the leader of the company understands those goals and what they need to be, and they mean to him. And however, if you deliver on them together, okay?

What skills and qualities do you look for while hiring? What kind of questions do you typically ask from candidates?

Summarized By: Jeff Musk on Fri Sep 11 2020
it's a mixed bag. I think it depends on the project that I'm that I have in mind usually, and then the rest of it, we kind of learned the skills on the job. It's a matter of I think it's important to have a strong foundation knowing the basics of machine learning, the algorithms that get into if you're going, she's learning job. You're going for engineering job, having, uh, fundamentals of engineering, you know, and know exactly what you're gonna do. So do preliminary questions, technical form screens on those. And we do a kind of cultural fit assessment, like with the violent company worked with you. And then finally, I personally, I'm looking for this part disability to think out of the box. So ah, lot posed you with business problems and then ask you for analogy is simpler analogy that represent the business problem because that's an indication of Are you gonna be able to translate something you learned in a textbook into solution actually addresses the problem that we're trying to tackle a team early in your area? E think as the road is more seen, that becomes the most important. The technical ability is kind of a binary yes or no? Right beyond that, I think it becomes a function of maybe you don't need to be the best engineer on the road, right? But if you can think out of the box and intuitively how to use the fields you have, they're they're more valuables. I will think of it as, uh, much rather have an average set of tools with the world's best carpenter than the world's best tools with an average carpenter is kind of way I think about it.

Can you discuss career accomplishment(s) that you feel good about? Please discuss the problem context, your solution, and the impact you made.

Summarized By: Jeff Musk on Fri Sep 11 2020
projects. I think that come to mind here. One is, uh when we found out that the return rate was too high. Examples. We build a cross functional team that determined that there was an issue with sizing related returns. And we and this is an example of a smart goal were Don't wait, don't it? And deciding related predominates too high. We committed to reducing it with the machine that could predict customer sizes. He built an infrastructure deployed between clean experimentation framework to evaluate the impact. And then we largely experience the Dean Brew as we were doing this work. Do be able to support it. And when we launched an experiment and we launched that experience, we saw that there was a 10% reduction in size in military terms. Whenever we made a prediction to a customer. So, uh, that was fantastic. That was a tangible achievement. Was something that we could make a tax on on the team. Afraid a team called very bad because we used to customer pinpoint using the skills they've been learning, you know, their whole Koreans. And that sounds good all around

How did the school prepare you for your career? Think about faculty, resources, alumni, exposure & networking. What were the best parts in each of your college programs?

Based on experience at: Master of Science (M.Sc.), Computer Science with a concentration in Computational Life Sciences, Emory University
Summarized By: Jeff Musk on Fri Sep 11 2020
important part. Uh, my undergraduate education was the opportunities numerous professors gave me to do undergraduate research. Uh, and you know, I very much value that even today, because it taught me how to think, you know, with a scientific perspective, with the field, for experimentation and with keeping objectives in mind. Particularly Doctor that the president of the university when I was an undergrad there and he taught a class and I took that class and he didn't undergraduate research project with us in that class. And as an underground, I learned how to think. I learned how to research. He then, er, just write a book for which we then got published in the National Peer Review Conference, which at 17 was the most unbelievable thing to be able to do. And it was so exciting. And that opportunity kind of planted in me this hunger to keep learning and keep it on and then led to my career of the monster's Ph. D. Suit at Emory, where I continue to research on Dwork with into this very faculty do guy and solve problems. So the cross it will grade the fact that you were great at both universities, but the research component of my education really defined the way I work today.

What three life lessons have you learned over your career? Please discuss the stories behind these lessons, if possible. Stories could be yours or observed.

Summarized By: Jeff Musk on Fri Sep 11 2020
life has and I've learned over my career and life that matters. If something is both exciting and scary and you want to do it, just do it right. And I that included I think I was 15 years old when I dropped out of high school, left my parents and moved to America to draw in college. Right, And that was terrifying for me and for them. But, you know, we did it. And I was three years into my PhD when Zappos wanted to recruit me and I was on the verge of standing right. My dissertation, you know, finish my coursework. Gonna start researching and finishing that writing my dissertation And this opportunity came along and I dropped out and I did it. And it was great, right? So trust your gut and A so long as it doesn't feel wrong as long as it's exciting, it feels like the right thing to do and, you know, scaring you a little bit, probably with it. So take those leaps of faith, Another life lesson. And I call this a life lesson because my life is data and I think it will be indefinitely. Eyes never, uh, faking data. It's just always such a bad idea. And a perfect example is, uh, you know, one time presented a a number, it wasn't too much and had it validated. And two days that number comes back to haunt me from time to time with the Post. I gave it because trust is everything in this space. And just if you don't know before, you don't know, do not try to make things up because the second someone finds out otherwise and realizes you just made it up. Everything you ever say is in question from their own fort and the last life. Last minute, I would say that is kind of a cornerstone of my leadership style is always hire people that are smarter than you always hire people that you know can replace you someday. And that's very contrary to what very un intuitive, because it's like, Well, what about job security? Your success is a leader is gonna be begged in the work that you're able to do with the people. You surround yourself, apply and you're gonna leave someday. And when we leave your feet style, you know otherwise you really didn't achievement so always higher always hire people smarter than you ever make a decision to hire someone that you're better than for the state of protecting your own security, job security and things like that. So, you know, I surround myself with people who are better. One thing that I e. That's big critical.

What starting job (after internship) would you recommend to students who hope to grow professionally like you? What other parting advice, dos, and don'ts would you give?

Summarized By: Jeff Musk on Fri Sep 11 2020
internship adds apples on their analytics team. So you know it. It went very well on I ended up leading the organization over the last couple of years, so but my advice would be is and this is postal advice. It's what I did. Just never let it stop being funny. Always chase after the job that you're gonna enjoy, uh, chase after the internships and jobs that they're gonna seem because you're gonna achieve more. It's fun. You're gonna be more productive if it's fun. And with that becomes success in which success will come money and title and progression and whatever, Right? Don't stop having fun chasing money and type this and this and that, Uh, and I can't stress that enough like it seems like such one of those, like, motivational kind, you know? But it's the most important thing. I think the second it stops being fun, you're not performing the best of your capabilities, and you're not achieving where you could be achieving