Instructure Manager, Data Engineering
University of Utah Bachelor of Science (B.S.), Information Systems and Computer Science (minor)
Current Time 0:00
/
Duration Time -:-
Progress: NaN%

How did you get to where you are today? What is your story? What incidents and experiences shaped your career path?

Summarized By: Jeff Musk on Wed Feb 12 2020
Okay, So, uh, when I was in college, I, ah, I needed some kind of a job to just help me pay for school and life. And and so I was willing to take just about anything, and I ended up finding a job at a company called Fusion IO. They make, uh, he's really fast PC. I express, uh, storage drives for servers, and the job I landed with them was a quality assurance technician. And what that meant is basically that I would have to look at these PC I express card to make sure they had all their stickers and screws and and everything like that before they were boxed up and sent to customers. And there's a terrible job. I hated it. But, um, it got me in a position where I could, uh, where I saw so awesome problems. And, hey, I have the skill to be ableto make a difference here. The problem was, is that we were keeping track of all these inspections just with excess spreadsheets so divided up by workwear and say All right, the serial number. Yes, it's got out. Screws. Yes, it's got stickers. Check, check, check. Um, in the It's just so hard to maintain that if we ever wanted to go back and find, ah, the inspection for a specific serial number or oh, are you know, if one of us happen to be out that day and they needed instruction data, there was just no way Thio to get what we need It and I was in school for computer science of the time that, you know, I could build something that would solve this problem. Uh, in my boss at the time, encouraged me to do that. And so I just started building this little website. They used PHP and, uh, my sequel back end to keep track of these, uh, inspections. And it grew over time. And I, uh, you know, version two point. Oh, I've built lift by phone and Django. And anyway, it got me into this world of collecting and managing data. Uh, and from there, um, that that team was was within the operations group at Fusion Io and in the operations department had a lot of needs around, uh, analytics and reporting and things like that. And so they started forming this team and it worked out that I was able to join the team is kind of a junior, huh? I think that I think my title of systems engineer, but it was just kind of a Hey, we're going to give you these little things here and there, and we'll see how you do with them and and go from then. And our task was to build a data warehouse from the ground up and start reporting on these things and yeah, that the company needed. And and that was kind of my first exposure to the world of analytics. And from there I moved over to my current company and structure just through Ah, that former co worker. And, um, we did the same thing over here. We built the data warehouse from the ground up and and as it's grown by my role and responsibility, has grown with it. Uh, so at this point, I'm now the manager of Data Engineering here in structure that started out with with pretty humble beginnings, but it worked out okay. Huh? Worked out okay for me

What are the responsibilities and decisions that you handle at work? Discuss weekly hours you spend in the office, for work travel, and working from home.

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
basically two different categories that my responsibilities fit into. As faras the technical side goes, Number one is servicing the analytics needs. So we have a number of of analysts who need data to be able to write reports. And so my job is to go and get that data for them and put it into assistant where they can access it. Um and so So the managing the data warehouses is the big responsibility there. On the other side of that is we have a number of different systems. Uh, that we use for enterprise applications are C R M system and our air P system and a number of others. And but we have a lot of needs in terms of moving data back and forth between these different systems. And so, uh, the system to system integrations are a big part of what I do is well, and that's the technical side. We'll manage your side eyes more. Just typical managerial. Um, you know, hiring and recruiting and working with the people and managing the workload for the team and all those kinds of things. Uh, my Let's see, my weekly hours. I'm pretty lucky I'm able to work just about 40 hours a week, and occasionally there will be things that need to be done outside of business hours or or things like that. But for the most part, it's It's a really stable 9 to 5 type schedule. Um, I do have the option to work from home if needed. I got little kids at home, so I don't do that as often as I would like to, just cause they're they have a hard time with that. But, uh, working from home is definitely an option. And then, uh, there is a little bit of travel involved. I typically go to one or more training conferences every year just to stay abreast of all the new technologies and and options that we've got out there. And we also, uh, we have, ah, an office that were that we have in Budapest and we've started hiring a few engineers over in Budapest as well. So I've been over there, too, to meet with them and work with them and get to know them a little better. So, uh, that that the traveling and uh, outside of irregular hours is kind of a rare thing for the most part, it's a really stable, uh, typical work hours, and I don't have toe to do crazy hours or anything like that.

What tools (software programs, frameworks, models, algorithms, languages) do you use at work? Do you prefer certain tools more than the others? Why?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
but I'll try to keep this brief. So, uh, way have, ah, ton of tools that we use for Well, starting in the middle and kind of work our way out from there. So, uh, in the middle, we have our data warehouse, and at the moment, that is a tool called Snowflake. And it sits on top of, uh, Amazons s3 offering, which is just storage cloud storage. And and it allows you to put files out. There are s three, and then it can connect to those files and read them. And you can just write simple sequel queries to access the day that you put out there and you can stay locked down the amount of computer you use and you pay according to the size of the the they call it the virtual warehouse you pay according to the side of the machine Europe you're using for the queries and how long it runs and those types of things. So snowflake is the heart of it. Um, prior to snowflake, we were using Amazon Red Shift, which is a a cloud data warehouse offering. Um, and that worked well for us, But we, uh, kind of push that to its limits. And we're looking for something that performed a little better and and the snowflake fit the bill in a number of different ways. First of all, it's, uh, it's decoupled the storage from the compute power so we can store as much as we want, um, and not have to pay extra for compute, for example, with red shift, Um, you buy a node at a time and that note comes with, let's say, two terabytes of storage. And if you feel enough, you have to buy another note, and that comes with more RAM and more compute and everything. So, um, by decoupling those you can kind of control your costs a little bit better. Um, so it's been advantageous that way, and then the other thing that's pretty good about it is, uh, it's really performance and really meets our needs a little bit better than then Red shifted. So that's kind of the heart of of our systems. And then we've got tools to get going in and tools to get data out of that. So the main tool that we use the living language we used to get data in is Python's. We have a whole bunch of python scripts that air connecting to various SAPI eyes and databases and on, uh, systems. And we pulled it out of those systems and load it into snowflake from there. Um, and from there, we can model the data and do whatever we need. Thio manipulate that to get what we need for the reports. Uh, and for the reporting, we use a tool called tableaux, which is a visualization tool, and and you can, you know, bringing your data and make pretty charts and graphs out of it and and go from there. Um, that's kind of the heart of it. But we have a whole bunch of other like, uh, ancillary tools that we use as well for different use cases. Um, for example, that we have certain data sets that are large enough that they they'll bring down the server if we try to just run a simple python job with it. So, for example, huh? The structure makes canvas and and the web log data for canvases just massive. Uh, it's something on the order of 2 to 3 billion rows per month, and we haven't. We've go back clear toe 2010 or whatever, and just have so much data there, that's it's hard to manage. And so, um, we're using a tool called airflow, which allows us to kind of work a straight and, ah, uh, sparked cluster out on Amazon has ah, on offering called E M R. The last graf produce. Um, so this airflow tool allows us to write that python ish script. Um, and then it, you know, basically spin up that cluster the spark cluster and run the job through that and dump all the data out toe Amazon s3. And from there were able to copy it into snowflake. Um, aws stems on Web service is is a big part of what we do. So we're using Sam, we're using Lambda, which is Ah, a tour where you can essentially right a little script and and it will run basically on whatever trigger you define it. And you don't have to worry about, you know, how much memory do I need? How much storage do any done this machine? How much? Ram? Um, I'm gonna have to buy it. Just you just write your code and put it out there and it figures out the rest, Eh? So it's really nice that for the system, the system integrations. We have a tool that we use called Del Bumi, and it's kind of a simple dragon dropped tool where you say All right, I got this source system over here. Here's the the query or whatever to get the data that I need out of that system and that I can manipulate the data from there and then connect to a destination system. And it manages all the connections and data and everything like that. Um, and that's, you know, we probably could do, uh, pipeline or something similar, like to that for those types of jobs as well. But we've settled on Bumi in most of those cases because it it makes more sense, and it's a little easier to maintain and things like that. So lots of different, um, options and and, uh, tools that we're using their and it keeps it. It gives it a lot of variety and keeps it interesting to get into those types of different systems and figure. Choosing the best tool for the the job has been one of the more rewarding parts because I'm not shoehorned into Well, we're python shop, so we have to do it. You know, this is the python you to do is we have to do it that way. It's it's really what's the best tool for the circumstance, and then we go and use that tool. That's been a really nice thing about working here. It's called El Bumi.Uh huh.

What things do you like about your job? Were there any pleasant surprises?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
really like. Ah, uh, tech companies in general seemed to be really good to work for. They typically, at least in my experience, they, um in order to recruit engineers and these highly skilled people, they have to offer, uh, you know, good benefits and make it a cool place to work. And and they pay well also. And that's been really nice. Could be a part of that, Um and so there's a lot of good things about working here. Um, but I think in terms of the work in terms of what I'm doing, one of the one of them better things about it has been the flexibility to, um to build things how I think it's best to build them on. Like I said before, it's not stuck using this certain tool for everything that I do. I have the freedom to get out and and choose the best tool for the job and that it's been really nice. And, um, I've grown a lot because of that, you know, For example, when I was when I first started, I you know, I was fairly experienced with python and e T. L Work and and Billy are the data warehouse. But the system, the system stuff was all brand new to me. And then my boss said A we we have this need and I'd love for you to step in and fill it. We'll tell you, figure out the best way to do it. So a lot of research on different platforms and and do some trials with different companies and figure out OK, I think you know, this was gonna be the best for our needs and and let's go, go from there. And so that's been really good. And I've been able to grow, uh, in my technical abilities. In my scope of responsibilities as well. Along with the company is something he's grown and the need to have gotten a little different.

What are the job titles of people you routinely work with inside and outside of your organization? What approaches do you find to be effective in working with them?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
work inside the Enterprise Systems group. Um, so I report up through the vice president of Enterprise Systems and Predator that I reported to the director of Analytics. And, um, you know, I'm a manager of data engineering and have data engineers that work for me. Um, also within our group. We have Salesforce administrators, and we have a r p And listen and, ah, you know, different different titles for the different systems and groups that we work with. Ah, within the company re, um, we work closely with all right kind of the this more senior level of people. So we're writing reports and getting data for, say, that the vice president of sales or his year. He's got, like, a, uh I guess revenue. I forget. I forget the title. But the a revenue officer who's looking at, um, you know, the the size of the deals and there that quantity easing in pricing and things like that. So, um and then we'll work with, you know, the vice president of customer success. And in their efforts to try to, um, you know, look for ways to improve their service to the customers and we work with um, the the vice president of Engineering to measure engineering performance and, you know, a number of buds and then your tickets and those types of things. So, um well, look, we're working with these other other business owners that work for the other different parts of the business and and, uh, in the system, the system integration work that I do. It's often with, uh, the son of the lower level individuals as well. So, for example, I work directly with the revenue accounting, um, in, you know, tweaking the data that's moving from Salesforce into our Air P system so that it's got it's able to be sliced and diced the way he needed to be, um, for his processes and Ana and then outside of the company, um, I work with vendors with sales consultants and with, um, sales reps looking at different technologies of the software. And, uh, I think that's probably about it outside the company. I do have one kind of customer and chasing project that I am involved with, but, um, for the most part, I'm not dealing directly with with any customers. It's mostly been with with representatives from other companies, so for example, we use Snowflake is I Data Warehouse. And so I work closely with our solutions consultant that snowflake, too solve problems and and you know their customers. Success. Tina's oil has helped me through some of the things that we've gotten stuck on and and those types of things.

What major challenges do you face in your job and how do you handle them? Can you discuss a few accomplishments?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
strengths of the systems that were using. So an example of that is, um, you know, a particular system. Here's an example. So we use next week is our e r P system and they that sweet has an A p. I, um That's very particular about the way that you communicate with it. And, um And so, for example, are pulling data out of another system and trying to push it into net sweet. And it has to be, um, you know, very particularly transformed and integrated with that sweet and, um, you know, being able to figure out Okay, this is this is how we structure the data. This is how we manipulate the data so that it managed so that it can communicate with that system can be a challenge sometimes. And then there are other other examples. So, for you, another one of our dancers is is, um it's a no sequel, um, document database. And we're trying to bring that data into our data warehouse, which is a, you know, rows and columns table and the, you know, just is different the way those things are stored. And so, um, kind of thinking through the the engineering of those, um, you know, working their way through those types of challenges is kind of kind of different. I also, as the manager, I face challenges, um, in terms of hiring and finding people and, um, you know, dealing with nutrition and and different things like that. So, um, you know, managing expectations of, you know, say we're building this project for someone else in the company and and then managing the expectations of one lakh project will be delivered versus the research is that I have to get it done and amount of time and things like that on those are some of the challenges that I that I deal with, um, terms of accomplishments. Uh, we've had a lot. Um, you know, they somebody needs this specific thing, and we're able to go get the data building report and deliver it to him on that. That happens really regularly, but I'd say the big accomplishment that's been, uh, most recent is moving from Amazon. Red shift to snowflake was a really big job. Um, hey, joke all the time that it's comparable toe the heart transplant because you've got so many, uh, so many different puck's coming into into and out of that database and, you know, hundreds and hundreds and hundreds of processes there are are connecting to that data warehouse every day. And, um, you know, managing the shift from one to the other has been a big job, and so I'm proud of the way that's turning out.

What qualities does your team look for while hiring? What kind of questions does your team typically ask from candidates?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
ability. Big part of what we evaluate. We're hiring people. Um, you know, typically, if I'm looking for a data engineer, I want somebody who's built a python has been a sequel. Who knows the ins and outs of databases and how to connect with them and how to work with him. And, uh, so technical ability is a big part of what we're looking for. But we also need more of the soft skills. I need somebody who's able to look at some code and reverse engineer it and understand how it's how it's how it works and where did where it could be breaking and how to optimize and things like that. And I need people who can work with these other groups, these other people around the company. Um, so you know, you're really good engineer doesn't necessarily mean you able to work with people somebody for, for somebody who can kind of do both, consider both through that fence. Um, we don't have a standard set of questions or anything that you ask, but that when we bring people into interview, then we're always looking, um, technical ability. And if if you're a good fit with the team with the culture of the company with, you know, are you gonna be a good co worker is important to us, and and, uh, we're looking for people who are self motivated and driven and and looking to excel and do well in their careers and so on. The questions we ask that will help us try to answer some of those questions and and just help us get to know you and help us understand what kind of oppression here. And if we think you'd, um, be the type of person who could just dive in and and and immediately start contributing and helping our team with what we need to do.

What was the hiring process like for your job? What were the roles of people who interviewed you? What kind of questions were asked?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
uh, I was brought over here by a co worker, uh, that I worked with before. And so I already knew him and had a relationship with him. And he was like, he was the director of analytics here at the time. And so, um, so I interviewed with with him and then the rest of her team, and so we had I forget the titles, but a handful of analysts and, um, and just system administrators at the time. And so they brought me in. We had a conversation in the conference room and, um, same kind of thing that we do when I interviewed Narrow. We just try to get to know the person and and get an understanding of their technical abilities. And if they seem they're gonna be a good cultural fit with our team in our company and if they would be a good, um contributor, right from the get go. And so that's essentially what I went through when I was hired six years ago, and it hasn't changed a whole lot. Different groups in the company, uh, have different ways of hiring some, you know, in some cases, uh, when we've had more specific requirements. Uh, or in one case, we had a couple of really good candidates that we're having a hard time choosing between. And so, um, we, uh, devised the test, so he was essentially a sequel problem for them to go figure out. So we gave him. Here's your table's Here's your, uh, you know, the what the scheme of the database looks like. And we need you to build a report that shows this and on then had each of them right, the the sequel to do it, and that helped us make our decisions. So I think that's a little more common in our engineering group. But we've used that a little bit in our, uh, enterprise systems group as well. It's a good question.

What are some future career path(s) for you? What skills, certificates, or experiences do you plan on acquiring?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
we're going to become a or director of analytics or, um, you know, keep going up from their vice president of of, uh, type group were, uh, you know, even even CEO could could be on the path. Or I could stay more technical and and become more of a senior engineer SR architect of you know, of these enterprise systems that we're working with that So there's a couple of different past there. And to be honest, I'm not sure which one I am interested in going down. It's kind of hard to choose between the technical side and the managing side because both could be a really good past. But, um, yes, I guess if I was interested in going down the technical path, I'd be much more interested in the technical skills. So, you know, always keeping abreast of new technologies and new ways of solving problems and getting really good at that, going really deep with my my knowledge of those types of systems, and that would be really good. And then going the opposite direction is where you go. The manager route, then I wouldn't need to be, is deeply invested in those technologies, but I would still need to be aware of what was out there to be involved in choosing, Uh, you know this We're gonna go with this system versus this system. And here's the president, Haas, and why it matters. And, uh, may not be as involved in implementing or developing on those different systems, but, um, definitely involved in the choices. So lots of different directions if you go. But, um, I know their certificates for, uh, Amazon Web service is and and really probably every every platform Or, um, you know, every every different kind of group of products that you could be developing on would have some kind of a certificate that you can, um and then just, you know, putting the time for experiences valuable as well. Like I said that I've kind of been able to grow with the company and expand my skill set and the number of tools that are familiar with and and things like that. Um, so yeah, so I mean, like I said, mine was quality

What are different entry-level jobs and subsequent job pathways that can lead students to a position such as yours?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
for from most people to take. But it was when it worked for me. Um, we occasionally will hired interns that air, um, or or like a really junior analyst that's just getting their feet wet. Just graduated or just starting out in this data space. So, you know, they'd still be learning sequel. Or they'd still be learning python or something like that. And and, uh, so have a junior, junior analyst or junior engineer. Junior data engineer. Now could be a pathway as well. And thank you. In other. In other cases we've got, um I think probably the main, um, open door here that in structure would be a customer support Analysts were constantly hiring those and and it's really a junior type position where you're on the phone is talking with customers and helping and work through problems. But from there we've had many, many, many people who I haven't been able to excel and shine in that rolling and move into other roles from there. Uh so, for example, um, you know, if you if you're able to really dive in and and help customers solve these problems, they have different tiers of customer supports of the level one time of the easy stuff and level two and level three gets more technical and a few egg salad in a level three roll. At that point, you're qualified to be an engineer building Jan Vester building some of these other products that we have. Um, because if you're able to get in and solve those really technical problems, I mean, you understand how the code is working and you're able to get in and look at the code and saying, You know, based on these conditions, I can see this is why it would fail here and and recommend solution and things like that. So, uh, not to be an avenue as well, um, to a position like this.

What were the responsibilities and decisions that you handled at work? What major challenges did you face in your job?

Based on experience at: Enterprise Analytics Developer, SanDisk®
Summarized By: Jeff Musk on Wed Feb 12 2020
acquired cretin I owe. So the responsibilities there were similar Er I mean, like I said, I started out as the quality assurance technician and then moved into a systems engineer role. And I think when I left my my title was Enterprise Analytics developed her, um, so really similar responsibilities. I was building the data warehouse and the e t l jobs that were bringing data into that warehouse and the reports that we're taking data out, Um, and you know, similar challenges. We were We were, you know, technical things. At that point, I didn't I wasn't in the manual. It also. It was technical challenges that I dealt with, and it was, you know, challenges having to get the data into a certain format or or, you know, things like that. Some systems can be really hard to work with, but for the most part, it was fairly straightforward. Yeah. So

How did the program prepare you for your career? Think about faculty, resources, alumni, exposure & networking. What were the best parts?

Based on experience at: Bachelor of Science (B.S.), Information Systems and Computer Science (minor), University of Utah
Summarized By: Jeff Musk on Wed Feb 12 2020
is my major. That was what I thought I was interested in. And as I got deeper and deeper and deeper into the program, but I realized it is, I wasn't sure it was quite the direction I wanted to go on. And so I finished enough of the classes toe to get a minor in computer science. And then I made the jump over to information systems in the business school. And I remember, um, in one of my very first class is one of my professors described it as, um, you know, computer scientists are the ones who are writing, writing the code and then information systems is they're the ones who are figuring figuring out how to make money with what the computer science people are doing on that 10 resonated with me and and so having the business side of it was interesting to me as well. So, um, so computer science gave me telling a foundation of coding and scripting, and and this is how you know, to think logically the way a computer does. And then and then information systems kind of widened that knowledge, and I was able to learn more about databases and websites and different systems and then all of the business side as well. So, uh, like I mentioned, I was in the operations group over at Fusion Io and there, looking at logistics and and, uh, optimizing production and different things like that. And that was some of the things that I studied in in school for any information systems. And it was really interesting to me. And but the thing that's prepared me most is, uh, just having that that general understanding of of using computers to accomplish business, um, business tash business problems, uh, so being ableto two, right, a programmer, a script or a job or whatever, that can solve this business process or assist or make it easier, or for, ah, make it better. That's been what's really valuable to me in terms of learning and and, uh, all that kind of stuff. Um, yeah.

Would you like to share something that is not on your resume? This may include your passions, facing setbacks or adversities, a unique experience, or an unexpected help.

Summarized By: Jeff Musk on Wed Feb 12 2020
Yeah. Um

Do you have any parting advice for students and professionals hoping to get to a position such as yours? What 3 dos and 3 don'ts would you suggest?

Based on experience at: Manager, Data Engineering, Instructure
Summarized By: Jeff Musk on Wed Feb 12 2020
technical bill. Technical ability mixed with, um, with drive and passion and an ability Thio talk to people and work with our customers, which are just just within the company. That's what I mean by festivals. But I'm so it would be great to find somebody who was really, really good in terms of writing, sequel or writing python. Er, we're working with databases or something like that, but also finding somebody who who just eyes ableto look at a piece of code and reverse engineer it and think of, you know, this is how it's working. This is how I could get and then dive in from there and start solving the problems that we face is a group. Andi would be really, really helpful. And and so I guess three DUIs is is Get out there and and learn and figure out how to work with with, uh, some of these platforms and tools and languages and different things like that. You know, I'd love to find somebody who new python really well and knew howto set up a post grass database in and connect to it and start inserting data and pulling it out. And I could write sequel against it and and, uh, you know, build a report that way. I'd love to be able to find someone like that. Um, but the technical ability isn't everything. I also need somebody who can you think logically and and, uh, worked with people and, you know, say, you know, go talk to somebody in the business and figure out what the requirements are for this project and and, you know, if they if they're saying I need to be able to see, you know, revenue by, you know, whatever the requirement is being able to translate what they say to And, you know, I I need this this in this from X system. You know, any of these three tables and that'll be able to give me the data that I need to thio write that report for that person over or something like that. So and it's not just the technical side, that's a big piece of it, but also the the working with people side is important. And being able to kind of translate between the business needs and the technical needs is really kind of a sweet spot there. Good