This interview question is designed to get signals on how good you are at applying ML/AI to real world applications. Valerii: To be honest, I have made no preparation for ML system design. I was showing that part, because that's the only thing I can do, probably [laughs] design a system on paper. So, extensive experience. There are talks about that. I don't know to be honest. It's hard for me to answer because I made no preparation myself for that. ( 35:21) Valerii: Well, maybe. On the web, there are some analysis and design overviews on YouTube. I've done my fair share. However, they’re in Russian, so only people who speak Russian or understand Russian can do that. But there is information. Look, the process to get hired at Facebook is standardized. Also, you can have extensive experience. ( 35:21) Alexey: [laughs] That’s cool. The question is about the book you mentioned – the book was Machine Learning Design Patterns, right? ( 52:56) Valerii: I was. I don't know. I am trying to take a look now because there is an exponential decay if you don't compete, and what is even more important, if you don't win, your score is decaying. Kaggle is an addiction. So the best way is not to go there because you can suddenly find yourself doing the Kaggle again. ( 2:32)

Valerii: If you expect that data scientists have some relations to machine learning in these companies, you are mistaken. People who do machine learning are called machine learning engineers. Right? [laughs] And these people have to pass through the software engineer loop at Facebook, and some additional rounds of interviews. For machine learning, and again, for a software engineer, there are different stages, but there are, I would say, a couple of interviews that are very important in terms of assessing your level. ( 9:36) Valerii: I said that I need a loss, which comes from the family of the proper scoring functions. ( 24:52) Kmeans. Try to implement Kmeans from scratch sample code from flothesof.github.io. Bonus point for vectorized version in numpy + completed in 20 minutes. Follow-up with worst case time complexity and improvement for initialization. Gradient boosted trees: Better performance than logistic regressions, can find non-linear interactions, typically doesn’t require much tuning. Valerii: What else? Should we take a look into other metrics? Probably, yes. But we know that the fraud is very class- balance skewed. We know that class imbalance is extremely high there. We also know that it might change. So that means that if we would like to take a look into the metrics, these metrics have to be class-balance insensitive, probably. Because otherwise, yes, class balance changed, metrics change, but the model’s the same. Okay, so what are the most favorite metrics? Is it precision and recall? Recall is class-balance insensitive, while precision is class-balance sensitive. So, forget about precision. Can we replace precision with something? Why not specificity? Also not that. Okay, something else? Maybe. We know that there are some thresholds of expected fraud level, which we can just go with and then we can. ( 16:43)Valerii: You're accounting for the software engineer plus machine learning. So yes, the company needs machine learning engineers. Then again, what was the first task for us? “Understand what we want to achieve.” As soon as you understand what you would like to achieve, it's much easier to achieve that. Without understanding, of course, randomly, you might achieve a desired goal, but the chances are not high. ( 51:20)

Full Book Name: System Design Interview – An insider’s guide Volume 1 And Volume 2 By Alex Xu (Set Of 2 Books) Alexey: So one way, of course, you do this at work. Another way is to find people who can help you with that. Is there anything else you can do? I don't know, maybe watching some conference talk maybe? ( 35:09) You can also make use of other creative data collection techniques. For example, you can build a personalized experience in your product by collecting data from users. If you’re working with a system that uses visual data, such as object detectors or image segmenters, you can use GANs (generative adversarial networks) to enhance the training data. Other things to consider include: Alexey: Yeah, indeed. So, the original question I actually asked you is about the difference between system design and machine learning system design and I think it's very clear what machine learning system design is. It requires some domain knowledge, to some extent, or making some assumptions. Then you need to walk through the process of solving a particular problem. ( 22:05)

DataTalks.Club

Don’t forget to bring up advanced issues specific to these models. Eg. For logistic regression you could talk about regularization with lasso or adding interaction features to deal with non-linearity. If your model training uses an optimizer, talk about the loss functions you can use. Talk about hyper-parameters and design choices for each model. This is a chance to show off your depth and go beyond the typical shallow ideas people can grok from a data science tutorial! Online Evaluation As I mentioned in my first article, I think of systems design questions as improv presentations. The interviewer gives you a task, you clarify it and then present a solution. It’s crucial to go into the interview with a game plan for discussing your design. There’s a lot of ground to cover in creating an ML system and you also need to show some real depth in a few areas. Here’s a flow for an ML question, in reality, it’s easy to blend these topics at any time or take a deep dive. Make sure you look for clues from the interviewer that they want to hear more about a topic, or if you’ve covered enough and you can move on. Don’t forget that you’re not only trying to please the interviewer, but the panel of people who will be reviewing your performance later. Don’t give up any opportunities to show your experience and skills. Remember, unlike other interview rounds, you’re driving this interview, and you want to show both technical knowledge and leadership skills. Product Objectives and Requirements Clarification

Given that Alice has 2 kids, at least one of which is a girl, what is the probability that both kids are girls? (credit swierdo) Alexey: I think there is an article, or more like a mini-book, from Google, which is called The Rules of Machine Learning and I think there the first rule is, “You don't need machine learning.” Or something like that. ( 52:32) The second part consists of over 200 knowledge questions, each noted with its level of difficulty -- interviews for more senior roles should expect harder questions -- that cover important concepts and common misconceptions in machine learning. The search engine for Data Science learning resources (FREE). Bookmark your favorite resources, mark articles as complete, and add study notes. https://aigents.co/learn Go over questions for requirements gathering. It’s easy to forget key questions when you’re nervous!Two of the crucial signals you need to provide at this interview are the ability to think of useful data to feed into your models and your knowledge about transforming raw signals into usable numeric features for your models. Here’s a hint, this is probably something you can think about ahead of time for your interview. For the company you’re interviewing at, think about the useful data sources and features you could use. At the same time, many models have thousands of inputs, so you can’t spend the whole interview cycling through this. You can split this up into a couple layers of abstraction. Data Sources Alexey: Yeah. So you need to say all these things and then once you say, “Okay, this is the thing we are measuring. This is the baseline model, (like linear regression or logistic regression).” And then you start building on top of that, right? ( 24:57)

For example, if you are performing binary classification, you will use the following offline metrics: Area Under Curve (AUC), log loss, precision, recall, and F1-score. April 29th: I launched mlengineer.io blog so you can get latest machine learning interview experience. Get Book Universal Access in Human-Computer Interaction. Design Methods and User Experience by Margherita Antona,Constantine Stephanidis Pdf This repo is written based on REAL interview questions from big companies and the study materials are based on legit experts i.e Andrew Ng, Yoshua Bengio etc.Alexey: To summarize – basically, machine learning system design is one of the steps that machine learning engineers have to go through when they interview at Facebook, (probably now I should call it Meta) Google, and similar companies. Machine learning engineers go through this interview and this is a way to assess how well they can design machine learning systems – these are the systems that have to do something with machine learning. Right? ( 12:50) Obviously there’s many more items here. Notice that the concepts are still vague, and would require clarification to actually use in a model. Eg. don’t just leave a feature as ‘history of items liked’, that’s not a numeric value you can train a model with. Feature Representation Alexey: Okay. Let's talk about machine learning system design. This is a part of the interview process and you said you did a lot of interviews as the interviewer. I imagine also, when you were joining Facebook before that, you also had to take this interview. So can you tell us about that? What is machine learning system design, and why is it an important step in the interview process? ( 9:12) After asking questions, you should carefully choose your system’s performance metrics for both online and offline testing. These metrics will differ depending on the problem your system is trying to solve. Some companies may not care at all about infrastructure for this interview, while others may actually combine ML with Distributed Systems. Make sure you’re clear on expectations for how much you should discuss the actual infrastructure for the interview. Even if infrastructure isn’t important, you should still keep in mind the limitations that modern computing imposes. No, you won’t be able to run a million high dimensional pictures through a Resnet model in real time. See the Infrastructure Components section below for some important ML infrastructure. Model Development