“The Coding Machine” at Meta – with Michael Novati

what does a coding machine mean it's very controversial because anyone can write a lot of code in theory a junior engineer could sit down and just press the keyboard all day and write a lot of code the zoomed out thing is impact so what is the code doing you know writing a lot of code all day long but is it moving projects forward is it helping launch a product that might need a team of 5 to 10 Engineers that just one engineer can do is it unblocking a lot of people by just pushing a lot of things forward is it refactoring a lot of code extremely quickly and fast so that new Engineers don't even have to deal with old libraries these are the type of examples that make a coding machine a coding machine I'm still a coding machine to this day at formation I'm writing code all day long I think I have 20 commits today already oh wow it is just a morning Michael naati spent eight years working at meta from 2009 to 2017 where in some years he was a number one code committer across the whole company he started as an insur and in just 6 years was promoted four times getting to the E7 level which is the equivalent of principal engineer level at other tech companies Michael is now the co-founder of formation our remote software engineering fellowship program in today's episode we go into what the coding machine archetype is as meta for stock plusus engineers and how Michael became the first ever coding machine at Facebook met engineering culture and why the company built so many custom Journal tools how the hiring committee works at meta and how they make hiring decisions and a lot more advice for doing well on Tech interviews and additional stories from Michael's time at meta if you enjoy the show please subscribe to the podcast on any podcast platform and on YouTube Michael welcome to the podcast hi I'm happy to be here thanks for inviting me so let's start with something really surprising and interesting about you at meta there's these things called archetypes uh they're used for performance reviews and promotions as I've learned when I researched the company things like fixer text lead and there's something called the coding machine which apparently came about because of you can you talk about that uh yeah sure I can talk a little bit about the coding machine um this was kind of back in maybe 20 2015 or so um so in general about the archetypes um there's kind of two purposes for them at meta overall they are kind of uh their fairness is really important at meta so the archetypes are kind of these um examples of what an engineer who's at the principal senior staff kind of level would look like and the archit types are modeled after existing Engineers who are there so they're kind of a a way for the company to kind of pattern match upcoming Engineers against these existing engineers and try to have a fair system for who's at this level and who's not um and um the reason why there's not just like one or two and there's like you know handful of them is that U meta kind of has this they don't use this analogy a lot internally but I've always found this a natural analogy is like the pro sports team analogy um when you have a professional sports team you have uh a bunch of baseball players say or basketball players and each person has a a different position that they play but all of those people at the pro level are pretty good they're all quite good at every position they can throw baseballs throw basketballs Etc but you know Steph Curry is the best three-point shooter um I think I don't know that much about basketball um and in the analogy of the archetypes there's similarly people have Specialties so everyone once you're at that level you can do all the basic things engineers do or you know on a daily basis but um there becomes these different Specialties where you're exceptional or one of the best in the company or even best in the industry at um so the coding machine is something that was it was invented for me um and the kind of I won't go into all the history of like the back and forths of how it happened but generally speaking there wasn't really a an archetype for what I was doing at uh already um so this was kind of a case where they had to sit down compare me to people who are already at these E7 plus levels um zoom out and focusing on impact that's like the kind of word you'll hear a lot when you talk to Facebook people about performance reviews is impact um and they try to say they they kind of noticed that overall Michael is impacting the company very similarly to other e7s but there isn't an archetype that he fits into I'm not a specialist or um um a fixer is a really good example like I don't write one line of code to save $10 million simple cut dry easy that's what fixer is yeah um I have a friend who's a fixer and you know he writes a few lines of code a year and they are extremely impactful because he's spending most of his time talking to people and deep diving into these like really weird intersections of issues um and that's pretty easy to to quantify it's easy to understand the impact on the business um so some of those archet are a little easier now coding machine it's a little harder it's very controversial because anyone can write a lot of code in theory like a junior engineer could sit down and just press the keyboard all day and write a lot of code so like what does a coding machine mean to be at this level um and the the zoomed out thing is impact so what is the code doing you know writing a lot of code all day long but um is it moving projects forward is is it um helping launch a product that might need a team of five to 10 Engineers that just one engineer can do um is it unblocking a lot of people by just pushing a lot of things forward um is it refactoring a lot of code extremely quickly and fast so that new Engineers don't even have to deal with old libraries um so there's there's kind of these are the these are the type of examples that make a kind of a CO a coding machine a coding machine now how they kind of created this there's maybe the story behind that is just a lot of back and forth it's a an executive and who really uh sponsored me um his name is Tom Allison and I think that he's I think he's head of Facebook quote unquote the product now um wow and or the app they call it I think and uh yeah he was really kind of championing me and working with the other vvps at the time to kind of connect the dots and make this happen um yeah and I'm still a coding machine to this day at formation I'm writing code all day long I think I have 20 commits today already um it's a busy day and and it's just the morning yeah just this morning uh yeah so I you know it's kind of it's um I would say like in terms of point back that that professional sports analogy um it's like that's kind of the thing that that's my thing that I'm the top 1% at on the professional sports team it's like we need to uh move this new prototype forward insanely quickly then I might be the go-to person or you need to refactor a lot of code really fast I'm the one person you would go to and the whole company to talk to about doing that and that's what kind of Justified me being at that level trust isn't just earned it's demanded whether you're stter founder navigating your first audit or seizing secur to professional skill in your governance risk and compliance program proving your commitment to security has never been more critical or more complex that's where vant comes in vant can help you start or scale your security program by connecting with auditors and experts to conduct your audit and set up your security program quickly plus with Automation and AI throughout the platform vanta gives your time back so you can focus on building your company businesses use V to establish trust by automating compliance needs across over 35 Frameworks like sock 2 and ISO 271 with vanta they centralize security workflows complete questioners up to five times faster and proactively manage vendor risk join over 9,000 global companies to manage risk and prove Security in real time for a limited time my listeners get $11,000 off vanta atv.com pragmatic that is v.com pragmatic for $1,000 off this episode is brought to you by work OS if you're building a SAS app at some point your customers will start asking for Enterprise features like s edication skim provisioning and fine graen authorization that's where work comes in making it fast and painless to add Enterprise features to your app their apis are easy to understand and you can ship quickly and get back to building other features work also provides a free user management solution called authkit for up to 1 million monthly active users it's a drop in replacement for odd zero and comes standard with useful features like domain verification role base Access Control bot protection and MFA it's powered by radic components which means zero compromises in design you get Limitless customizations as well as modular templates designed for quick Integrations today hundreds of fast growing stups are powered by work OS including ones you probably know like cursor versell and perplexity check it out at work.com to learn more that is work.com do I understand correctly that a this happened when you were at E7 level which is a very senior level I think equivalent of senior staff or principal engineer at companies that do have like uh levels and then B what can you give example of what kind type of work you were doing at the time you know just give us an example of like what your like week to week looked like CU you mentioned like refactoring are doing impactful stuff yeah um so the conversations start when you're at the E6 level which is generally considered staff um you know three is Junior four is midlevel five is senior six is staff and then seven is senior staff I mean getting 3 four five is kind of your more standard common traits between those levels and at six you start to think about what makes me a little different than others um you're participating at a higher level and then that's when you have the conversation to get to Seven I was a coding machine from day one in unofficially um like literally as an intern the first thing I did on my second or third day was I just noticed that the org chart tool looked really it was it was kind of a handmade tool at at at Facebook and looked really really like janky like internal tool where you can look up like the or orc thing like I feel most companies have their own if they're not using something like workday these days but a lot of them build it right clearly Med um and I don't know why like I mean Facebook built all their own tools so I guess I know why but um it was just kind of weird like it showed like two levels it did like lots of scrolling it it's really weird so I was like I'm just going to rewrite this and then and you know the the vertical height was weird so I just made it horizontal so Mark Zuckerberg was on the left and his reports were to the right and their reports were vertically below them so it's kind of like a horizontal layout and I just like shipped this I I I wrote it my manager approved the the pr was second third day as an intern shipped it and didn't get permission or anything like that just drove it out the door didn't even really know what I was doing like kind of socially or social norms wise at the time I just thought it was like cool let's do this let's crank out this code um and that was like really well received and so so you shipped this on like I don't know second or third day same day landing and then suddenly the internal tooled out everyone at the companies using it just like completely changes the visual design and people thought it was fantastic because uh the horizontal nature which was kind of actually just a judgment call because the the page was too tall vertically like if it was a vertical layout it just too much scrolling uh but people loved it cuz it kind of flattened the hierarchy it showed that you know anyone can have influence across at any level um and on a horizontal line like an intern is at the same line as Mark Zuckerberg so it's was like and people love this and it so it kind of reinforced this idea that like wow this is just how things work here I can write as much code as I want and ship it as fast as I want and we'll see what happens and then there's lots of Adventures after that which like 2009 how how big was a company uh about 200 and 20ish I say about 200 engineers and about 700 employees at the time so relatively small yeah still in downtown paloalto across a couple offices yeah um sorry that was a bit of a sid trck a story we're talking about uh what I did on a day-to-day basis I think as a as a coding machine but um so I was a pro I I called myself a product engineer I was always officially allocated to a product team um I started off on internal tools which is a product team at at meta um and then I worked on Facebook groups and the first version of Facebook workplace I worked on news feed I worked on um Facebook groups for schools and colleges I worked on messenger for kids that was one of the last things I worked on um so I was always on a product team um and but I I was always doing a lot of refactoring and code cleanup on this kind of quote unquote on the side so a typical day was kind of um I would say contributing as like a sen E5 senior engineer kind of locally on the team that I was on um someone who's you know working on a pretty small team of 5 to 10 Engineers leading projects helping break things down helping Mentor um and then that was kind of like I would do that in like 30% of my time and spend the rest of my time on this massive refactorings and cleanups and kind of these um codebase wide efforts um wow so like in 30% of your time you did the work that you know like for an E5 or E6 that would spend I don't know like you know like 60 70 80% of their time depending on how you look at it yeah wow yeah but that was but that was kind of like like that's kind of also why the coding machine progression is is hard because it's like if I'm kind of acting like that that role on a team it's like if I did three if I spent that 33% of the time multiplied it by three and I just did like five three replaced three senior Engineers that probably wouldn't be enough to justify getting to like the the kind of E7 plus level at meta so that um had to kind of be weighed in with the this like outsized other impact that I was having um yeah I don't if that makes sense and the the concept of archetypes it's it's super interesting that first of all you know someone championed you like was it your manager at the time it was actually my skip level manager you're skip level but but like you're skip level Champion saying all right here we have this this IC doing great work but they're not they don't really have an archetype and let's let's create one to to recognize and and you know help them grow professionally but taking us it back the concept of archetypes is very unique to to Facebook or or or meta these days I haven't seen it at many other companies and but what what I do see is there's kind of a glass ceiling of uh people being stuck unless you become a tech lead or or more of usually it's a tech lead uh where you go on and you have an impact why do you think few you've now you know with your current business you're also you have window into how other companies operate and how how they interview and how to get and why do you think this is that that meta still seems to be one of the few companies that has come up with this concept of you know career progression for senior software Engineers on the IC track with archetypes yeah it's I think it's a good question it's like a I think it reflects the company's culture um it's an engineering first culture there was tremendously talented individual contributors that I worked with early on in their careers and meta wanted to support those people um and kind of break through that idea that you have to be a manager to progress um so they wanted these people to be empowered and it's not just the levels like um feel like now the days the companies are so large people are kind of um maybe the way that IBM felt to me early 10 years ago is maybe how people might start looking at the structures of these big tech companies now they just have like tens of thousands of employees it's a lot more rigid um but meta like really wanted to have these individual contributors be Superstars that were kind of socially influential to at the company and and kind of you would you would see at all hands you wouldn't just see a presentation from the vice president of a department you would see a presentation from the highest ranking individual pres individual um contributing engineer as well um and they kind of had this this equal presence but I raised a lot of uh or flipped a lot of tables you could say about this at the time because I felt like there still wasn't even a enough support of ic's at the the senior levels um at the kind of more Junior levels of management there's an there was about parody more or less between like a a first level manager and a staff engineer the M1 and E6 they were roughly equal um percentages kind of normalized and stuff um but once you got s just to to click on that when you say roughly equal do you mean in terms of like influence scope comp compensation expectations that kind of stuff yeah all of the compensation man like scope all that stuff was roughly equal and the kind of ratio of senior managers to Junior managers was roughly the same as E7 to E6 which are like the parallel levels um and then once you got beyond that um to the kind of director level which was D1 and E8 um there was kind of 10 next more directors or something like that I mean there was like 10 E8 at the at time there was like 20 30 directors and then that's when I started asking the question like okay so there's this great career progression for ic's compared to all other companies that that I know um but there is still kind of this limit and I was really pushing back on that I wanted like the infinite limit um I wanted to know why there weren't equal numbers of directors and and kind of principal engineer or uh whatever those partner Engineers I whatever the titles got beyond that there isn't really titles yeah um and uh so I pushed a lot on that and and at the time the reasons weren't necessarily super clear so I was really pushing on it and in retrospect kind of zooming out I really see that there's a limit that an IC can have in terms of of impact as an IC um without having these manager like uh kind of behaviors um even if you write 10x the code or 100x the code um you cannot replace 30,000 Engineers with one engineer so you can be a VP of engineering though overseeing 30,000 Engineers um so no matter how kind of multiplied out like uh I don't know it's like some anime analogies where these characters get like super like super like jacked up you know they they they they can only get so powerful though as an IC and and managing all of those if you're if you're the manager of 10 of these Superstars you still have more influence SL impact over kind of the direction of those people and the company um so I still you know maybe this is like a I still don't have like a the best articulation of that but I do understand now more why once you start getting to those extremely senior levels that you can't really get past a certain point as an IC but yeah uh is is it fair to to say that for these really senior IC roles the E8 the the principal distinguished or partner Engineers tenure becomes really important because I would assume that it's really hard to come from the outside and have this outside influence whereas if you've been there for a long time you've helped build some of these systems you know them very deeply you have a super strong Network it's just a lot easier to have that outside influence is this a fair observation it's an excellent excellent uh observation and question and point and discussion topic for sure um because this was actually one of the most interesting kind of tensions I started observing when I got to that level um the anyone who's an E7 plus um there's kind of a private group for those people to interact um that's kind of like you know back chainel maybe they'll you know kind of syn on thing whatever this probably most companies probably by the way and uh and there's like a kind of inperson event too that they were doing at the time and there was basically two groups of people at this and in this bucket there's the outside people and the inside people and they very different I those are't the right words because that sounds actually very it sounds like yeah but I understand what you're saying external people who joined joined Facebook externally as at a high level and people who grew there from a low level um and there was definitely a tension like the the Facebook culture is so strong the values are so strong the people who Rose internally were generally promoted extremely quickly trajectories to me um and they had a very Facebook ingrained like uh behaviors people who came externally at these really high levels they brought um different perspectives from their previous companies so a lot of extremely senior people from Google and Microsoft brought kind of their perspectives um but the kind of those they they definitely did not immediately align to the Facebook values the same way the internal people had built up over the years so there was definitely this like a difference and generally speaking the external people who joined externally had a harder time having impact at those levels and not a lot of them um were at Facebook for a very long time and a lot of them kind of left I I don't know if this has changed in recent years um I would probably expect it would have changed but at the time when there there was only about a hundred of people at these levels um there was definitely that that difference um and I I think it's a testament to how strong Facebook's values were actually meant something and were not something on paper like if you were move fast and break things was a value and if you moved really really fast like and you broke something you were almost rewarded is not the right word but like it was not a bad thing that you like were not fired for breaking something because you were trying so hard to live the values or um if you were making a really bold bet because be bold was a a value and you really pushing pushing the the limits of of uh what people thought or pushing people to think differently like that was really rewarded even if your idea wasn't great for the business at first and it never worked out um so I think that's kind of why that happened I I can't speak to today though what happened since then um if that's still the case um there's definitely a lot of uh AI Talent Wars that are starting there's a lot of extremely senior people in AI who are being brought in from external who have unique skill sets that are not repl are that kind of Irreplaceable and there's a lot of stuff going on there that might change that but yeah so I'm interested in your story because when you told me that in six years you went from intern to E7 which means you went e intern which I don't know e0 or something E3 a new grad E4 E5 senior E6 staff E7 I I think that's like four I counted four promotions and increasing difficult ones usually a career path like this would take at least a decade if you're on a strong trajectory can you talk about what happened how you basically roll through the ranks uh so uh quickly and you know what what you learn doing this I was known at at the company companywide as being like the single-handed most all in on Facebook person that you could imagine uh when Facebook rolled out public facebook.com email addresses for all users I like shut down all my other email accounts and only used that one um was just like all really all in and people were just like you don't have a Gmail account this is crazy um but anyways in terms of the progression um so I would say one level at a time um it's kind of like uh I have a not afraid of heights but like if you're kind of doing a hike or something and it's really steep Cliffs and you know it's like if you if you look if you zoom out and look at the cliff you're like oh my God this is like no one could like walk on that path or or climb that Cliff this is like this is impossible but if you kind of go there look down and just focus on the next step you can get there and you might realize that it looks a little different than when you were zoomed out um so I was really took one step at a time um and was focused just on on what's the next level what what are the requirements am I doing this yet Etc um and there's two key things though that I I think um worked really well for me um so and there are kind of very different different things but the first one was uh my manager relationship and this was not um I don't mean this in like a i you should be friends with your manager or I was friends with my manager um in fact I was a very difficult person to manage and I don't know my man liked me that much just cuz I'm am a little impulsive and I'm very passionate when I'm doing my work and it's a little hard to Wrangle me sometimes when I'm really in there but um what I mean by the manager relationship was there was a lot of trust and transparency both ways so my managers knew who they were working with and I trusted that they knew who I was and that we would we were on the same page about putting me in the right spot to work on things um and it goes both ways so like I also my man I also felt like I could help my manager do well at their job so if that's maybe they're a little overloaded with Junior people on the team helping to Mentor those those people and um having that kind of more um trusting supportive relationship both ways I think really helped progress you know I could they my managers could be really blunt with me like Michael you're I'm getting feedback you're like causing too many bugs on this area of the code it's bothering this team like that's going to hold you back like work on that and I is and I I'm like not defensive and I'm like got it I will work on that but I will not slow down um and we kind of got that relationship that's really important for me because it really helped me just move through the system on the second thing completely unrelated um I always had a notepad of of things I was doing that might be useful in performance reviews or in summarizing my work later on um so if things happen in the moment um I don't know fixed a really weird bug that was emergency bug that was flagged at like the company level and I jumped in and fixed it I might just like put a note on my notepad that I did that um then when it comes around to kind of performance review time or one-on ones with my manager I could kind of go over these things I was doing and bucket them and be like you know I'm doing a lot of work that a lot of these things are are contributing to the the requirements of the next level so what else am I missing to get to this level um and then working on those things um so that that kind of really helped me it's it's funny that you say this because uh this concept like when vir when Julia Evans blogged about she called it a brag document in my book the software injurers guide book I actually like it's one of the best advices I I gave to people who were focused on the career I called it a work log it doesn't matter what you call it but I've gotten feedback from people saying oh I did this one thing that you know they heard about from somewhere from online and it made such a big difference and it's such a small thing but especially when you're doing a lot of unrelated things or or here and there it like and by as a manager when I used to be a manager I loved it I loved the when people just brought this to me because I was like oh I had like as a manager I didn't know about like 90% of these like stuff that was not the main project and like it was just really pleasant surprise like so I I think you know I'm assuming your manager's View kind of you know shifted on you as well and knowing like oh you're doing all these other things which is sounds like meta appreciated this as well Facebook yeah exactly that yeah it's a little tied in it's I said they were unrelated but it is tied into the manager relationship because if your manager trusts you and and knows how to interpret your words and you give them this document they know how exactly how to turn that into the right thing and and what that means for calibrations and performance reviews and stuff and so I plus one to that for sure and speaking about Facebook's engineering culture back then it was called Facebook now now now it's called meta so I'm I'm going sometimes back and forth but what was it like and like starting with that like can you share like we know Facebook had internal tools what kind of internal tools were they like how can we you know like imagine if someone you know went back in time what would you you know describe to them what did you see yeah um this is a good question I started off on the internal tools team um I was originally going to go to I was originally going to go to grad school and do a PhD and I really wanted to work on like uh kind of social networking like things but for companies um and not just companies just like work cuz social networks were really like connecting people in a way never seen personally and I was like you know that would be really cool if like our work was also similarly connected if we could uh just as easily find co-workers of co-workers and message them and stuff like that um which now seems like a obvious but at the time it was not obvious um people you know it was very different but um so something that's really cool about about Facebook at the time was that um internal tools were products so they were products just like user-facing tools built on the exact same code base um deployed to an employee um subset of Facebook um rather than like the public website obviously um so it was a internal tools were seen as as product building um they weren't seen as like like these annoying things that the company has to do like a lot of tools were seen at the time like a you know it's like no one wants to be on the tools team it's the the slow moving boring team but at meta was like the our Facebook at the time it was like probably the most exciting team because you could build product at a speed for building product for 7 8 900,000 people was very much faster than Building Product for at the time I think it was like 200 million people um publicly so that was really really exciting um meta still I think to this day has a culture building a lot of in tools from internal tools from scratch um they built it um one of the main reasons is they built all their infrastructure from scratch there was no Cloud providers at the time so they had all their own infrastructure they even built their own servers and and eventually hardware for servers themselves um so that mentality resulted in a in like really custom tools to work with all this custom stuff that really just didn't make sense outside of it um and but yeah um I think because of that because all these tools were like treated like products though there was definitely like this engineering product ecosystem the whole company has had as a result even if you weren't working on these tools you were using these tools um you really felt like Facebook was a technology company that was solving even it's internal finances using internal tools and wow you know site Integrity like they it really felt like no matter what job you had at Facebook it was a product engineering company um and I think that's I think that's still stuck through to this day too so so then just to be concrete like there was like a spec like a custom Version Control it wasn't like GitHub or something like that custom code review custom build tool and I know you know Facebook also open sourced some of this or at least really relased some of this like the build tool I think was it was a buck uh and like fabricator they also released for some but like everything was just like completely custom right or like internally like built for to work however you want it to work yeah they I think maybe in terms of everything you said except for Source control um they were using SVN as the backbone um and then people used at the time I I can't remember if they used git first but they basically used mer jump to using Mercurial locally on people's machines to manage their local branches but then you would always uh merge using SVN to the main the main branches and repos um and the reason they chose Mercurial was because it was a lot easier to work with the open source team to like hack hack into it things that they wanted so it was still the same ethos even though it was um not their thing they the g g people were not as open to making some of those changes so um yeah building all all those to the code review tools a lot of these tools have seeded um like the companies of them their own now like I think stat Sig and I think there's some others um so so stat Sig is experimentation right or it was seated by the experimentation platform yeah gatekeeper internally I mean these are not like more I I also know graphite they're they're a team five meta Founders who are bringing the concept of stag diffs to production uh like again in inspired by by this it feels like there's a lot of teams who saw like Facebook had something really cool that worked really well at a large company but it just doesn't exist outside and now they're productionizing it and seems like there's demand for it Mark Zuckerberg said in uh it's like a why combinators startup School talk I think it was 2009 or 2010 um he said that he wanted Facebook to be the place that was known for for like training people how to build really good product um and kind of training people to potentially be really good Founders um and I thought it was like a bit weird at times cuz at the he also didn't talk that much publicly at the time and I was like oh but like what what about training people to be really good engineers at Facebook to stay at Facebook forever um and I guess it was the audience it was startup audience but it got me thinking a little bit and and when you zoom out and look it's like Facebook was exceptionally good at taking passionate ambitious young Engineers who don't have much experience and building extremely strong abilities for them to execute and deliver and we're seeing a lot of those products now being like really robust strong products that are kind of you know coming out and and helping people be on Facebook out too one question I have is it true that there was an internal tool just for booking meeting rooms and that it it was awesome as well someone told me yeah uh there was uh my intern project when I started was uh was also a it was a a meeting scheduling tool because it was like we just wanted to click on three people find a meeting for them or find a time they can meet find a meeting room done um so I had bu that tool yeah like when when I first heard this I just had such a hard time believing cuz this was Facebook was already a lot bigger that a someone would build this B people would also use it and C there would be a team team maintaining it and somehow that all all happen because it kind of sounded like some anyone could build this in any company but maintenance would be a pain you know funding most companies would have a lot of politics of in terms of like okay we need a staffing for this team we need budget we need to justify an IND theia and a few years later oh let's buy a vendor solution or something uh so it it it really surprised me that even for a relatively small but really important thing it kind of happened there yeah um it's it it sounds kind of like a when you talk about it now it sounds like a kind of I don't know Wild West place but um or like you know like there's no problems there's these things are just it's like the you know the best place for engineers but there's also there are those challenges too when you chart start to grow in scale it's one of the reasons why as as Facebook got bigger um I was kind of less like I that was not my my thing you could say um like the meeting rooms for example right like so that was one of like 40 tools that I worked on when I started was the tool to help people find and schedule meetings it was it was just a very quick I don't know fast thing it worked did 90% of what was needed it was kind of that vibe um and then you know as the company got physically bigger and physically had way more meeting rooms and you start diving to the next level of problem product problems like what if people book the room and don't show up like uh do you free the room what if someone else wants to use it what if someone's like sitting there and not doing a meeting but they're just doing they're just sitting there for fun um you know there's all those problems that you start just like I was saying product problems even though they're internal um and you know and then so now you start getting into okay do we need to have s in the rooms to know if someone's actually there and then if we need to buy Those sensors we do need a budget for it and then we need to install them and we need to install them across 20 different offices and then some offices have like restricted floors for external people and you can't put them in those ones and like you do end up getting into all those details too um and I really love the product details I think it's like I love like going to every single detail of a product and really like ironing those out but I do not like the budgeting side or convincing other people that this is worth the budget side um I'm just like you know give me the keyboard and let me code um that's me yeah so you you did a lot of coding but uh you were a pretty early employee at at Facebook relatively did you get to work with marber because I know a lot of early employees do have like either collaboration or or or some stories of of uh of interacting or working with him what is yours yeah for sure um so yeah I friends with him um because I was there early and I did cross paths a lot and um got to know him a little bit over the years um so I mean I can tell the story of how we became Facebook friends that's kind of like a thing start as an intern um you start as an intern like you know it's all these interns from school even though like Mark was like two years older than everyone or something like he was not uh he was like a College Dropout it wasn't like he was like a you know a IND industry veteran but um it was still you know he was he was well known and all the interns kind of friend request him on their first day and uh hopefully he'll like accept it you know and um and it was interesting the way that I the way he kind of we became friends was um he used to play Risk the board game um I think it was like once a week people would get together was like four or five people and he would show up often and I really liked risk like I would always play Risk a kid for whatever reason I don't even know why um and one day we're like down to the final three people and it was like me him and I was losing Mark was like in second place kind of the the leader had us uh was taking over so I did a really uh let's say delicate strategic maneuver where I made an alliance with him to share resources to all go after the first place person so he went all in against the first place person my turn's next I did not go all in against the first place person I took over Mark's remaining resources he just dwindled going after the first place person um and you know um I think and he accepted my friend request very shortly that evening I think um but um the reason just like showing kind of um uh he it kind of um it's weird I like basically backstabbed him in the game really bad blatantly into his face um but it's the game that's like what risk is and it's a it's a strategy game and I think that um you know he appreciated the strategy that I had that I um it it made him feel more like he could trust me even though I backstabbed him he knows my where my strategic thinking is coming from um that's something I've seen with Mark over the years with in product reviews both myself doing product reviews and also product reviews with other people um he ask a lot of questions he's very very detail focused he really dives into like trying to figure out can you tell us what the product review is cuz I don't think it's as common outside of Facebook yeah it was called the Zuck review at the time because there was uh he was the only person doing them but um there was maybe like 10 20 product teams of five to 10 people at the time roughly and once a week they would do a a Zuck review like a 15 minute presentation kind of thing usually the PM or the Edge manager would would do them and sometimes members of the team would join if it was relevant um so it kind of it's like a doctor's office there's like a waiting room of the teams because they're so short um and you know one team would go in do their Zuck review they would come out and you would see like they got good news or bad news and you would see the expression on their face um if it and um Zak would kind of just give lots of feedback and steer the direction and they're really short so it's very efficient um feedback back but um yeah those were kind of interesting the the more Junior PMS would really stress about these because it was like their FaceTime with the leader of the company and um and like I was just saying about the risk story you know like Mark is kind of judging constantly you know does he trust you does he trust your judgment trying to read the situation so it was a little stressful for junior PMS going in there and uh they would really like sometimes the night before be like trying to think of all different angles and they're like I got this backup slide in case Mark asked this question and this other backup slide in case this one and then you'd go into the review and like you would interrupt in the first slide and just go into a completely different direction um you would see that sometimes like um in like the nicest way possible like it's very not mean not aggressive person just you know um at the end of the day it was about everyone delivering really good product and wanting to build the best product and really trying all the hardest they could as individuals to to get there and not sugar coating anything not padding on the back for no reason like we just all got to be really honest about what this product looks like and how do we make it better and and yeah so you mentioned you know you TR you tried to work hard to build the best product but you also wrote tons of code so I'm going to assume you must have caused some outages because it just happens and move fast and break things what was your worst outage or worst bug there was this product that I don't know what the inspiration was I didn't come up with a product idea but um we wanted to allow people to upload like a photo template on top of their profile picture so nowadays we see that on LinkedIn all the time with open to work or hiring um that kind of thing like this didn't really exist at the time they wanted to let people at Facebook do that it would help increase engagement and people who haven't changed their profile photos for like 5 years might use use this to Spruce it up a bit or something you know um so I was kind of I built I built this kind of a prototype of this thing um really fast like in a week or something after it was handed to me um and then unfortunately there was some terrorist attacks in Paris and um a lot of people were expressing support on social media uh for the people who are impacted and trying to like hope for uh good recovery and um we released a uh using this new system we released a a template of the French flag that people could overlay over top of their their profile picture to show support yeah um what we didn't expect was 100 million people to do that within like a day or two literally and uh this was a prototype so this is an interesting thing I basically um the way that I built this is you know you have like a a template object uh photo object all these things are Federated across tens of thousands of databases so each kind of object has a home database um so you know we need some some way to tell this relationship between profile object photo object um and or sorry template object so there's there's kind of a an edge concept to connect the two and you can have a one-way Edge that just goes from one to the other or a two-way Edge where you go in both Direction you can go in both directions starting at the profile to the template or starting at the template to the the profile um so um sounds simple sounds like systems design 101 kind of system design interview that at formation we're like helping people with all the time you know it's like a classic problem and then like the system would just like fetch these two objects and like combine them or something like right yeah and typically at Facebook most things start with the the profile like a person logs in and that's like the the root of all queries so to speak you start with a profile and you get the profile's friends they kind of edges it's like a graph and they're you're starting at the jumping in at the uh the profile and then uh crawling through the graph from there and if the profiles are all Federated across tens of thousands of databases you can kind of kind of crawl out through a nicely distributed system without overloading any particular node um but when you have this template which is an object in the system just like a profile if you have an edge from that to the profiles and there's 100 million of those written in a day you have 100 million edges going on that one node one database that that template single French flag template is stored on um being written constantly and writes are a lot more expensive in a system on in at least Facebook system and and and that node is like one machine or virtual machine or some behind it basically exactly yeah um so and I mean there's lots of other objects on that that Ma machine too like there could be other profiles there could be groups there could be like notes there could be events like all kinds of stuff on that machine it's just it's kind of the Federation is not by object type uh generally not by object type um so that machine basically melted I don't know physically I didn't hear if it it may have physically melted no probably not but um it so that machine like crashed and then and you know if I'm trying to grab my friends and my friends on that machine like that fails so it causes cascading effects if that kind of thing can happens wow and um yeah so it's interesting now the interesting thing is like it's a really bad bug and um it's interesting we help at formation we're helping people with systems design all the time it's one of the most common things we're helping people prepare for in interviews and um when you talk about these things like this sounds like the most basic systems design system like you know you have Federation and is a common and and partitionings are very common topics like you know um and this sounds like an obvious thing like why would you potentially have a a a two-way Edge on a and like on a system that where one of the nodes might be like super heavy um uh this is not an exact example but um Twitter had a lot of outages with the fail whales back then um because they had a system that was similarly vulnerable when uh Justin Bieber tweeted or something they had a similar system where that yeah cuz they had the fan out problem right yeah so similar yeah I mean these are like known problems like you know why yeah um and it's interesting cuz it's a in I see so many people kind of preparing when they're thinking about systems design interviews and stuff they're like thinking about like the textbook answers for all these different things that can happen and this is just one really simple situation that talking through why like why did I make this two-way was I like ignorant or was it um not and the reason I did it was that this was a prototype and when you're productionizing something thing um you need to put a lot of logging in for performance for for collecting metrics for for analyzing like is this working is it not all these like behavioral metrics all these things that you would do and you have to put a lot of time and effort into thinking about what are those things where do we log them um how do we log them how do I explain to the PM how to do it and if we just use the raw data model itself to store some of that and we keep that two-way Edge it gives us more information it let us just count the number of edges is the number of people who have used this you know like it's it's it's simpler for a prototype um and we don't have to like build all this stuff if we don't ship it I don't want to rip out all this logging I don't want to like have wasted time explaining to the PM how the loggings working those kinds of things so totally simpler reasonable um that's why the decision was made and then the second this machine melted we knew exactly what to do and we so how how did you resolve this outage right because now now you actually had a problem which sounded like it's a data level and usually with an outage if you might you know the best way is to roll back the change clearly not an option here uh how did you how did you fix it so that actually people could you know start to use those you know profiles um yeah this was so this actually was beyond beyond my my area the um DB there's a DB infrastructure team that stepped in um I removed the edge so we made it a one-way Edge yeah um due to the success we of the feature we also productionize it and addressed those kind of more logging and things like that that we lose by chopping off that edge um but um yeah so the the infrastructure team kind of handles the rebooting the databases the replaying of SQL logs that might be necessary because of loss transactions and things like that um they handle that level um as a product engineer you need to be aware of what's going on and maybe send over a a sixpack of beer or cake or cookies to the people afterwards to to build up that the reput I'm assuming it was not the first time for them like some someone adding an edge and overloading a few nodes let's just say the that the kind of site reliability team had one of the most uh well stocked bars in the entire like alcohol bars in the entire company they just kept arriving thank you for this thank you for that it was and actually not joking um it was extremely well sted um I don't know if that's still allowed or what if they but lots of favors there that that that that that's a good one so something interesting you told me uh be before this one is that when it came to hiring uh Facebook has this content with the hiring committee but it was also open door so you could go in and you could Shadow and sometimes and you did shadow um multiple times and then you participated can you explain to us like the ones who we we were not at Facebook uh what was this hiring committee uh meeting like I understand these were when they decided on on the uh you know higher no hire decisions how did they go what was it like to listen in and and what was it like to participate and what does this tell about Facebook's hiring process for software Engineers maybe just a really quick kind of recap of how you get to hiring committee the kind of met or Facebook um interview process starts with a lightweight recruiter screen get to know you um and then you'll jump into a a coding interview it's called the technical screen it's like a 45 minute interview usually two questions um if everything goes well you'll go to the on-site um where you'll do for most people you'll do two more coding interviews a behavioral interview and a systems design interview there's some subtle variations there you might do cool names right these used to have cool names I remember there were J Jedi and Jedi pirate and uh and uh Ninja n ninja was coding pirate was system design and Jedi was the soft skills behavioral at the time it was half behavioral half coding still they really wanted the coding signal um now I think it's all behavioral um but yeah um and then also just because this is what we do at formation all the time uh two like critically interesting things that are about meta's interviews specifically the technical ones that really throw people off too are um the they are whiteboarding style and there is almost no Small Talk um and these two things like can throw people off in the technical interviews um a lot of the times you'll you have 45 minutes you're expected to code code up solutions to two problems so you know you jump in and sometimes the interviewer will just be like hi my name is blah nice to meet you here's a problem um and it can throw people off a bit because some people are well everyone's nervous I don't know anyone who's not nervous in interviews but um you know it can be a bit like you know you want to like settle in build a rapport like maybe if I screw up the question they'll like me um no not for the coding interviews you uh just got to write the code um so that's like one thing and then the second thing is whiteboarding style um now that things are remote I don't even think you can compile the code they either block it or they tell you not to um so you can't just like guess and check and run to to run your code to see if it works they want to see you walking through a a clear and clean thought process problem solving process for um how you're approaching the problem how you're solving the problem they want to see your code being clean they don't care if it has some minor syntax issues or you forgot a built-in functions signature and you're you just guessed it for the the interview type thing um anyways those those are two kind of things that are specific to their interviews but anyway so you have this kind of you do your on-site you do your technicals your behavioral your systems design or maybe two systems design um and then you uh if everything goes fairly well or really or better then you'll go to hiring committee um if you got a um there'll be like a quick debrief with the people so with the people the interviewers I mean so you'll the interviewers will quickly sync sometimes over email even uh were there any red flags were there any Strong Nose you might just end the process there um so usually t a formation all the time if you do a meta interview and you hear back really and and you don't hear back really quickly then it's usually a good sign you were not quickly know um you at least PS so people get nervous when they're like oh no I didn't hear back in a week and that's not a bad thing that's an okay thing or maybe even a good thing so um anyways you'll end up in hiring committee at the time they met when I was there in the the mid-2010s um they were meeting like twice a week I think um and it was open and this was like really I don't even know if it was like officially open or like I just went um like it wasn't publicly open but it wasn't closed there was a public calendar for for engineering recruiting and it was on there and um often time like all the recruiters who were representing a candidate to try to get approval for hiring would bring their candidate there with a packet so people were going in and out um and I just I I showed up and I would just started going to all them and I would have to figure out where they were and it was kind of like a you know secret secret handshake like not figure out what door to go to like sometimes it would move last minute and kind of figure out where it was sometimes but if you if you found it you walk in and sit down at the table um so what happens in candidate in hiring committee is um a recruiter will present a packet this is actually pretty interesting I have um very few Engineers talk about this topic by the way the hiring committee recruiters are definitely would be able to but I haven't seen a lot of them talking about it so um I won't give like any Secret Sauce away but just the general process um a recruiter presents a packet um to uh a quorum of of leaders so there had to be at least three director or VP engineering people there at the time um I don't think it's like it's not like a legal constitutional number but it was that was just like they wanted more the they wanted a quorum of of DEC decision makers um and recruiter would present the packet to them uh a packet has a bunch of information it has all your interview feedback it has um this is like super super interesting but it has the the interview history of the interviewers themselves it has the a histogram of the interviewers feed of uh past ratings so like is this interviewer like super binary they're are strong yes strong no historically are they like lean yes or lean no so it helps the reviewers kind of calibrate interpret people's feedback are they a really experienced interviewer or are they a brand new interviewer um it would have the questions they asked it would have how many times they asked those questions like did this interviewer asked this question for the first time or have they asked it 150 times um it has like a lot of information in the packet um that was presented to the to the kind of directors the directors would read it over look for anything any Flags usually at this point like the default would be that someone would more than likely be hired so they're kind of looking more for Flags or think inconsistencies um I can give tons of examples of what those are which are you know this is this is one of the things that formation we like one of the reasons why people work with us is like getting to some of these nuances and stuff are are pretty important but might take hours so I won't go into all of them um but yeah they're looking for flags and like different things um the most common thing that they're kind of looking for is level I reiterate the fairness and consistency that that meta wants to have in its hiring process and and they want to bring in someone at a appropriate level where they are an appropriate level consistent with other people who perform or have similar scopes of responsibility so they want to make sure that your level's right um and that was probably the biggest thing that the that was discussed was kind of like is this person a senior are they midlevel if they're a midlevel do they have too much experience is their career progression a little bit slower than we wanted for midlevel and maybe we'll reject um or are they actually at the senior level and this is simple or are they did they interview at the scene and they and they did they interview and appear with some of the traits that a senior engineer might have but they're clearly midlevel from their background and they want to like make the call on where does the offer end up um stuff like that they're kind of working through all those those things um so why did I go is maybe an interesting question like what what this sounds kind of boring maybe um but I was like very interested let's get into it making sure that the I was like paranoid that the that Facebook would lower its hiring bar as they scaled and uh even now I don't really like saying lowering the bar because I think that especially working at formation and um the kind of like the secret sauce I think to the to interviewing and finding a job is finding the right job now um so just because you didn't quote unquote meet the bar for meta does not mean that you're a bad engineer and not hirable it just means that there wasn't some kind of alignment it might be your it might be them it might be you like it's you don't really there's too many ways to overthink it but um finding that like right role is like way more important than just checking out the boxes and passing the interview and feeling great that you got the offer so I think that that's like uh in yeah so in the time though at the time I was just like paranoid like like I don't understand how the company can grow 10x and not like not not lower the bar accidentally or like I'm paranoid about this like I want an extra check so I was like going these things to like see who's being hired and seeing if I had any flags and I would chip in occasionally with my my two cents and um you know sometimes it had some impact um sometimes it didn't and well I guess look at how things work out now turns out those sessions were pretty helpful now doing what you're doing at formation right like like really uniquely coaching people helping people the fact that you were in the room is a big difference yeah um it also yeah really helps really helps people kind of just yeah even if it's just to reassure them when they're if they don't get selected but they thought they did really well um I think I have you know you can say things with words but I really have the experience there to to help people like you know get that it's might not be that they not might not be that they failed or that it's just like a could be a lot of reasons yeah so in in terms of interview processes uh you are intimately familiar with Meadows but of course because of what what you do at formation you're familiar with a lot of other big techs what company do you think has the most similar recruitment process as as for software Engineers as as meta has these days if any um yeah there's a it's a pretty similar so I think the the lineage is pretty interesting because like it kind of the the interview processes are fairly similar a lot of the top companies and there are some differences here and there and some things have changed a bit over the years but generally speaking you know Le code wasn't a thing it didn't exist as a word until 2012 I think was when they were founded um and the the problems haven't changed the the interview Styles haven't changed since before leak code since after leak code um there's it's pretty consistent um kind of style that has been around and um Facebook didn't invent these things they kind of borrowed a little bit from Yahoo Yahoo is like one of the big quote unquote successful companies that kind of was on the shrinking side at that time when Facebook was growing so they borrowed some things from Yahoo from uh Microsoft from Google Google was the preeminent place to work at the time too so you know and Google borrowed from other companies and now we see um a lot of the AI companies um like open AI has a fairly Facebook ish process and Vibe um so yeah we're seeing this kind of um and ultimately like the I think the reason why this is the case is that these interviews especially the the we call them the Le code interview but like at formation we like don't really we don't like pigeon uh pigeon hoing the the naming to being like to being about like doing a problem clicking a button and something saying like you passed or that that kind of thing like the point of these interviews is to test stack agnostic and language agnostic problem solving skills and that has always been the point that's like what Engineers do um these and these interviews haven't changed as stacks of evolved like they they're really trying to take away any biases that might come from being a JavaScript programmer or a C++ programmer just focus on the core Logic the the um the the problem problem solving process um and that's kind of what they're trying to do and that's like what Engineers do so it's like it's very understandable that all many truly Technology and Engineering companies use these processes and you know the companies that don't are generally smaller companies because they do actually want expertise in specific Stacks or specific experiences or non-te companies that that don't really have the same uh the same ethos um yeah so we actually see it see a pretty the variations of these um some companies do some more practical non dsna style stuff but um yeah sounds like it's it's smart to if you want to get into one of these companies just go through prepare because it's useful everywhere right you you you master it once and it's it's useful probably for years or even Beyond yeah um yeah one of the interesting things at formation that we've seen that kind of shocked us is we actually see people coming back after they come to formation will help them with prepare for like these data structure algorithm system design interviews they get a job and then two years later they come back and do it again and pay us again um it was kind of one of the surprising things like we um I mean from a business point of view that's like the dream is you want to have a a yeah customers that like yeah a service that's useful to people for their entire lives is way more way more valuable as a business than a service that's useful at once um so that's actually kind of pleasantly surprising to us um and I think I think like the I mean we're we're going a little bit of tangent about interviewing now but um like I think the interview preparation process I draw a lot of analogies to like a personal trainer like you want to get in shape like you might have been in amazing shape 10 years ago you remember what it was like to have the habits to be in shape to eat well to do your exercise and like you can do it again but you're not in shape now and you need to maybe get in shape again and whether you got a gym membership or a personal trainer or uh you kind of just make do it yourself like there is like kind of this recurring pattern of getting ready for these interview processes um there's infinitely many stacks and combinations of Frameworks that an engineer can work with and when you work with your Frameworks and your stacks for years and years you kind of um you know you it's like you're working out but you're only doing leg exercises so maybe your legs are like really strong and your upper body is like not as strong because you haven't been doing that and you know you have these different Transformations that you make just from your day-to-day job so kind of check when you're making a a career transition to a new job like getting back in shape again um makes a lot of sense yeah and I I know you know people complain about how interviewing is different than the job but it is and it it's how it is like I think you can either fight it or or or or accept it so but speaking of uh this topic what advice would you have for more experien software Engineers who are interviewing right now just in the sense that we know that the job market is a bit tougher than before what kind of tactics have you seen or strategies that help people land offers at some of these better known companies and what are things that are more important than before yeah it's definitely changed um over over time I mean when I joined at a school it was like um I think there were so few Engineers um like I know people who are trying to get H1B visas were just being handed them because they had such a surplus wow and now there's like 20x over subscription of people trying to get tech jobs with um in the US and specifically as one example but um like there there was a different buy back then I would say but but now it is more competitive there's a new team matching process that we're seeing a lot of companies have Google's had this for a long time interesting um Google's had this process for a long time but meta's more recently added the team matching process we're seeing a couple of other companies smaller companies but not not um do kind of this it's like you you know you passed the on-site but now there's another another uh you know Li you're kind of in limbo for a while until you actually match to get the offer um and then there's some yeah so we're seeing we we are seeing like that competition be um distributed across a couple of different places um adding a couple extra layers um a little bit more filtering on the front end we're seeing more um more companies give online assessments like right off the bat instead of jumping into the technical interview um to kind of filter more people there um so we see a lot of people have like 10 years of experience they've never used the tool like code signal or hacker rink and they fail one and they're like what is going on I've never seen this tool before and we're like you know we have to practice that um yeah definitely seeing the the competitions definitely distributed across each level of this process for sure nice all right uh can we wrap with some rapid questions yeah sounds good all right so let's wrap the questions I just ask the question and then you just fire out what whatever comes to your mind or the one that you can recall I've been wanting to really is how many diffs did you close or code in your most productive year when you were a coding machine did you keep track of that um I did keep track of it because I was kind of in a competition at first I was competing with other people sorry was rapid but I was competing with other people to try to get more code than them once I surpassed everyone I was competing with like this release bot that release like the release bot no yeah so I was like I got to and then after that I think I stopped like counting but um it was a lot yeah uh in the thousands uh I think it's still I think I write I honestly write more code now though even after leaving Facebook at formation I'm writing a lot of code um so yeah yeah so like multiple diffs per day basically for sure yeah it's a probably around 5 to 10,000 a year like in the double digits per day but yeah you know that doesn't really mean that much some are small some are gigantic it's like hard to yeah yeah but but your your code actually did meaningful work you know that that was part of like what you got from a different yeah which language did you write the most code in when you were at meta and which one do you write the most code in now uh so then hack which PHP evolved into hack which is their version of it um hardly any JavaScript and then after leaving I kind of had to learn JavaScript as a almost like a junior um and react and get up to speed on that and now I'm a very uh all JavaScript yeah front end and back end and which one's your favorite programming language what's the reason so it's grown on me but uh I would say right I would say typescript specifically which I don't really call it a language but um so at meta I wrote this sounds like I sound like uh you know telling the like long time ago stories as like a grand grandparent or something but um I wrote all my code in Vim with no IDE no syntax completion all I had was syntax coloring like I almost had the code base in my head to code um and now that I've slowly warmed up to typescript um I actually really like it because it helps me catch a lot of hardly any guard rails I still can code almost as fast as I would with no types but it actually catches bugs catches helps kind of catch things also helps me uh you know be a little less lazy if I'm just going to throw some parameters onto a function to think a little more about like do I type how do I type these I move them into like a structor you know that type of thing but yeah and what's a book that you read that had an impact on you I think it was called uh histories it's by Herodotus the like Greek historian um and the reason why it really like sticks with me is it like a lot of the stories in there don't seem to make a lot of sense like factually and I'm like a very like factual literal person and when I read this I'm like wow like this is these stories don't seem to be real but like everyone's trusting these and it showed me that like storytelling is really important to try to communicate things that are happening whether the like you're literally saying things that happened or not um the point is that to kind of influence and to share and to be human we have to be able to like communicate stories so that that kind of showed me this like the power of Storytelling uh historical events thanks very much for being on the podcast this was very very informative and lots of fun yeah thanks for having me it was a great time to revisit some old stories and uh yeah had a great time thanks thanks to Michael for sharing his thinking stories and how a coding machine thinks about things you can find Michael on LinkedIn and Reddit with links in the show notes for more details about meta engineering culture and how hiring processes work at various tech companies see deep dives from the pratic engineer also linked to the show notes if you've enjoyed this podcast please do subscribe on your favorite podcast platform and on YouTube it helps the podcast a lot thanks and see you in the next one