Shipping projects at Big Tech with Sean Goedecke

successful projects there's a narrow path to success and there's a million paths to failure a narrow path to success the default state of a project is to fail projects only succeed because people drag them Kicking and Screaming to the Finish Line If you leave a project alone and nobody's worrying about whether it's going to ship nine times out of 10 that project will not ship the ability to kind of come up with a technical demo kind of moving mountains one demo of like hey look at this thing happening can cut through weeks of conversation about whether it's possible and if you just go off and produce that you can really kind of blow people out of the water in the context of like a large company being technically strong if you work on software is really a superpower because it makes you connected to reality it makes you connected to the real world to the actual product that you're building it it's sort of like a lot of the work these other people are doing around strategy and marketing and figuring out what needs to be built that's a huge multiplier how do you sh projects at Big tech companies Shan giki has worked as a staff engineer in two companies that employ thousands of software Engineers he spent 5 years at zenes and now works at GitHub Sean wrote a blog post about the topic titled how I ship projects at Big tech companies the post went viral online and sparked a lot of heat at discussion in this episode we Shan we dive into what do shipping projects actually mean when you're working inside a larger company what is more important in order to be seen as someone who ships projects your technical skills an engineer or your soft skills to navigate the organization the story of why Shawn was considering looking for a new job when his manager told him to not worry too much about test failing in the staging environment how did gen tools change the ability for us Engineers ship projects Sean is a great person to talk about this as he's worked on the GitHub co-pilot team and now works on GitHub models advice Sean has to succeed in a Flor remote team when working across time zones and a lot more if you enjoyed the show please subscribe to the podcast on any podcast platform and on YouTube Sean welcome to the podcast thanks it's great to be here I'm big fan of all your stuff going back many years so really exciting to get to talk to you so to start off what is shipping in in your post uh you wrote and I'm going to quote you I know it sounds extreme but I think many injurers do not understand what shipping is even inside a tech company so what is it my post kind of came from this experience of seeing um a lot of Engineers around me and myself in the past getting into this frustrating position where they were delivering code they were feeling happy about their output particularly the volume of their output but they weren't um getting the kind of recognition they felt they deserved uh and the sort of management chain seemed happy or or seemed unhappy or even worse just seemed totally indiffer to what they were doing um so that's that's kind of the context in which I I talk about shipping and the context in which I wrote my blog post so in that context uh I say that shipping is uh kind of a socially constructed fact which sounds like this postmodern way of talking about things but it actually cashes out really concretely into the proposition that a project is shipped when your management chain believes that it's shipped and like talks about it being shipped so shipping kind of means doing things that your management chain is happy with and I say management chain because like for small projects this can be your manager and just your manager but for larger projects um it's usually a layer or two above your manager particularly for like staff roles um and sometimes it's uh it goes all the way up to the CEO and in those cases the metric of whether a project ships is is the CEO happy um with what you with what you've done that can be a kind of frustrating way to think about things I think for a lot of Engineers uh certainly the comers on hack can news found it that way I think partially because it relinquishes a lot of control you know it's not necessarily about the quality of your work it's kind of about how it's received uh but I do think you you have a bit of a happier experience going into these projects with open eyes and I think you tend to be more successful if you're actually driving for this kind of like organizational definition of shipping rather than just saying I closed all the J ticket so the Project's shipped I deployed the code to production so the Project's shipped uh that that definitely does not mean that it's shipped you can deploy something to production uh and it can be a complete disaster this episode was brought to you by DX a platform that helps organizations measure and improve developer productivity the DX team includes notable researchers behind the Frameworks like devx and space so they're constantly asked by leaders what metric should we use to measure developer productivity to help simplify the landscape DX just recently published a DX core 4 a unified framework for measuring developer productivity that encapsulate Dora space and devx the DX core 4 includes four dimensions including speed Effectiveness and quality that provide a focus set of metrics that help track productivity at all levels of the organization you can learn more about the DX score 4 and how it's being used in companies like Dropbox Etsy versal and fiser at getd x.com cor4 again that's get dx.com slc4 something that feels wrong about this definition is is it kind of sounds like you know we've Shi successfully if the management chain all the way up to CEO is is happy and they're like all right great job but it kind of goes really against like I think what a lot of us feel is shipping which is does it work for customers and users the people who care about it this goes a little bit back to you know like agile like the the original agile Manifesto which talks about customers doing stuff for customers how do you think that is connected cuz clearly I can see on one you know if the customers are happy the COO is happy but you might have things where like I don't know the customers are not happy and the COO is not happy and like you you've been doing this for quite a while but surely you needed to reconcile this and also think about this fact I want to say a couple of things on this uh first kind of you touched on this in the question but I do want to stress that in any like remotely healthy company uh your management chain wants customers to be happy uh they want customers to be happy because they want customers to be paying money they want more people to be using the service you should be aligned with your management chains goals here if your management chain want customers to be unhappy and want the company to make no money then yeah you might ship and make them happy by deploying code uh well by doing whatever but you know you probably shouldn't be at that company you're definitely not going to have a fulfilling experience so I think like this idea that you're making your bosses happy instead of making customers happy kind of presupposes that you're working at a company where those things are intention and I certainly in my experience I've been lucky enough that that at Xander and now would GitHub you know people tend to want customers to be happy um the other thing is sometimes your management chain uh has good reason to make decisions that are going to make customers unhappy and sometimes you don't have the context for that so there's lots of reasons um that are very important to the company I could we could spend the entire rest of the podcast listing them off and that might be a really interesting podcast but just to give you an example um sometimes a company is in trouble and needs to position itself as being in a particular space uh recently AI is a good example of this so you need to ship some kind of box ticking Fe in order to like have a really positive impact to like the success of the company users might not like it that's a shame it would be better if they did but sometimes you need to ship it anyway uh I want to be clear that GitHub is not an example of this like our AI features are actually actually topnotch that's part of the reason I'm here um another really common reason why why why companies might want to justifiably ship something that users don't like is uh security and Regulatory Compliance um if you've ever been at a company that's that's gone for say fed ramp uh you'll have to ship features that kind of satisfy these like these um what is what is fed oh I'm sorry uh it's a a US Government kind of regulatory uh set of requirements I guess that make it so as I understand it the um entities within the US government can use your software y yeah it's one of the now I remember as well one of yeah it's a series of of regulations if you want to do business yeah and there's I guess examples not just this but a lot of other regulations yeah exactly and and some of these things particularly in the realm of security controls mean that you're compromising the kind of customer experience a little bit and doing things that customers won't like uh because for instance you have to be more careful about the way you you manage API Keys uh you might have to make them uh expire over time users really like having their API Keys live forever it's very convenient uh but you can't always do that um those are two reasons there's a million others but just to tied back to the original question like the fact that you're writing code that doesn't make customers happy might make you feel bad as an engineer but it's not necessarily uh counter to the interests of your company and there's lots of cases where I think it's uh quite literally your job particularly as a very senior engineer to kind of suck it up and to prioritize the company's the company's goals over like your own goals uh and and how you might be building the project if it was just your personal project and it didn't have the kind of stakes that working at a at a large company has you touched on on how as a senior engineer very senior and you're staff level engineer it is your job to help the the business to understand the business as well what was the point where you recognize this you learned this because this is typically a learning that I mean ideally it comes as as you grow but what was the point where you realize okay that that is my job now that I'm at the staff or above level uh yeah I I learned this one the hard way um sort of sort of the story of of um mistak so uh I I was a very enthusiastic and very motivated engineer early on I was just happy to be there I was really excited to like not be in Academia anymore uh so I was absolutely absorbing things I was I was I I cared a lot about what I was doing um which was good but it it kind of uh pushed me into a couple of pathways that I think weren't super productive so one of them was um back at zendesk you know or at this point like 6 years ago uh we had this Suite of uh staging tests for the staging environment that were pretty unreliable um for a whole bunch of reasons not worth going into but it was my like holy mission for six months to like keep them green and keep them like up to date and I shipped fixes I shipped improvements I shipped reliability improvements to the to the tests I spent a lot of time and effort kind of like catching up to other Engineers work where they shipped some feature that changed the UI which changed the test selector you know this this was something I cared a lot about um and I felt like I wasn't getting a lot of like rewards for it but you know people seemed kind of happy about it and then after about 6 months um I said something in a meeting where I complained about you know some feature we did that that that kind of broke the tests again uh and my manager said you know something like oh well you know you just don't have to fix them and I was so mad I was so angry I I I left the meeting I think I actually left the meeting Midway which I've never done before or since I just got up and left and I went into another room I went this was in the office I went into another office um and I uh I updated my reime that was what I did was was this feeling of like like engineering craft is not like I don't know the number one or or what was it it just once against like your core beliefs that that time I think I had just put in so much work that honestly a lot of it was like a sunk cost thing I I felt that it had to be important because I'd spent so much time on it in hindsight um I did think like it sort of offended my sensibilities to have tests that were constantly failing that felt that felt bad as an engineer uh yeah I was I was so mad um and then I think a couple of days later like I when I recovered when I kind of felt good again it it it sort of like instilled in me this sense that you can care a lot about something and you can have good engineering reasons for caring about it and the company might not care at all and if you're in that position that's your fault that's not the company's fault you either have to deal with that and like accept that you're not going to get rewards for doing this work that you're doing and not expect them or you need to like reprioritize so you're more in line with what the company wants that that that's sort of what I took from that experience um and it took a couple more instances uh of me being invested in things and and kind of you know suffering this misalignment between what I wanted and what the company wanted but that was definitely like the first kind of impulse and I I think I was a mid-level engineer at that point I don't think I was even senior but that started like the pathway for me to be thinking about these things in the way I do today but how did your reconcile this right cuz like I can feel your frustration on a why you know if you a test Suite like why I have a test Suite which is not green all the time Etc but in and how did you come to understand why either it was not important or not as important as you thought like for for for these staging tests specifically or or what was important what what did the company care about that maybe you missed at the time I think I made the mistake that a lot of more Junior engineers make of seeing something written down as like a procedure or a rule and assuming that it was true so I I had thought that like this social fact that you couldn't deploy with red tests so we had to get the test green every time we deployed which was a lot of work I thought that was like an immutable fact and then sometime after I got knocked back from this and I stopped maintaining the tests and they were R more often of the time uh we had this critical deployment that had to go out and the tests were red the tests weren't green um and I remember we were just like oh whatever we'll manually test the red cases we don't actually need to get the sweet passing we'll just deploy the code and I was like oh yeah when the company needs code to go out the company will find a way to get that code out and these like rules that are in place to like keep code quality up and stuff those aren't hard rules those are like soft rules those are rules that will bend uh if the company has like a clear and present need to get something done around I think that that was kind of like that was the end of of of me worrying too much about about those tests I think yeah and I I assume like I'm just going to assume that these tests were more flaky and they were more less reliable closer to end test which end testing Suites are always a huge headache for every single company that does them and you know there's yeah I always see companies investing a lot and then scaling back and all those things so there there's also that part right we're not talking about like unit sess or T that that that signal if they break there's 100% chance of regression or like 99% chance of regression no n to test operating on a a small set of accounts such that one test could change the state of the account and fail a subsequent test uh Ruby test cppy bar and selenium for anyone listening who's in a similar painful situation so jumping back to shipping projects and in in production in your experience what are things that get into the way of shipping to production emphasis being both on shipping and and production yeah I mean this is another another list of of of things we could go through for the rest of the podcast because uh successful projects there's a narrow path to success and there's a million paths to failure in my experience projects narrow path to success yeah projects want to fail like the default state of a project is to fail like projects only succeed because people drag them Kicking and Screaming to the Finish Line If you leave a project alone and nobody's worrying about whether it's going to ship n times out of 10 that project will not ship um so yeah many many things uh to get concrete um one like common category of things is that it's some detail nobody thought of so just to do a rapid fire of like examples uh from my experience hey we got to like a week before ship and we realized we've forgotten to do the security review or we've gotten to like before ship and we we thought teamx was going to build this thing but actually they they thought we were going to build this thing and this thing doesn't exist or we were relying on this service but this service is actually you know in beta it's not actually deployed yet you know it's slated to go out Q2 next year sorry or hey this service is in production but um you want to run uh a th000 requests a second through it and it can only support 10 it's just not designed for your stuff so it's just endless endless endless like you're coordinating with some other team and there's a understanding about who's delivering what and in big companies like projects need to go out with some on the order of like five to 25 teams coordinating sometimes like there's a lot of cooks that go into like shipping anything in a large company and it only takes like two of them to to be really badly misaligned to sync the whole project so one thing I I've observed at companies like this where it is really complex to ship as you said 5 to 25 uh teams collaborating together which is a lot but again if you worked at these larger um companies the ones that that do you have or similar size it it does happen but there are two ways I I've seen people respond to this who are tech leads specifically one type of tech lead goes like you know puts up their hand like look this is like too much for one person to do uh our TPM was not on on top of it technical program manager who typically coordinated it's like it's not my fault right like you know what can I do and then there there's also the the tees who who like you know who don't have this response and and they actually one way or the other go through it but what is your take on this cuz when you do have such complexity it can feel like a full-time job of just coordinating and you know there there is some push back that I hear from Engineers uh warranted or not saying like look this is not software engineering this is project management and my main job is software engineering I would need someone else to do it uh how when you were in this situation what was your uh kind of response and also especially keeping the mind that you do want to shipping at the end of the day I think there's two coordination tasks that you're sort of talking about there and they can luckily they can be done in parallel or I don't think anything would ship at big companies no matter what there the task of making sure every team is like actively working on the project and coordinating like in communication and agreeing that they're going to ship by a certain date that's what I see as more the uh the TPM or program manager role of coordination uh but there's also in parallel I think a purely technical sense in which these teams have to coordinate and maybe you've worked with different TPMS than I have but in my experience you need uh an actual engineer on the team who is a a full-time engineer usually a senior or staff engineer to have the entire technical structure of the project in their head in order for for something to ship and this is something I said in the blog post and this is something that actually was not controversial everyone seemed to agree which is the project ship because one single person has the technical context in their head they don't ship because like teams agree and manage to coordinate teams agree and coordinate because one person among those teams has the has the whole flow in their head and is able to kind of anticipate problems and answer answer questions because of it and I think a lot of companies kind of formalized that into having a a drri engineer as well as a TPM and as well as all these other things um there's a single engineer that's that's that's responsible for like the technical success of a project and and that's that's the person I'm talking to when I write my blog post uh because that's the that's the position I've been in a lot of times and sort of you learn you learn hard lessons you know and drri being directly responsible individual yeah that's right sorry directly responsible individual that's what it stands for so I I I like how you're you're phrasing this but is it am I understanding it correctly that you're saying that if you're the tech lead on a project or you're an engineer on on your project aiming to be this operating at the staff plus level level regardless of your title like you're like okay I I want to know what's going on then it is your job to understand the whole project understand your team's work how it ties with everything else and understand all the parts how it can go wrong and you know make sure that these are addressed at least and then if that's done you're part of the project should be fine you know there's a lot of coordination communication that you cannot control but you can control making sure do I have this whole thing in my head understanding it and having Clarity and you know like feeling good about it is that roughly yeah that's right that's that I think is what I'm saying the question of whether it's your job is an interesting one I don't actually know like if people typically do this work because it's their job I think they typically do this work because they care about the project succeeding and they recognize that it's not going to succeed unless somebody steps up and kind of thinks through uh the whole thing end to end but whether it's your job or not if you are in this position of shipping these projects it can feel like thankless work but in my experience it is really not thankless work people do notice and people do talk about it and your name will come up eventually as somebody who can be relied on which will lead you to getting more impactful work which will lead you to promo packets and so on at least that's what's uh I've been I've been lucky enough that that's happened to me so I think whether it's your job or not you know it can sort of be like perform for the job you want right not the job you have maybe like I I I'm going to plus one that having been a manager on my on my team I always remembered people who help either they bring up they bring up risk that are on projects but they also bring Solutions saying oh here's the risk here's what I think I should do and when they do it enough times I'm like okay this person they're actually really good at whatever they're doing whatever their their level is and the lower the level right the more ammunition that is for the next promotion and also on on uh calibration meetings for promotions or or performance reviews it was a very powerful argument when an injury manager brought up well this person they they are drisking the project the project shipped on time because of them because they stepped in even though it wasn't their job so it does take I think a while to to catch up but uh managers are are usually they they want to boast about their direct reports at all these occasions which is invisible to engineers I I I I agree with you it's hard to go wrong with this so so talking about te kind of Technology skills or hard skills versus soft skills or politics a lot of what we talked about shipping to production seemed like it's a bit of a soft skill understand the business know the priorities you know make your management chain happy and and these things but how do you think the technology skills fit into to shipping you mentioned they're important but but can you give examples on how important they are or or you know what what comes first right cuz there's I'm trying to figure out if if if in your opinion or your experience is one of them more important than the other I wrote a lot about political skill because I think Engineers typically understand that it's important to be technically strong and they typically don't understand that it's important to kind of be aligned with the with the politics that's why my focus was there but yeah if I had to pick one I would say the technical skills are more important shipping projects like shipping projects only succeeds like you can only ship a project when you understand it end to end and it takes a lot of technical skill to be able to understand a project end to end and to be able to Pivot and to be able to answer questions that come in from various angles um and and one thing that I didn't write about a lot but I think is really underrated is speed in shipping projects not just speed of implementation but speed of reaction speed of uh communication when when something comes in out of left field you need to have contingency plans like right away you need to be communicating it right away when there's a blocker you need to be addressing it right away because all these things add up like because there's so many things that can kill a project like the one like you know grand theme that can kill a project is being too slow because if you take an extra week or an extra two weeks that's an extra two weeks that something can happen you really need to seize your opportunities and try and ship as quickly as possible uh sort of before anything can go can go too badly wrong and you just don't have like theow to do that um without enough technical skill to kind of you know be able to like say throw it end to end which is a lot yeah and reflecting on this I'm just thinking back of the best tech leads I remember especially when I was a manager on my team and you know they weren't the ones who so whenever a I think you sol with the problem oh here's a problem this team says they need another like two weeks to build this stuff the tech lead that was kind of okay was like okay okay uh let me talk to them see if we can you know get it down to week and a half or a week if we can rep prioritize the awesome Tech lead was like why what what is keeping them up let me I just looked at your code base and I don't really see a reason I actually talked with their engineer and they said like they just need this like small ticket done uh I'm I already I already wrote the diff it's it's ready for to line for them we'll we'll get unblock yourself for tomorrow and like it's this as as you said the person with the technical skill who can actually go there and understand and Challenge and also sometimes especially if you're working at a company which has a internal open source policy meaning you can make changes to their code base like yeah just just make the change and then the team it's it's it's it's incredible how technical strong technical skill with the ability of getting things done can really just move things You observe similar things yeah no i' I've seen that too I've seen that so often um one thing I've seen is like the ability to kind of come up with a technical demo kind of moving mountains like one demo of like hey look at this thing happening can cut through uh weeks of conversation about whether it's possible and if you just go off and produce that you can really kind of blow people out of the water uh one thing I say a lot is that in in the context of like a large company which which of necessity has many non-technical managers and non-technical stakeholders being technically strong if you work on software is really a superpower because it makes you connected to reality it makes you connected to kind of the real world to the actual product that you're building uh and it's sort of like a lot of the work these other people are doing um about around strategy and marketing and figuring out what needs to be built uh that only like that's a huge multipler on like a tech company but it only kind of starts multiplying when it locks into somebody with the technical skill to actually make it happen so if you're that person just by like having a lot of technical surface area you can like unlock the the sort of multiplicative factor of like all these other people in your organization just by being willing to to answer technical questions to make technical demos and to implement things uh quickly that are bothering them that they think are important like you you can really move an an organization just by being technically strong if you're willing to put that technical strength in the service of of of people who have like thought hard about what needs doing but don't have the technical skill to do it and I I see a lot of people who who are so technically strong but are pouring that technical strength into like making the code a little bit cleaner or or writing tests that are a little better and that can be really useful but I think you could be doing so much more can we Zone in on to to this specific thing because I hear what you're saying is if you are technically strong and you can Surface in a in a way that is a lot more helpful to the business visible to others but what I'm seeing seeing as a lot of most software Engineers are technically strong and as you say they might be pouring it into less visible things what what might be some ways that I can you know like like surface this a little bit more I think the most impactful piece of advice I can give is just to go out and talk to people um and that's a little spicy sometimes because going out and talking to people uh sort of circumvents some of the process that these large tech companies have in place by which like communication goes through from Project managers to designers to Engineers but if you're an engineer and you you have a lot of horsepower and you're looking for a place to apply it and you feel like the the backlog on your team is not the most impactful place uh Sometimes the best thing you can do is to go and talk to your skip level go and talk to your project manager your product manager your designer and just say what like what do you think needs doing what do you think uh like I can work on one thing that I I do see Engineers do this and and this is maybe not the most like politically Savvy way to use their superpower but one way have seen Engineers use this superpower is to go to support to the to their support counterparts who who raise tickets and to kind of invest that horsepower into making things better for support fixing bugs that like waste time for support this is not the kind of work that I think is is is going to be super impactful on a Premo packet um but it is worth doing um and I think a lot of Engineers who already do that for support could be pretty well served by taking that and also doing it for design and and and product instead of waiting for the these design and product plans to to percolate through to to epics to issues to scoping because a lot of the time there there are quick wins that you can just knock out quickly um but I like this it might make your manager um unhappy and it it might it might not be a sensible move depending on the climate of your of your team to kind of go around the process like that yeah I I I like this but I I I do think that if you're unable to go to customer support and fix a persistent issue you know like on the side at all like over a year or two years then like it is a good question of like how effective can you be like I've actually noticed that some of the the most effective Tech leades did do not on a regular basis but every now or they were able to do this so yeah I I guess if if if you're trapped and looking for inspiration this could be one way to try and also just figure out like what the response is right if your leadership team is like oh you should have not done that like well do you really want to be work be working there when you're like fixing doing something delightful on top of the job that you have is uh is taken like that probably not again it depends on the external climate and all that but uh it can be I I like the suggestion I I think it's worth trying to understand why if your leadership team comes back and says you should not be doing that cuz yeah maybe they're wrong but maybe they're right maybe there's some like existential problem facing the company that you ought to be working on instead like a lot of the time like Engineers can spin their wheels like it doesn't matter like a a problem affec in the company can be as serious as it gets and there can still be Engineers spinning their wheels because that that team structure doesn't ow them to act I've seen that happen yeah it's it's a good question on who's who is it down to obviously as an engineer especially when you're senior above you should understand that but also if if if you're not as experienced and the management team or your manager is not efficient in telling you like this is really important the the the company is on the line or our product is on the line well yeah that's a little bit on them as well that's true speaking of trust with your leadership team as a tech lead it's really important that you you do have trust with your leadership team and this sounds pretty vague uh but it is important what are good ways you've seen of doing this yeah so by far the best way to like main TR maintain trust with your leadership team is to ship is to deliver and is to deliver consistently so if you're that engineer we talked about earlier who's like saving projects and drisking projects and making sure things get across the line that builds trust um at minimum it it it builds trust uh in in leadership that you're technically strong you know like and and that goes a long way that that gets you put on higher profile projects that gets you kind of called in when something's important and and kind of starts this like snowball um but in order to do that in order to ship on things that like your management team cares about uh I think you do have to like maintain the capacity to do that so what I what I mean by that is if if you're at 120% on your team's jur tickets and you're just burning down and burning down and burning down um you're not going to have the capacity you need when something comes in that that is more visible and is more kind of higher higher profile so I do think if you want to like maintain this trust with your leadership team one big part of that is is is to kind of keep your regular working Cadence at like closer to 80% or 60% and then kick it up to 120 130 uh when something does come across and that way you can kind of save your powder for when you for when you really need it um I think that's really important uh the other thing is is is to try and communicate like them so I think a lot of Engineers kind of get this sense that that talking uh like Senior Management is is this kind of slimy thing is this like non- likee engineering value like you're not a you're not a hacker if you use words like Circle back or or synergize um I think yeah maybe that's true but if you want to build trust with your management like you need to kind of play the game a little bit and you need to like learn how they communicate um and and sort of communicate back to them a little bit like that uh just to make that really concrete um if you're shipping a project the most important thing almost like more important than technically delivering is understanding what the project is for so uh a project can be for um box ticking for an audit it can be for attracting new users it can be for extracting more money from existing users uh it can be for satisfying some VP or CEO's like dream of what a future product could be there's so many reasons and all of those mean you have to work differently you know products for new users have to be really polished products for like large Enterprise users don't have to be polished at all but the requirements are usually really inflexible stuff like that but you have to know what the what the project is for but management won't tell you that unless you can communicate uh in a way that they know how to communicate back to so if you to your manager and you say hey can I build a really crappy UI for this that is terrible to use every single manager in the world particularly in writing is going to say no of course not you have to build something really nice um because they're afraid that you're going to go and build something awful and then point at them and say hey they told me to do it like you know so you you really do have to like to pick up this like crucial information you have to be able to communicate this like you know is it okay to make trade-offs with user experience instead of saying is it okay to build a crappy UI like you have to be able to talk to talk like that I it's a hard lesson for some Engineers to learn I think but it's it's really important yeah this is very interesting because it triggers memories for me where I was the tech lead on a project which it it didn't make sense for us to build it and I talked with my product manager and you know she she confirmed like no it it doesn't make sense in fact like we're only doing this because there's an executive who struck some sort of deal with this company we needed to integrate something something and she said like look like it makes no business sense try to talk them out of it we got details but it came down to something political honestly that they they they were saying oh we have a deal with this company we cannot not do it Etc so it became a little bit ego and she said like look here's what's going to happen we need to build it there's no way out of it we need to do it as fast as we can uh we're going to retire it in a year uh we're going to have very very few users of this thing relatively so and she the PM wasn't telling me she's like just these are the constraints so as a tech lead I kind of turned to the team and told them like look here's the deal we need to build this it we can do it quickly we can skip some of the the test it kind of needs to borderline work but we don't need to pour much complexity and this was a complex project so in the end actually helped us deprioritize a lot of the edge cases it was like don't care just add an error message don't do this and and in the end actually what happened is we shipped it we had like a ridiculously low number of users as we predicted and and we retired it in 12 months later the code was deleted and it one on one end it was one this box scking exercise which didn't make sense but we had to do it and the shorter we did it the better it was it was the first time I I realized that you could do stuff that doesn't really make sense but it was a necessity and we could have argued a lot more uh or dragged our feet about it but in the end we're like you know what let's do it quickly ship it uh thumbs up I'm really impressed by the the the trust relationship that you had with your team and and with your product manager that you could just be upfront about it and say hey we're we're doing this to check a box we can trade off user experience no problem like that's sounds like it made things easier it it made that this was a rarely like it was a really like close relationship with the the product manager we were like really kind of one unit it also helped that we were in a different office location from HQ I'm not sure if at HQ it it would have been said this clearly because then people talk and you don't really you know I think it kind of undermines you know like leadership morale when you say like basically the leadership like we don't agree with leadership and we're right to do so and actually the the data showed that we're right to do so but either way so one thing that is changing software engineering a lot is Gen AI uh it's it's now in a day-to-day tool chain you're clearly you're working on some tools that I know you cannot talk about uh at at GitHub but a lot of software Engineers are using tools like GitHub Cil or or some competitors to make their their work easier but I I'm wondering because you're working on both this tooling you're using it you're also seeing what developers use what parts of the software engineering job do you think are becoming easier with geni tools specifically with with getting things done right yeah um well one thing I I said earlier was that the the sort of cardinal virtue of of of shipping projects is is speed is getting things done quickly because so many things can go wrong when you when you delay uh and and one thing gen is really good at is speed it's it's it's getting you like 80% of the way and it's getting you like volume kind of for free um and one like I'm a huge believer in like gen for programming I I use it every day I use it all the time um I use GitHub co-pilot I I I chat to co-pilot in the editor like when I'm on personal projects I I use chat GPT and it's just the level with which you can like go from zero to something is is so good uh I do think like it's the kind of thing where I think Engineers that tend to work on one code base and tend to not work across a sort of broad spectrum can sort of underrate how powerful it is because it's it's it's not going to be better than you at writing like Ruby and Ruby on Rails in the codebase that you've worked on for 5 years you're going to be better at it than that sure but it will be better at you probably at working on like goang in your company's load balance or repo that you haven't touched and are unfamiliar with like it'll be better at you like in all these domains where where you don't have experience and because so much of shipping projects requires like understanding code and writing usually small bits of code in in repos that you're not familiar with I think like gen can kind of be this real Difference Maker there where it can you know you need to build like a page in your company's internal admin tool in order to ship a feature that probably doesn't have to be super polished that probably doesn't have to be like you know 100% there ji can probably do most of that for you you know that's that's the kind of thing where like you can save a day or two um and that kind of compounds over the course of of a of a project so uh we had Simon Wilson on on the podcast who is software engineer who's using a lot of geni and one interesting thing he said that gen AI makes him a lot more ambitious so is it fair to say that with geni if you have it at at your workplace which most companies will have at this point as a software engineer you can just be more ambitious and do you know either sign up or start doing these projects or just contribute to project which you know before these tooling you might have been like n too much time I'm not going to be able to get my head around it it's a codebase that I I've never touched I've never programmed this let me just leave it alone I I personally haven't noticed this huge increase in ambition at work uh just because like typically the hardest parts of like complicated software that has to be correct the hardest part is is is that extra 20% uh on top of what AI can give you so I I would be reluctant I think to kind of swan into to let's say the load balancer like github's load balancer and say Hey you know me and co-pilot like built this built this feature that I think is really cool like you know like I wouldn't expect a super positive response where I have found like the chat GPT or like gen lets me be more ambitious uh is is personal projects for sure when I'm working on like stuff by myself or learning stuff by myself I almost feel like I can learn anything I almost feel like I can I can get into any area uh like so much quicker than before just because the ability to ask questions of of um of co-pilot of chat PT the ability to like just ask endless questions of of someone that never gets tired of the questions that always gives you like considered answers that's that's sort of the dream right when you're when you're learning something new so I I love it for that kind of work and I love it for like prototyping and spiking things out and you know I sat down the other weekend and I was like what would it look like if you had five models playing Texas holdom against each other who would win um and I you know under a day I had that I had that working and I turns out it's GPT 4 like it's just uh it's just not something I would have done I wouldn't you know building a a a Texas hold and poker like simulator to like back this back this app it's just you know I don't think I would have bothered doing it because it would have taken you know multiple days it would have taken a lot of effort but with llms you know I just kind of had it up in half an hour was great and this is a little different question but related to this do you see parts of software engine that are getting actually harder as a result of more gen AI being used more people using gen a or or or just because of these tools themselves uh yeah I mean I'm I'm in a great spot to talk about this as somebody who's worked on co-pilot and is working on GitHub models one part of software engineering that's getting harder is building these gen tools which is increasingly becoming a large part of software engineering uh llms are really hard to make into into working software uh the text text modality is hard to work with hallucinations is hard to work with the fact that it's all moving so fast it's a really fun area but man is it more difficult than um than than than the work I was doing before like on on so many different levels like even just the fact that many people you work with are kind of struggling to keep up as well you just don't have this kind of Baseline level of understanding that I did say at sendes when I was working on apps and we'd had apps for years and everyone had been working on App Stores for years everyone knew what an app store was now I'm at you know GitHub working on llms and people are kind of learning as we go like what are evals what are what are logits what does it mean to put like a a grammar enforcer in front of the sampler when you're when when you're like you know enforcing this Json schema how does that work like so many so many things the learning curve is just is just so steep so that's definitely getting harder if you actually have to like build this build these tools man yeah that's harder than than than standard kind of engineering crud work well and I guess more and more of this over time I would think it would leak over into software engine the sense that these days if you're a backend engineer you're kind of expected to understand how like the front end works at a conceptual level you know if need be okay like you're probably not your job but like every now and then you should be able to touch a little with CSS or HTML same way I I would expect you know like the term full stack engineering is exploding for good reason and I would assume that in a few years software Engineers understanding things like you know what is a rack pipeline or what does it mean to train a model versus to fine-tune it again you probably don't need to do it but just knowing some of these and and increasing more Concepts will just be part of the day-to-day job especially as more companies you know like incorporate llms into their different projects yeah for sure uh assuming we still have rag pipelines assuming we still have fine tuning like I think some of those things are definitely still up in the air I mean even two days ago open AI released a new a new way of fine tuning models that isn't low rank adaptation so who knows where it's going to go EX exactly well it's an exciting space yeah sure is uh another area that is very challenging and interesting is working across time zones you you're doing this you're you're based in Australia so uh I'm assuming you work with teams who are in vastly different time zones what is your experience uh working with teams who are in different time zones what what works well and what doesn't yeah um well one thing I I do want to say just on the the topic of me being based in Australia is uh a lot of people got angry about about my post about shipping and and and said that you know they're going to plant their feet and be really disagreeable and tell their manager to screw off and and and do whatever I got to say that the calculus on that looks a little different when you're an Australian engineer and you don't live in San Francisco and you don't have you know 20,000 jobs that you can walk to from your from your front door so I think one thing about being in a different time zone is is is I think you sort of don't have this uh there are some areas in which your experience is is not like your us peers which which definitely affects how you work and maybe makes you a bit more prone to kind of do what the company needs as opposed to planning your feet um but in terms of my my direct experience working across time zones as zendesk I I sort of worked with like teams in different time zones but my team was kind of collocated which was nice and then I had a bit of a route Awakening at GitHub I was hired as the first Australian engineer uh on my team yeah just me uh with with the goal to kind of grow this APAC group uh but I think 2 weeks after I was hired uh there was a hiring phrase um as there were many hiring phrases around that time and so for for 9 months uh it was like five us engineers and a US manager and me in APAC doing doing my doing my ape thing and and that was a that was a very interesting experience um it was a it was a very lonely experience in some ways but but there were there were real upsides that I think kind of hard to anticipate if you haven't been in that position one of them is that like my mornings were really busy because everybody was like closing up their day and I had to catch up on everything that had happened but my afternoons were almost all clear so I was in this position where like every afternoon I could do deep work and I think that really set me up uh for success I I do think like I'm a pretty high agency engineer um and GitHub was lucky uh that it got somebody like me who is happy maybe with a little bit less support I I do think like being the solo engineer for nine months in a different time zone like not for everybody y so now having seen you know like both collocated teams working with with with different times is also you working by yourself when it comes to remote work like talking to engineering managers or people who are organizing this like what is the strategy that you've actually seen work the best and and and most efficient am I incorrect to assume that it's going to be the collocated teams where there there's multiple people yeah definitely uh definitely don't just hire one person in a time zone if if you if you can right now we have like a sort of squad of of APAC time zone engineers and a squad of us engineers and I think that works well um one thing I think that's really important for for managers kind of trying to do this and I recommend all managers do this uh I think every us company should open many hiring RS in Australia please that would make me very happy but one one thing that I think is is is a really good benefit of this approach and something we've done at at GitHub the whole time I've been there is to sort of double down on the ability to have people working uh follow the sun 24 24/7 so many many many times when we've had critical projects or like critical bugs we've had the US Squad do a bunch of work and then hand off to the Apex Squad and then the Apex Squad hands off to the US Squad and the result is that like it sounds like a silver bullet uh but you can basically twox the speed um when you're when you're delivering some things things that where the the handoff cost is is pretty minimal like like bug investigations that kind of thing and we've we've really been able to deliver I think some projects that we just could not have delivered if we've been restricted to working in the US time zone only um users as well when a user reports a bug in the afternoon and they go to sleep and they wake up and it's fixed that feels really good and that's that's harder to do uh with na Engineers they kind of have to work really late for that to happen but we sort of got that for free like anytime an important bug came in just got given to the APAC engineers and if we were able to solve it by the end of the day it was just this really good user experience of like wow they like literally fix it overnight like I I wake up and it works that's that's amazing um I do think one way in which GitHub is really well set up for this kind of work is that it's very async first and that's super important for like managing these these two kind of time zones because you can't all be in meetings you need to be producing kind of written artifacts and demos these sort of things that can like speak for you and and and do work while you're asleep um so definitely like uh you need to kind of hire people who are comfortable both writing and reading um documents and and and demos and that kind of thing I think that reading part's important I think it's comparatively easy to hire people who are happy to write demo to to like write memos and write rfc's I think it's slightly harder to find people who are happy to read rfc's but that's that's almost more important on these um on these on these distributed teams like people need to be communicating async for it to work and you're now in a I think of an increasingly enviable position in the sense that you are working full remote at a at a team and a company that does support full remote and we're seeing data that the percentage of these uh job uh options is unfortunately just going down a bit who knows if if it'll stabilize or or go up in the future but right now it is going down so and and the the willingness of Engineers wanting to do that is is not going down so I I'd like to ask in your experience what are the practices and the traits that help you excel in a full remote environment and what what do you think you needed to uh to actually do this because as you said you know you worked for nine months by yourself and you mentioned that it's not for for everyone yeah I mean to sort of sound like I see some people talk on Twitter these days I I think part of it is that I'm built different um and I mean that in in in like a maybe you know uh not necessarily entirely positive way but like some people I think need human interaction need to be like talking with people they work with uh whether it's on Zoom or not I think I can go a long time communicating with people iyn that's just kind of how I'm put together which makes it a really good fit for me uh sort of first and foremost that's that's sort of how I've like because I I know a lot of good Engineers who I think could excel in this in this role kind of flame out because they can't stand being by themselves and that's just not something that's that's that's that's bothered me so I've kind of had that obstacle like removed which is which is which is really nice um of course then you still have to excel uh and I think you know what I focused on what I've done well you know we've we've talked about it you can go and read the blog post like but the the one sentence is you know if you're in this enviable position and you want to keep it you kind of have to align with what the company needs from you and you have to take what your job is really seriously um I see a lot of Engineers who like have their own goals uh for how they want to work you know they they want to have all their code perfectly clean or they want to always have unit tests or they want to work on like front end stuff only um and they sort of push those goals up against the company's goals and that's fine you can you can do that and you can make those those decisions um but if you're in a spot that you really want to keep uh like working fully remote for a company like GitHub that I love you know I have a lot of reason to try and kind of absorb the company's goals and and and and to try and make them my own and I I think that's broadly been successful and has has kind of put me in this position where like I'm able to keep doing this and hopefully I can keep doing this for some time to come I I I I like this reflection especially on the part of you can go a long time without communicating async I guess a lot of this will just be learning but is it is it safe to say that did you know what it would be like to work proper full remote before you did it or or was it like just you know something that you needed to go through to see because as as I understand at zendesk it was more in in office and and together with the squad and this was the first time where it was a proper formal job for you well recall recall the timing right like I I I moved to GitHub in 2021 um and uh Australia Melbourne had had one of the uh one of the longest one of the most extreme lockdowns uh in the world so for a decent amount of time before I moved to zendesk I'd been working like relocated in a time zone with people but I'd been working from my house I I hadn't been working from the office I hadn't seen these people so I think that was pretty unpleasant in a lot of ways but in hindsight it did kind of give me the confidence that like I could do this remote rle at a company like GitHub and that I could kind of I sort of knew that I could work without going into the office and before I I wasn't so sure how I would cope with that turns out I cope really well but I don't know if you can anticipate that ahead of time you kind of have to try it and see on the flip side a lot of us who have worked throughout 2020 2021 have likely had that experience and I think people going to reflect on like how did it feel like could you do this for longer uh did you actually Thrive because some people thrive some people hated it yeah for sure I I do think it was very different being like in the same time zone as people from being alone in my time zone that that was a big shift that was something that I I think I got I got lucky that the that I ended up being a good a good fit for that because had I known that was going to be the case I I might have been a bit more skittish about about taking the role I probably still would have done it oh yeah well some things you just got to wing it and sounds like that's what you've did let's close up with some rapid questions so I'll I'll just ask a question and just like bird out whatever uh it comes first to mind what's an interesting framework or Library you've recently used and what did you like about it um I mean this is a silly answer but Ruby on Rails man I I love Ruby on Rails it's not new it's not sexy um but I I am a huge Ruby on Rails believer um and it's just it's the most programmer friendly way to build large web apps uh and its flaws are honestly ining to me is GitHub still using rupon rails I heard that it was started off rupon rails or it migrates to rupon rail so so internally it's being used or this is just used on the side oh no no G GitHub is still like in very large part still still a ruby on rail shop we run a lot of different things for a lot of different Services you know sometimes Ruby on rail doesn't make sense but if you go to GitHub and you go through github.com and you interact on GitHub almost every single thing you do is going through Ru on ra still that's a good fun fact reminder yeah uh what is a book that you would recommend uh and why um well one Tech book is actually underneath my mic right now it's the little book of uh learning by um by by France fur which which was sort of something I picked up when I was getting into llms and I wanted to have a bit of a better understanding of like uh the the sort of maths that was that was backing them um and it was just a really nice like short prer on on on how it worked didn't go into too much detail it was it was excellent um but to recommend a book that I like I read every year I I got a shout out the name of the Rose by Alberto Eko uh I read it every year it's an incredible book um about an Italian Monastery in the 1300s where somebody commits a murder and then they have to do an investigation I think it's a very funny book but it's a book about people doing their best in complicated systems which which kind of speaks to me and maybe some of the things we've talked about so yeah name of the Rose by Umberto EO go read it well thank you very much for sharing all these details and being on the podcast no problem was a lot of fun thanks to Sean for the Deep dive into the topic of shipping projects as a tech lead with the focus on how to do this at midsize and above companies if you'd like to read more about Sean SS on software engineering check out his website or connect with him on LinkedIn both Linked In the show notes below if you've enjoyed this podcast please subscribe on your favorite podcast platform and on YouTube and if you're interested in more tips on how to lead a project as a software engineer check out the Deep dives from the pragmatic engineer Linked In the show notes below thanks and see you at the next one