Distributed Engineering Management Best Practices with Justin Cormack

On April 6, I hosted the 3rd session of a monthly Company Building webinar series for founders and operators of developer-focused and/or commercial open source startups. The goal of this series is to share tactical knowledge -- the nitty gritty know-hows that can make the grinding experience of company building a little easier for entrepreneurs -- from experienced operators who are still in the game. (Our 4th session is on June 6, 4PM US Pacific Time. Discussion topic: Developer Relations best practices with Kevin Chen, the 1st Developer Advocate at Kong. Please register HERE).

This session is with Justin Cormack (CTO of Docker) on the best practices of managing and scaling a global engineering team.

To make our wide-ranging conversation more useful and digestible, I edited and condensed the session’s transcript into the following topics (jump to the section that's most relevant for you):

• Justin's entry into tech
• How Docker became a more distributed company
• Best practice: coordinating engineering by time zones
• Ideal team size and synchronous vs asynchronous work
• Collaboration between engineering and other company functions
• One mistake Justin hopes all entrepreneurs can avoid

You can find the audio/video version of this conversation on the Interconnected Voices YouTube channel or watch it below:

Be sure to check out transcripts and audios of Session #1 with Matt Ekstrom (ArangoDB, ScyllaDB, MongoDB) on early-stage sales and Session #2 with Clint Smith (Discord, Datastax, MySQL) on global company operations. Bookmark the "For Founders" tag for future content.

Justin’s Background

Kevin: Justin, I really appreciate you taking the time. I know it's late into your day in the UK, but thank you for joining us.

Justin: Thanks for having me. It's really nice to see you again. I really love your weekly emails. Interconnected is one of my favorite things to read and I always look forward to reading it. It's really, really interesting. And there aren't other people writing with the same kind of breadth and it's one of the things I really appreciate. I guess I've got a weirdly broad background too, in many ways, so it kind of resonates with me.

Kevin: Well, thank you so much for that, Justin. I appreciate you giving Interconnected a shoutout, but today it's all about you! So let's actually start with that background. I know you've been at Docker for many years, both as an engineer and now rising up to become its CTO, but I'd love to hear, first of all, your pathway into technology. And then we can dive into some of the engineering management topics that we will talk about.

Justin: So I think my route to technology was fundamentally a lot tied to my mother. She was a math teacher and always loved helping us understand things. She did computer programming a long time ago when she was younger, so she introduced us, me and my sister at an early age to computers and technology. And that's kind of where it came from, but I was never quite sure what I wanted to do. I always had broad interests, so I ended up tangentially leaning towards technology, but not jumping into it full time. I worked in finance for a while and investment management, but gradually moved more and more towards software.

I’ve always been interested in business and what’s going on there as well, so I did a few startups. I did startups in IoT, LED display, public advertising a long time ago. I ended up getting interested in Docker indirectly, because I got interested in working on unikernels, which are basically trying to encapsulate operating systems at a more developer focused level. And we did a startup, Unikernel Systems, that was acquired by Docker five and a half years ago now. So that's how I ended up at Docker.

Docker is a really fascinating place. The people are great. I always like working there. It's a really interesting thing from, you know, how do we take open source and turn it into a business? How do we take something that's got mass adoption and created a whole market, and turn it into a business, which everyone else feels they can sell off of as well? And it's been a really interesting journey through that.

So I started off working as an engineer based in Cambridge, UK, which we'll talk about the international things a little bit later on, but basically I worked on Docker desktop. That was our first project we did, which was great fun. We shipped that so it became really successful.

And I evolved through lots of different roles in Docker, working across a lot of different parts of the business. As you mentioned in the intro, I worked in security for a while. Security was always something I'd kind of worked on the edge of, but hadn't really jumped fully in. And it was really interesting to work on that. I mainly focused on product security, but really running all of the security at Docker.

Then Docker restructured at the end of 2019. We sold off the enterprise Kubernetes business to Mirantis and focused really on the roots of developers and developer needs and the initial adoption of Docker, which was all through developers. In that role, I was really the most senior engineer at Docker. By then I kind of started moving towards this sort of CTO role. At the end of the year, I spent a lot of time talking to our investors, Peter Fenton, in particular at Benchmark, and talking about the directions that were interesting and what we need to do. So that's how I ended up moving into the CTO role early this year.

Docker’s Journey Becoming A Globally Distributed Team

Kevin: That's fascinating. I want to actually start with your beginning journey into Docker then. I didn't know that you came into Docker via an acquisition and that's kind of interesting. This was back in 2015 or so, is that right?

Justin: Yeah that's right.

Kevin: So back then Docker was an American company but with some European roots, right? Its founders are French and so on. And you have always been based in the UK. What was that initial, entry way into working remotely or in this globally distributed engineering setup as a acquiree of an acquisition. Would love to hear more about that experience.

Justin: Yeah, I think it was really interesting and if you follow on from there, it's affected a lot of how Docker changed down the way in terms of international things.

So we had the advantage of being quite a big team. I think there were 10 of us at the time. So it wasn't a sort of two-person acquisition or anything, and that gave us a bit more leverage. We were quite clear from the beginning that we didn't want everybody to move to SF. For pretty much all the previous acquisitions except one, everyone had moved to SF.

So we were a sufficient nucleus that was the beginning of Docker's real international spread. It was kind of fascinating though, because I remember at the time they really weren't sure how this was going to work out. And I think we actually had in our contract that said that someone from the team should be in SF at all times. That was sort of a hedge, like you’ll be in America, but that’s not how it worked out. Until COVID, I spent a lot of time in the US. I probably spent more time than the other people. I used to travel every two or three months to the US and spent a few weeks there.

At the time Docker was still mostly SF but that has changed over the years. You mentioned Docker was initially a French company and so fairly soon after we were acquired, in maybe late 2015, or early 2016, we set up a Paris office. We've always found it easy to hire people in France because Docker has this reputation of being a French success story, which is really interesting. So we've had a lot of people in the Paris office who were initially in SF. It used to be quite French speaking on a Friday afternoon in SF: you'd hear a lot of people speaking French, that kind of thing. But now most of the French people are at the Paris office. That was the beginning of really becoming a globally distributed company.

Best Practice: Coordinating Engineers By Time Zones

Kevin: I see. So let's get into that then. It sounds like you, or your previous company was the reason why Docker became distributed, almost out of this acquisition. And now it's becoming more popular, more generally practiced, or even fashionable these days for new startup companies to just have a corporate entity somewhere but you hire from wherever. GitLab, which is a company we're both very familiar with being kind of the prototypical example of that in some ways.

How has your engineering management practice evolved since you've risen up the rank? What do you do to manage all these engineers from all time zones who may even speak different languages, but are all working from the same code base or computer language? What are some best practices you've gleaned so far?

Justin: So it's evolved a lot over this period. I think we kind of had to learn to do it, and we weren't very good at it at the start. Also there's a big change pre and post COVID. Until COVID we were just starting to become a remote work company.

Some people have always been remote at Docker, but it was a small number and we were starting to have a lot more remote working. We had the SF office and a Palo Alto office, and people were moving to whichever one was more convenient or working from home. More people even before COVID were moving away. And we were hiring people outside the offices.

But when COVID happened, we just got rid of our SF office after the Mirantis split, because we were downsizing. And then we just talked to staff about offices and working and gradually people came around to getting used to working from home. And we shifted now into a situation where we're just rolling off the last places for offices and going fully remote. So there were two transitions really, one was to distributed and one was to fully remote.

So the distributed one culturally felt a bit different because we still had offices and we still had these clusters of people always with local management. So we'd have engineering management in Paris, engineering management in Cambridge, engineering management in SF. And there'd be this kind of close knit teams generally working together.

We had times when some of our teams were very split between people mainly working across time zones and not with people in the local time zone or office, which was very weird. And we've really gravitated towards the model where we try and keep coordination within particular time zones but also ideally local offices. I think we're now more focused on the time zone as a unit rather than a place, but time zones are difficult. France to the West Coast is even worse than the UK to the West Coast - it's another hour off. And it's difficult for people to spend enough time together, to work really, really well together. It can work and open source projects often do that, but coordinated working on individual close-knit code bases and products is really hard.

So we're doubling down on that small connected team. We're moving now to teams that are very self-contained, ideally geographically localized, maybe around five engineers, a product manager and engineering manager, and UX. Those small teams are ideally in a single time zone. Really aligning them to solve particular customer problem areas.

Kevin: Got it. I mean it is kind of a wild, wild west. I don't think anybody really has this thing down as far as, “this is how you're gonna manage 100 or 200 engineers scattered across the globe.” A lot of companies are figuring it out as they go. And it sounds like the way you're approaching it now is actually using time zone as the unit. So if I understand it correctly, is it that you will intentionally cluster say a few engineers that need that close knit work or in a particular unit to be in the same time zone, so they can coordinate in a more synchronous fashion?

Justin: Yeah, absolutely. So far we've been restricted in which countries we're hiring, because we were limited by where we had companies. We're going to move to less restrictive hiring and use a global hiring company so we can hire anywhere now, but we were kind of stuck.

We were across the US and Canada. We're in the UK and France, which are an hour off, so pretty much the same time zone. I think we're going to be cautious in expanding the time zones above those areas for now, until we're larger. We already refocused Docker into 70 people now. So keeping that small, tight knit thing is important. I think it'll be a little longer before we're going to move into the Asian time zone probably, but we're going to expand the countries we hire in the existing time zones we have, which is a fairly large block of the world at the moment.

Ideal Team Size; Synchronous vs Asynchronous Work

Kevin: Totally. So to get really tactical now. Based on your experience so far, organizing what I would call engineering pods or small engineering teams along the same time zone, how many engineers have you discovered is the ideal size for a pod like this to work on something together where synchronous work again is a preferred way to approach it?

Justin: Around five engineers, plus the engineering manager, product manager, and UX. We've had larger and smaller teams before, but that seems to be kind of optimal where we're trying to aim at at the moment and see. We're really trying to get the focus where the team has all the resources it needs. We've had central functions for design before, but we've had bottlenecks with central functions and we'd like to really get every team their resources.

Kevin: And what's your opinion now on this balance between synchronous work and asynchronous work? These terms get thrown around a lot, about engineering practices, especially in open source companies. And for those of you who aren't too familiar with the jargon of it all, but if you do any programming, you will know.

Asynchronous just means somebody does it somewhere and then it can wait and somebody else can pick it up somewhere else. And you can do this without really coordination at the same time, while synchronous is the opposite, where you do want to talk through certain things and work on something together at the same time to get the results.

What's your thought on the balance between these two approaches in terms of growing an engineering practice that serves the rest of the company well?

Justin: Yeah, it's difficult because there's a lot of different kinds and forms of communication. The traditional open source model of mostly communicating through issues and pull requests is very async, with perhaps some meetings to set direction but not very often. This tends to be very different from the product-led thing, the pair programming and sprints and stand up meetings and that kind of close-knit communication. So I think there's different kinds of communication and models and they work for different things at different stages.

We are an interesting company in that we have a lot of different kinds of work we're doing and software we're working on. We have open-source work that's in the community and fits more with the very async model. And then we're running SaaS services where people are on call, which is a different sort of communication.

We're kind of unusual in the variety of things we do as a company, which makes it more interesting, but also makes our communication structures a little bit different. I will say getting people better at writing things down is incredibly valuable, whatever you're doing, and writing is in a sense async. But it’s not necessarily about what form of async communication you use, but how well you're doing it. I'm a big fan of the Amazon style writing things down before meetings. We've been moving well towards that culture. We had a bunch of people who are ex-Amazon, who helped introduce those writing cultures to us, which is really valuable.

It's interesting when you can change from presentation/slide cultures to writing cultures over time. And I think being fully remote has changed things as well, because lots of people have white boarding and synchronous meeting cultures that are actually harder to do remotely. Whereas writing is actually a culture that works very well across different scales and sizes of communication. It helps people on board as well. So it has a lot of advantages.

Kevin: Absolutely, yeah. I’m clearly, as a newsletter writer, a big fan of writing. I do think it's probably still one of the most scalable forms of communication, that could both spread out very quickly, very easily, but also get a lot of people on the same page at different points in time. Like you said, probably one of the better, asynchronous ways of communication.

Justin: There is one thing about writing that's actually interesting. It's about language. So people who speak different languages generally often prefer reading to listening, because they can read it at their own pace, but people do find writing in their non-native language more difficult. So there are some nuances around that that are interesting.

Kevin: It's interesting you mentioned that reading is better, for say non-native English speakers, because you can take your time. You don't have to listen to somebody else talk about what we're doing right now, which is a lot harder to absorb in real time. Any advice or thoughts on making writing more accessible as a tool of expression within a company and helping non-native speakers of certain languages write better?

Justin: I think a lot of it is practice. I think writing things down in their native language first often helps people. I actually had an interesting conversation with someone at Amazon about this, and Amazon does not have any English-only rule with that writing policy - you could write it in any language. That has problems potentially too, but it’s an interesting slant on it.

One thing that I've come across with open source projects is that sometimes they're divided by language. I remember the first time, I was an early user of OpenResty, which is a lower, larger plugin for Nginx and that was originally in Chinese. And I used to read the mailing list with Google translate at that time. It was interesting because it gradually transitioned to more English. The author moved to America, but there were communities then that were divided by language. We’re still seeing that some of the cloud native projects a little divided by time zone and language. These things are just difficult.

Collaboration Between Engineering Teams and Other Functions

Kevin: I want to move on to a couple more topics before we open up to Q&A. I know we already have a couple of questions in the Q&A tab here in Zoom. One thing I want to talk to you about is, of course, Docker is a very technical company. The product is very technical, but obviously engineering does not live in its own world, right? It has to work with the product team, the design team, and also the sales and even probably a marketing function as well. What are some best practices you could share with our audience in terms of modes of collaboration between your engineering teams and these other functions in the company, especially in this distributed or multi-time zone world?

Justin: Yeah, I think it's been interesting. Docker has changed a lot and these practices have changed a lot as well. We went from being in the enterprise market to developer focused.

I think the engineering team’s collaboration with sales, marketing, and product are very different. With marketing, we've had this amazing community that's built up really earlier with Docker and our products. And marketing has been around growing that community and building events like DockerCon that are amazing. We had 20,000 sign ups just in the first few days when we opened the registrations for DockerCon this year last week. We had 80,000 people last year at DockerCon. These events are what people really value and a lot of our marketing is about nurturing those communities that we've built up around the products.

The engineers love to get involved in that and people really love communicating with engineers and spending time with them, because they're the people who build these tools that they love. And so the interaction between the community and marketing has always fallen out quite naturally.

Sales has changed more in a way. We had a full, large enterprise sales force when we were in the enterprise Kubernetes and runtime business and that was very much me going to meetings and talking to banks about security and things like that, and shipping relatively slow-moving enterprise products that support the enterprise sales process, so engineers often feeling quite detached from sales. It's more difficult to involve them with that. And it's just that engineers don't like enterprise sales, because they don't like being sold to like that themselves most of the time.

Whereas now we have a tiny sales team and we're mainly bottom up SaaS. We're re-exploring what sales is like for a bottom up developer focused company. So it's a totally different relationship we have with sales now.

There was a massive disconnect with Docker, because the grassroot's adoption of Docker was people who are moving to cloud early, but we were trying to sell products to people who were trying to do on-prem and moving into cloud late. It was a kind of mismatch in the end. I think we're in a great situation now where we're really just engineers trying to sell to people like themselves. We're building products for our peers. Developer tooling companies have this amazing thing, where you are building things for yourself and can say that these people are like me. They're not necessarily exactly like me - they might be less advanced in this journey - but there's much more empathy with them, and that makes it easier.

I think from the product point of view, that also helps. I think that, yes, our products are technical. And Docker's original thing was to take something that was deeply technical and obscure - Linux namespaces and containers - and turn them into something that everyone could use. But the market and the users of Docker have changed so much in the five years I've been at Docker. I realized that now they don't understand how to use SSH anymore.

We have over 10 million people using Docker, and it's all sorts of developers at all sorts of levels of experience and technical knowledge. Most of them don't understand the inner workings of how it is. It's just magic for them. And that's very different from people who had opened pull requests to fix an issue with the Linux namespaces five years ago.

So I think it's hard for product people to work with these things, because it is deeply technical. And I spend a lot of time as CTO working with the product managers to help them understand what's possible. And I'll listen to the engineers do that, help them understand what's possible and help them interpret some of the things that they're saying in terms of what users are asking for and thinking about the possibilities.

But there are product people out there who do work and their careers are on developer focused products. User experience for command line is still a niche area, but there are people who are fascinated by it because it's just an interesting topic. It's understanding the modern world. And what developers do now, other people will be doing in the future. So it's a fascinating area to work in product and we're hiring product managers, by the way, if anyone's interested.

Kevin: Never miss the opportunity to put out a recruiting pitch, right? We're all in tech. Everybody is hiring, but yeah, I think you're right. Docker has probably done almost too good of a job back in the days to obscure all these details in the kernel space and on the Linux level of things so that people are like, I don't need to know this. I would just use Docker, right?

Justin: But that's the right attitude I think. I sense that Linux isn't necessarily something you have to use. Sometimes the Linux desktop experience was based on the Sun Workstation experience and it was kind of nice back then. I remember the first time I sat in front of the Sun Workstation. I was like, "wow, this is amazing." But it's not actually necessarily the way people want to consume Linux now. I mean, people actually want to consume Linux as a deployment platform for applications, but Docker is a kind of an API that you can do that without having to use Linux. And the kind of way that people think of, it is like, "it's a programming API, so it doesn't have to be your life." You can still use a Mac or Windows. Developers think of their life in IDEs, in pull requests, and in code. The operating system doesn't have to be a really visible thing.

One Mistake Justin Wants New Entrepreneurs to Avoid

Kevin: Absolutely. So I want to wrap up with a broader question before we go into the Q&A session. Because you've been through a lot of startups before Docker, and now you're at Docker and you're the CTO. I'd love to hear maybe one mistake that you hope all new entrepreneurs, new founders in our space can avoid when they start growing and scaling their engineering team in the early stage.

Justin: I think that one day you'll have someone who doesn't fit and doesn't work in the organization. Donnie Berkholz, our chief of product, has a great talk he did many years ago at RedMonk about assholes ruining your open source projects. It also applies very much to companies that have people who are just negative and destructive of what you're trying to achieve. Recognizing these early and fixing the problem is really important.

If you're lucky it'll happen sooner and you'll get to learn that lesson sooner, but it will eventually happen at some point. There are people who just are not the right fit for your startup at the stage it is now. You just shouldn't be working with them.

You need to set the culture soon. One of the reasons I still love working at Docker is that the culture is just really nice. People are just lovely people to work with, very modest and great at working together. And we've felt that culture for a long time and it's been there since very early on.

One of the things that as a founder you bring to the company is those attitudes of how you want people to be and how you want people to behave to each other. For eight years now we've kept very much the same culture and quite a few of the same people have been there for a lot of that period too, not all of them. The culture has stayed longer than the individuals who helped create it.

To read all previous posts, please check out the Archive section. New content will be delivered to your inbox (twice per week). Follow and interact with me on: Twitter, LinkedIn, Clubhouse (@kevinsxu).