Tim Veil:

So good afternoon. My name is Tim Veil, and welcome again to another episode of Big Ideas in App Architecture. I am thrilled today to be joined by Joe Lynch, who is a long and distinguished career in software development, engineering infrastructure, all sorts of experiences in technologies. I’m really excited to talk to Joe today. I think we’re going to learn a lot about his philosophy not on just building teams, but technology in general. So Joe, welcome to the show. Very glad to have you here.

Joe Lynch:

Thanks, Tim. Appreciate it. Thanks for the kind words.

Tim Veil:

That’s what I’m here to do. When we do these episodes, one of the things that always fascinates me is just spending a little bit of time getting to know you, the audience getting to know you, your background. For me, when I was first starting out of college, I don’t think if you had told me then that 20, 30 some odd years out I’d be writing software for a living, doing all these things. I don’t think I believed you. I think I had in my mind I was going to be very different things. So I’m always curious when we start these episodes. To understand a little bit about how you got to where you are today, what’s been the journey for you thus far? I think almost more interestingly, what inspired you to go down this path, and is it the path that, maybe like me, you didn’t quite expect or was it the path you expected? So maybe just start us off by a little bit of little story, Joe, here.

Joe Lynch:

Sure. Yeah. I come from a long line of distributed systems engineers. No. I grew up in the Philadelphia area, have lived here all my life. I still live in the Philadelphia area. I think the biggest thing, the thing that drove me to building systems in the long term is solving problems. I like to solve problems. I think I figured out that I’m pretty bad at solving problems in the physical world.

So I remember I would try to make little boats and things that would float and they were really, really bad, but somehow I must have figured out that in the virtual world where things aren’t governed by as many laws, maybe I would get better at it, but truthfully, when I think back, I remember when I applied to college, I went to the University of Pennsylvania, I remember I had to check off the Wharton School or something else, and the something else for me, the only other thing I could think of was physics because I liked physics class, oddly enough, because I didn’t particularly like math. I wasn’t particularly good at it, but it was just a tool that you had to use to solve problems in physics, and I happen to enjoy physics for some reason.

So I thought for all of 30 seconds and checked physics and sent the thing, and I didn’t realize how different would my life be if I majored in finance at the Wharton School or something. It could be materially different. So I thought I was going to be a physicist. I graduated with a degree in physics.

Tim Veil:

Really?

Joe Lynch:

Yeah. I started at a PhD program, and I’m ashamed to admit that I left pretty fast, but I wanted the elbow patches, I wanted the pipe, I wanted the PhD on the wall, but I didn’t want to earn the PhD. It’s the earning part that I think is pretty important. So I went into technology in the year 2000 during the dot bomb time. At that time, if you had a degree in something semi-analytical and you didn’t know what to do, you went into consulting. So that’s what I did.

My mom used to ask me, what am I consulting on and what is consulting, and I explained to her, “Well, you borrow somebody’s watch to tell them the time and charge them $250 an hour to do it.” So that’s what consulting was. So in college, I actually did some programming for physics labs and stuff, and I had a job where I wrote some software that tested a lot of circuit boards and stuff that had to do with the proving whether neutrinos have mass, these particles that are important in high energy physics.

So I went to this consulting firm. It turned out that they valued people with my kind of background. Most of the people that I worked with were actually physics PhDs and math PhDs and stuff like that, and they put them to work solving business problems. I got to work with great people and learned how to build systems. The thing that was great, which I realized in hindsight is not all that common, is I got to own things end to end. In the beginning, I knew nothing. By the time I left, I was there 10 years, I had to sell my own work. I had my own practice area, but my SDLC, if you will, started with a cup of coffee with a VP trying to understand their business problems. It wasn’t like, “Where are the requirements?” or, “Somebody send me the requirement stock,” or something. That’s not a world I’ve ever lived in.

So I’m glad for that. I left after 10 years because I had a growing family and I needed to get off the road. Then I went to a company within the logistics space and managed an engineering organization there for a company you wouldn’t have ever heard of, but if you look around your room, chances are 60% or 70% of the stuff you have, this company was important in making sure that it came into the country.

Then I went to a tech startup, series A to series B largely within the ad, I guess you’d call it the ad tech vein. Great people. I got to work with data at a new scale that was new to me, distributed sys. I went past enterprise scale and went to, for lack of a better term, I’ll call it internet scale, and had to learn a lot about distributed systems, which I really enjoyed.

Then after a while, I felt like I wasn’t really growing or I hadn’t had that feeling where I was having trouble keeping up. I wanted to have that feeling again. I felt like I was spending too much time at the whiteboard talking and not enough scribbling on the notepad listening. So I wanted to go to a place where I would have trouble keeping up and if having trouble keeping up was the definition of success, I was pretty successful at Google.

So I was there about five and a half years. First couple of years, I was in storage. I managed the storage efficiency org. So this was the org that was responsible through engineering approaches, whether planning or making changes to systems, figuring out ways to save Google money on its massive storage spend.

The thing that was neat about it was I got to work with all these people that are, in hindsight, they’re the people that wrote the papers on Google File System and Spanner and Big Table and all this. It was a great learning experience. I didn’t have their expertise and I learned enough to be dangerous, but I wasn’t able to match that expertise, but I got to learn a lot. So the thing that was great was working with people across all the business lines because everybody has data to store, ads, search, cloud, photos, whatever you could imagine, YouTube, and also learning a lot about their storage systems and databases and working with some great people. I remember the tech lead on the team, they said something like, “I don’t get out of bed for less than an exabyte. Hundreds of petabytes was a roundinger in terms of the places that we focused. So it was a lot of fun. It’s a lot of fun.

Tim Veil:

Yeah, it’s a big scale.

Joe Lynch:

Then I worked in capacity planning, building the system that produces the capacity plans that underlie Google Cloud’s products, and then the last few years, I worked on the observability platform. I was responsible for the data ingestion for logs, metrics, all the things that you need to manage your systems at scale.

Then I spent a year at Twilio, where I was responsible for infrastructure, so compute databases, observability, cost efficiency, networking, et cetera, and that brings me to today. So yeah, I fell ass backwards into technology is the bottom line, but I feel fortunate to have met a lot of great people and learned a few things along the way.

Tim Veil:

No, I think I love your background. It’s eerily similar in many ways to my own, particularly your last comment there. I don’t think I ever thought I would be doing what I’m doing today. I was very curious. I like to take things apart and put them back together. I loved physics also in college or not in college, in high school. Ended up really, really enjoying math. In college, did a lot of advanced math stuff, but thought I was going to be … I don’t know exactly what I thought it was going to be, but I had a finance degree and started interviewing with a bunch of banks at entry level banking positions and stuff like that.

I thought, “What the heck is this? This is not for me.” So I went back and dual majored, and that really is what I think chartered the new course. I dual majored in MIS, and then went into consulting like you did, and the rest is history. It started ultimately writing upon-

Joe Lynch:

What firm did you work in?

Tim Veil:

Ernst & Young, my first job. A funny story about that, by the way, it was in the early or late ’90s, right around the time you were doing consulting. I remember distinctly they said, “Hey, welcome. You’re going to start on this date. We recommend you go out and buy three suits because we’re …” What? A business formal, I don’t even know what you called it back then. So I did. I dutifully went out and bought a gray suit, a navy suit, and a black suit. I think I wore navy the first day. I literally think day two or certainly within the first week, they sent a letter saying, “We’ve moved to business casual.” So those suits sat in my closet for years as a reminder of my first job at Ernst & Young never really to be worn again, but-

Joe Lynch:

The CEO at the consulting firm where I worked, which is a much smaller firm, he was old school. We wore ties for the whole time I was there till 2010. I left 2010. We had ties every day.

Tim Veil:

I do think about that now. I mean, obviously, I’m at home, you’re at home, we’re working from home. It’s sometimes a leap or a stretch just to think about going back into the office, but can you imagine going into an office five days a week now wearing a suit and tie? It just seems-

Joe Lynch:

Not anymore.

Tim Veil:

It seems so foreign. Can you imagine starting a team right now, building a team of engineers saying, “Hey, listen, we’d like you to come into the office every day and we’d like you to wear suit and tie”? I don’t think we’d get as many people as we did back then.

Joe Lynch:

Yeah, I agree.

Tim Veil:

So when you and I talked prior to the show, one of the things I found so interesting, not only about the background and the story you shared, but it’s just some of your philosophical approaches to I think building software, designing software. I thought maybe we’d spend a little time just talking about some of those key ideas. I think a little bit like you, I came to a lot of the lessons I learned just by working out problems that I had or the teams that I worked with had. I know you’ve got a lot of really strong ideas about things. I thought maybe we could just jump into a couple of those ideas and see where the conversation leads, but I thought your philosophy, a lot of your ideas just around problem solving, all the stuff that goes into building software was really fascinating to me. So maybe we just take one of those and see where the conversation leads.

Joe Lynch:

Sure. I think I have a habit of while doing something like designing software, participating in the process of designing software, I take an approach where I’m trying to figure out the patterns within there that are unrelated to the software that you’re designing like the process behind the process in a sense. So I suppose a lot of the observations that I make, they’re meta in that sense, for lack of a better term, and all the things that are useful are not original, and all the ones that are original are not particularly useful. So I’ll try-

Tim Veil:

All the better.

Joe Lynch:

Yeah. So I’ll try to talk about the ones that by definition I’ve learned from other people and places. They say that a smart person learns from their own mistakes, a wise person learns from those of others. I’m certainly not wise, and it’s a stretch to say that I’m smart because I have to trip over the same stone three, four or five times before I learn. The definition of insanity is to try the same thing over and over and expect different results.

So a lot of times I found myself in that situation, but I do tend to get interested in these, for lack of a better term, I’ll call them principles. So you have the solid principles and you have Conway’s law and Demeter’s law and design patterns and all these things. I tend to find them interesting. I have come up with one that I think that’s important that I keep in mind because of the fact that I enjoy this stuff, and I call it the GUT, the grand unified theory of software engineering. So Einstein pursued the grand unified theory for the rest of his life after he figured out general relativity and he died unable to do it, but I figured out the grand unified theory of software engineering. Do you want to know what it is?

Tim Veil:

I do. Some say you are the Einstein of software engineer, some say.

Joe Lynch:

Absolutely, absolutely. Yeah, it’s been said. So I think that the single law in software engineering is that aside from math and physics, there are no laws in software engineering. That’s it. If you think you’ve found a law, read this one again. So when I think back about all the things that I’ve learned, Conway’s law is not a law. A law is it something that’s generally speaking inviolable. It’s not a law, it’s a tendency.

Tim Veil:

I should know this and I’m embarrassed to admit that I don’t, but what is Conway’s Law?

Joe Lynch:

It just says that the software that you build will tend to reflect the communication structure of your organization. So let’s say you’ve got nine subteams working on a compiler, you’ll build a nine pass compiler. So that’s a toy example, but one of the ways that you can see it in reality is you can see companies that build certain software and all of a sudden their website starts to reflect their department structure or their department structure starts to align with the way they think about their modules.

Another one that I think is more obvious is that if you thought … So I worked at Google and they had this small independent teams concept, which is not novel, other companies like Amazon do it, but Google does it in a very specific way. Within that team, nobody can tell you what to do. The tech lead on that team is the chief. If Sundar wants to get something done, he’s got to ask a few favors to have … There’s no such thing as command and control when it comes to the sanctity of that team, and teams, they run independently. They will tend not to coordinate with other teams if they can avoid it, and they have very well-defined services, and services generally don’t go out, never span a team.

Then they only communicate with other teams in terms of meaningful messages via quarterly OKRs. That’s how you can get a commitment from one team to another. This sounds like microservices.

Tim Veil:

It does.

Joe Lynch:

Yeah, and what it’s basically saying is because of the fact that this organizational structure was in place, you’ve created a very strong incentive for the software to look a certain way. So sometimes what people will do is say, “All right. Well, let me decide how I want my software to look and I will do reverse Conway’s law, which is I’ll make the org structure and communication structure that will incentivize that software architecture.” So I was just picking that out as an example.

Tim Veil:

That’s actually super fascinating. I mean, I’m thinking I had not heard that, but, boy, it resonates. I mean, I had not heard that law described or it wasn’t familiar, but that is something that I have seen play out many, many times.

Joe Lynch:

Yeah, it’s funny because you think you’re, “No, that won’t happen to us,” but before you know it, it happens, but it’s not a literal law. It’s not like F equals MA or something. So the joke about my grand unified theory is just that I don’t think there is a grand unified theory except for literally math and speed of light. There are a couple of things that are just givens, and one more, which is it’s harder to get … Trying to get a date out of a software engineer is like trying to get blood out of a stone. That’s the only other one that I’ve noticed as well, but everything else is up for grabs.

Tim Veil:

Yeah. I do think you’re right. I mean, just in the many years I’ve been writing software, you reflect back on the things that were … Let me use the term taken as gospel, but taken as law, whether it was design patterns or just ways of doing things that if you weren’t doing them, it’s like you weren’t part of the club, you weren’t a good soft-

Joe Lynch:

Yeah, you were a sinner.

Tim Veil:

Yeah, you’re a sinner, you’re a sinner, get out. You’re right. Those have changed a lot over the years. I think actually it’s a pretty astute observation because I think when you look back over time and you realize that almost every couple years there’s some new thing that everybody’s doing that will ultimately fall out of favor to be replaced by something else. I mean, I think it is actually when you come across those things that people are raging or raving about now, you take a step back, you think, “This is probably something that a couple years from now people will have moved on from.”

Joe Lynch:

Yeah. You’re getting into a nuance, which is it’s related, which is going little further, which is like the no silver bullet thing. I think it was Brooks, the guy that wrote The Mythical Man-Month, who wrote the no silver bullet essay or whatever, which basically takes the position that you may think you have found it, but there’s no one thing that’s going to affect the efficiency or effectiveness of building software by even an order of magnitude. You may think you have found it, but you haven’t, object-oriented programming, actors, microservices, enterprise service, buses, probably ChatGPT, though I’m not informed enough to. That one’s a little wilder, I’ll say, but so far that’s turned out to be true.

Now, when I say that there are no laws, my only point there is there aren’t absolutes. I do actually think there are very, very useful principles that you can hold onto as heuristics and apply intelligently, but I just try to … because I get excited about them, I try to be mindful of the fact that they’re just that, they’re heuristics.

So you take something and sometimes what I find is a heuristic can be both a useful decision making shortcut, and it can also be a bias. It depends on context. So people used to say, “Nobody ever got fired for choosing IBM.” It’s like, “Hey, when in doubt, choose the leading vendor for something,” and it was almost also halfway a joke about the incompetence of, say, some technical leader somewhere who didn’t know what they were doing, but I found myself using it recently where we were building a Kubernetes platform at a company where I worked, and the team wanted to revisit every single decision. We had to figure out what kind of container registry do we want.

I found myself saying, “Nobody got fired for choosing IBM. Use Amazon’s manage container registry. For every single thing, use commodity options so that we can boil down to those problems that are distinctive for us.” Six months from now, you are not going to be thinking about, “Oh, man, I chose the wrong container registry.” You’re going to be thinking about, “Oh, I should have made the developer experience easier to adopt this platform.” So that’s an example of where I’m using it there almost as a decision making aid when most of the time you’ll tend to use it as almost mocking the simplicity of some people’s decisions, and I find a lot of principles work that way.

Tim Veil:

Yeah. I mean, I’d love to hear more of them because, again, it resonates so strongly and I think one thing that I’ve observed a lot lately, and there’s some similarities and some differences, but I have found myself saying this in coaching people on is like, “Don’t let perfect be the enemy of good.”

Joe Lynch:

Absolutely. Absolutely.

Tim Veil:

That description of searching for the best artifact repositories is a little bit like that. You spend so much time pouring over all of these details, which in the grand scheme of things turned out to be inconsequential. You take your eye off of the big picture, but why? Because there’s so much pressure on IT teams, there’s so much pressure on us as leaders, software developers, companies to get everything right. I mean, the stakes seem so high. I think there’s this pressure to be right about everything. Did you do your due diligence? Did you analyze every option for this thing? No matter how small it may seem to be at the time, I just think people feel like they owe it to someone or something to do these things, but what ends up happening is we just waste so much time pouring over this minutiae that we lose oftentimes the bigger picture.

Joe Lynch:

Yeah. I know exactly what you mean and I think we all do it because I don’t think we’re set up well for learning how to solve problems in a land where so many things are probabilistic. We learn in this environment with deterministic problems. Think about grade school, high school, college. They give you a problem definition. That problem definition is perfectly specified. There’s no ambiguity with the exception of, say, maybe taking … When I say problem, I’ll talk about, say, math, physics, chemistry, things like that. I’m not talking about an essay or something where you can take different positions, but the problem is fixed and then you just focus on the solution. There’s only one way to do it usually or I have certain well-defined outcomes that I have to achieve in that solution. So you’ve got to have the best solution and that’s what we’re taught.

What I find is that very, very few business problems or technical problems are framed that way. Most of the time, your first problem is defining the problem. My experience is that we often go from … If you think about it logically, the steps that we might take when we’re designing software, we start with there’s the background, the real stuff that you can describe, the phenomena, if you will, the stimulus for getting you to think about it. Then you form some problem definition. Then you say, “Okay. I’ve got that problem. Now let me create a solution space. Maybe I’ve got five different ways to solve it. Then I go and I zero in on the one that I think is best and then I design the hell out of that one.” Only at the end of that process do most people start to get peer review presuming for a second that the problem is large enough to merit peer review. I mean, not every problem is, but what happens, I find, is if you watch people that are just really blunt on design docs, those people … There’s some people that are always like, “Ooh, looks great,” and then there’s other people that just tear design docs apart.

If you watch them, there’s often two … When people are giving me meaningful comments, they often come in approximately two flavors. One is they’re like, “Yeah, that’s right, but don’t forget to update the X, Y, Z while you’re in there. Oh, yeah, remember we had to use the new RBAC framework over here.” Those are aiding the general discussion. Then you’ll have people that are like, “But why are you doing it this way?” They don’t say exactly that, but that’s what they’re … They’re fighting the fundamental of what the design doc is trying to do.

If you look back, if you unwind it, oftentimes it’s a problem. They’re indirectly pointing out a problem with the framing or a problem with the analysis where they’ve chosen which path to do. Given the fact that four are framing and that they chose path three, they’re like, “Yeah, this is a reasonable design for path three, but I think you should have chosen path two or I think you have the problem framed wrong,” but we often don’t talk about these things.

We often don’t even use the term … We talk about solution space a lot. How often do we hear the term problem space? I usually try to … When it’s a tough one anyway, when it’s nontrivial, I usually encourage people to think about the problem space, but it’s a space because there’s a lot of different ways to define a problem. There was a really good book that I read, and because I’m going to diss it in a small way, I won’t name the book because otherwise, it was actually a really, really good book. One of the things it was good at was it talked a lot about how you can apply basic design principles to project structure. It was very well thought out, but they were using this one example that was a literal example that they did and they were defending it as this was great example.

In the beginning they’re like, “The requirements, and we gathered the requirements and the requirements were … We need to build the most generic infrastructure that would allow us to be infinitely flexible.” Does that sound like what a business person would say? No. That’s an engineer’s confirmation bias at work, which is, “Ooh, we always need a generic framework that’s going to do X, Y, Z.”

So I don’t know, I think you often have to inventory the problem space and then go with the best problem definition and then inventory the solution space, and sometimes you need to do, for lack of a better term, what I would call a comparative analysis of options and get people to buy in to, “Oh, we’re going to choose option three and here’s why,” then design option three. I don’t mean to suggest that it has to be this mechanical thing, but that sometimes the disconnect is way back in the problem definition or in the analysis.

Tim Veil:

It’s really fascinating because, again, I think I’ve tended to see this a lot. We tend to see this I think sometimes with the customers that we interact with or the prospects that we interact with because, and maybe I’m thinking about it a little bit differently or taking it in a slightly different direction, but I think if there hasn’t been true collaboration on defining the problem, the problem space, and you allow a group of people to move very quickly down into solutioning, and then you get there and that’s when you bring in a broader team to say, “Hey, what do you think?” Boy, it’s really hard to pull people back from the investment of time they made in coming up with the solution. They become very emotionally attached to their ideas, things that they’ve done.

So it becomes … Especially if the concern isn’t around the solution but way back before the solution was created in the problem space, just winding people back. I mean, I’ve been in countless meetings where people just … It’s human nature, unfortunately. It’s not that they’re bad people to be defensive, “Well, what do you mean this isn’t the ideal solution?” It becomes very difficult. So I think I really like this idea of focusing in on the problem, “Let’s get that right. Let’s make sure we get that right before we move into the solution,” because once we’ve committed and gone down this solutioning path, it’s really hard for people to accept a rewind on that, and human nature is just, it’s tough.

Joe Lynch:

Yeah. Think about going even further. Sometimes you stumble on what are conception problems at code review time. It’s even worse, right?

Tim Veil:

Oh, definitely, definitely.

Joe Lynch:

13 steps before there was a misconception. I think it was Rich Hickey is probably one of my favorite thinkers. I don’t know if you’ve ever listened to any of his stuff, but if not, definitely watch Simple Made Easy. It’s probably the best, for me anyway, it was the most interesting … It’s like Meta, I guess, similar to this discussion. It’s the most interesting tech talk that I’ve ever seen, Simple Made Easy. Anyway, yeah, he talks about a lot of this kind of stuff, but the thing that’s very real is people dispute the actual math due to the research anymore.

Barry Boehm, who was one of the … I don’t know, if not the most prominent, one of the most prominent software researchers to ever live, coined this idea of the cone of uncertainty where you start out with so much uncertainty as far as what you’re trying to do and you explore, you explore, you design, you design, you develop, you develop, you get it out there, you move it. Before you know it, only at that time does it really start to come together. He makes the point that, which is obvious, the cheapest time to fix something is way back here, but he actually did research to show the math of how much more expensive it is to fix something when you’ve got a misconception in production versus when it’s just at the whiteboard. I mean, the numbers are enormous.

Tim Veil:

Well, and two things come to mind on that. One is this idea of OKRs, and the connection here is maybe a little bit tenuous, but what I have found is that OKRs can be great, but they do define these touchpoints between organizations. Sometimes, as you were describing, they may be the only thing that’s really holding these two organizations together. I’m going to have this shared OKR that makes this one thing.

Joe Lynch:

Yeah, you put them on a message bus and you get an act from the team.

Tim Veil:

What I’ve found sometimes is that this problem that we’ve just been discussing though finds its way into that. There’s been all of this thinking about a problem, all of this designing a solution, getting to the point where the solution is almost ready and it’s at that point and at that point only that we say, “Let me share it cross functionally. Let me share it with this other group because this OKR said that I should or would or did.” To your point, at that point, it’s so late in the game. The idea isn’t to necessarily share the finished product and get input.

When you’re building and trying to solve really complex problems and building complex software, just as you said, you don’t want to put all this time and effort to design and become so deeply emotionally vested, share it so late that it’s like, “Well, wait a minute, this isn’t anywhere near what we wanted and now we’ve got to go right back to the beginning.” I think sometimes these silos in organizations that create this environment where, “I’m only going to show you something when I think it’s done and ready,” I don’t know about that.

Joe Lynch:

I think there are legitimate reasons why that happens, like you said, due to human nature. If there’s one other thing that I think about often, it’s humans build software. Software doesn’t-

Tim Veil:

At least for now, at least for now.

Joe Lynch:

Wait till ChatGPT-6.

Tim Veil:

Yeah, that’s right.

Joe Lynch:

So what that means is all the vulnerabilities that we have as humans, think about how you communicate with your sibling or your wife or your cousin or whatever, somebody where you have … Every relationship you have is imperfect. Even somebody that you know really well, there’s disconnects all the time. We are, in that sense, a capital S System in the systems theory sense. Well, imagine the complexity of the system of getting just a single team of seven to work together and then an organization of Cockroach, I don’t know, 150 engineers or something, doing something as complicated as what you’re doing. It’s a challenge.

This is not intended to be a grand unified theory, but I have found an approach that I think tends to work well or maybe it’s just because it’s flexible and is resilient, but for getting through that validation process, and I call it concentric circle validation, not that this is … It’s not novel, but there’s … Part of it might also be catered to the way that I am. It might not be the right … So for example, if you talk to people, they’re like, “Whoa, we got to get the right people together. Let’s set up a conference call with eight people.” Oh, my gosh. I don’t want to start exploration of solving a problem with eight people. I don’t assume that that’s actually a good way to go about it.

So usually, I would prefer that there’s one person. First off, you have to define a lead. That lead has full decision making authority within that space unless there happens … There might be tech leads that can technically overrule them, but more or less they’re just there to support them. If somebody’s going to own a design within that space, they have the ability to make technical decisions where it’s very clear who above them does, but there’s crystal clear decision making authority. Usually, personally if I’m doing it, I need to scribble on a notepad, think about it, sleep on it, not take a long time. Well, in fact, Rich Hickey has another talk called Hammock-Driven Development, where he talks about this. He promotes laying around in your hammock thinking about this.

Anyway, personally, I find I have to noodle on it. Then what I do is I have my one or two trusted design partners, and I’ve only figured out through experience there are a couple of people where we have close enough values where we groove in communication wise and we can feed off one another and communicate in something of a shorthand. So it’s not so much that because I have these two people that these other two people aren’t great. It’s just somehow there’s a strong connection there. It’s a little hard to describe, but I think you know what I mean.

Tim Veil:

Oh, yeah, absolutely.

Joe Lynch:

So they’re your go-to people. You explore with them, “Okay. Yeah, it makes sense at the whiteboard.” Meanwhile, personally, I haven’t written anything down yet necessarily. Some people will say that that’s bad, but the first thing I wanted, I want to fail fast and writing something down for me is not the fastest way to fail. I want to disprove the approach in a sense, and so I’ll show it to people, have them beat up on it, and then start to … Basically, that’s my level one review, and then I go out to level two and that’s what I mean by concentric circle, and they have to be small chunks of people, not big chunks of people.

So it’s like, “Okay. I’ve got my level one peer group. Now I want to bring in my level two.” They’re the ones that are the really tougher tech leads who understand the related modules and they’re going to tear it to shit, but I know that if it gets through them, it’s solid. Then after that it’s like, “Okay. Now it’s for broader consumption. I’ll take your comments, but frankly, unless you find a fundamental flaw, it’s getting pretty baked,” and going down a process like that, it’s almost like getting a bill through Congress, I suspect, just tiny little bits, getting one or two people on board and then gradually working outward.

Tim Veil:

No, I think it makes a ton of sense. It strikes me as a very efficient way because you put these ideas in front of large groups, everybody at different levels of influence, different levels of knowledge. It can become, and I’ve seen this play out again so many times where these meetings where you’re just hoping to get a little bit of feedback turned just into the Wild Wild West because it’s not well-controlled, not well-organized, ultimately.

Joe Lynch:

Yeah, and ultimately, you’re trying to deliver business value, not software when you’re focused on building software, and you should be optimizing for time to value. There are a lot of ways that you can do that, but obviously, I could type faster, but that’s not putting aside again, ChatGPT or typing faster. Those aren’t the best. Those aren’t the best ways to minimize meantime to value. It’s to reduce your cues, your dead time, and those cues, some of the biggest ones are, “Oh, I wrote this document, but I got to get on the calendar with the product manager and they’re out for four days,” and then when they get back, they’ve got … I’m not picking on product, I’m just making something up. Then they’ve got to dig out of their email. So they’re in this another couple of days. Well, now I’ve created waste because I’ve paged this out of my head. I don’t remember what I was thinking anymore. Now I got to page it back in that.

Then the product manager gives me feedback and I realize I was off base in some way. I go back, do stuff, then I got to get on their calendar again, right? Then I’m overly expansive in who I ask to review my design because I don’t want to hurt anybody’s feelings. The issue is efficiency and inclusiveness are at odds sometimes. I don’t want everybody’s vote on everything, including my own. If I want to know what kind of pizza we’re ordering, that’s a fully democratic decision, but if we’re going to decide how we’re going to build module X, Y, Z, there’s a small set of subject matter experts that I want to weigh in. Ideally, that’s it. I don’t want everybody to comment on it because it’s inefficient and it will cause that queuing and you’ll never deliver the value.

Tim Veil:

So Joe, I know we’re coming up on the time we had allotted. Honestly, I think you and I could probably talk about a whole host of other things for much, much longer. So maybe what we’ll do, maybe we’ll just call this part one of the Joe Lynch Grand Unified Theory Series. So stay tuned for part two.

Joe Lynch:

Please don’t put that title out as clickbait. It’s obviously sarcastic.

Tim Veil:

Part one of the Joe Lynch Grand Unified Theory coming to you live. No. So maybe we close up on this because, I mean, the truth of it is I think this has been a fascinating set of insights and would love to keep talking with you about them, but as we’ve ended maybe a handful of other podcasts on just a little bit of looking forward, I’m curious, given how well read you are, how thoughtful you are in so many of these things, are there things in the industry of technology, anything right now that’s catching your eye as like, “Hey, this is something that I’m personally interested and want to spend a lot of time researching, looking at, paying attention”? I’m just curious what’s caught your attention over the last six months. Where are you going to be focusing some of your time and energy at that?

Joe Lynch:

There are a few things. One of them is frankly directly related to CockroachDB. That’s why I communicate with you guys. I think you’re the best at what you do. I think you’re out in front leading and you’re trying to simplify the overall experience of developing high availability services. This is a really, really big deal. I think a lot of people, they aren’t caught up to it yet, in my opinion, because the new SQL database is not yet a household primitive. It’s not yet commoditized. I think commoditized is not the right … It’s not yet normalized. It’s like people heard about the Spanner thing and it’s like that’s whizbang and rocket ships and you have to be on Google to use that. Then there are other ones that are coming along and, “Hey, I hear about CockroachDB and Yugabyte and things like that,” but I think to be a household tool that any architect at any company will bring out is AWS has to come out with a competing product.

As soon as they do that, I mean, it’s only a matter of time. There’s even references to it online to them working on it. As soon as that happens and people start to get a sense that they can get all the guarantees of a traditional relational database while having high availability and horizontal scalability and all the things you couldn’t have before, I think it’s going to simplify people’s lives.

The thing is that we need this in order to be effective in developing software. Things are getting more complicated, not less, and all the things that you have to understand in order to build an effective system these days, it’s mind-blowing. Something like CockroachDB in the way that you can be declarative about distributing the data and things like that, it’s huge deal as far as I’m concerned. I think it’ll change the game. I think it’ll take away a lot of the workloads from no SQL databases, not all of them, but a lot of them.

Then I think that a lot of CIOs who are used to having conversations like, “How much downtime can you afford?” they won’t have to have that conversation anymore. The play for these databases has often been scale and people will say, “I don’t need that. I am not a Google scale.” Sure, but do you want to go to your head of operations and beg for downtime? No CIO in their right mind would want to do that. Well, what if high availability just came for free? I think that could really change things. So just give me the 50 bucks after the show for the plug.

Tim Veil:

We took your home address down before we started.

Joe Lynch:

Yeah. That’s one thing I’m thinking about, and then the other thing I’m thinking about is I’m a little concerned with the Kubernetes obsession in the community, and I’m concerned that it’s being perceived as that silver bullet. What’s happening is a lot of people are making the awful approximation that somehow Kubernetes is the only way that one will deploy software, and that it’s the only way that people will ever deploy software, and that’s obviously not true. There are still and probably will always be workloads that belong on VMs. There’s serverless, edge functions, all kinds of stuff, and the industry … We’ve got Wasm going. We’ve got eBPF going. We’re always inventing new ways to build software.

What I find is that everything is so Kubernetes-obsessed that it’s creating this fork in all of our technologies. Anytime I go and read about a new project, “Oh, have you seen the guacamole framework? Oh, let me learn about guacamole.” You click on it and, “It’s the greatest thing since sliced bread for scaling your wiggle wands on Kubernetes.” Everything is for Kubernetes.

Well, it’s like, “Hey, what about the rest of the world?” The truth is that Kubernetes is complex. I’m not forecasting its downfall or anything like that. It’s a great primitive, but it’s not the only thing, and I’m concerned that it’s making our life pretty complicated and people are making some bad assumptions. The story of multi-cluster Kubernetes and having VMs talk to Kubernetes, talk to Lambda’s like that network connectivity story is so complicated, service meshes. I just think there’s a lot of incidental complexity there. I don’t claim to know what the answer is, but my spider sense says that we’re going to be writing about this five, 10 years from now just like we did about, “Oh, remember the days we thought enterprise service buses were great things?”

Tim Veil:

Oh, I mean, having been on the front lines of this for a while, I mean, it is enormously complex and it is a hindrance. It’s a hindrance and it’s another black box to some extent, where problems exist and people have a really difficult time troubleshooting them. It’s something I’ve talked about in a few other podcasts. I agree with you that it’s one of those things that it’s all the rage, but I’m not sold on whether it’s the future for everything as you’ve just described.

Joe Lynch:

Well, and the way that I look at it is I’m even putting that aside. Pretend I were sold. Pretend you were sold for a second. It’s simply impossible that it will be the only way that one deploys software. You would say that people are like, “Oh, you can run MySQL on Kubernetes.” Why would you? One of its most foundational principles is that it wants services to be disposable. That’s a great principle for what it was originally designed for.

At Twilio, for example, we had stateful phone call connections that had to be up for to 24 hours. Can you imagine that amount of state? It’s facilitated via the SIP protocol on UDP and all kinds of jazz, but can you imagine trying to do that in Kubernetes? It just wouldn’t make any sense. So even if you love Kubernetes, the idea that somehow it’s the only way is just, it’s silly.

Tim Veil:

Yeah, no, I agree with you. Well, listen, Joe, love the time together.

Joe Lynch:

Likewise. Appreciate it.

Tim Veil:

Thank you so much for joining us on the call, and I do think we’re going to do a phase two.

Joe Lynch:

All right. Sounds good.

Tim Veil:

A part two. Lot of fun.

Joe Lynch:

All right.

Tim Veil:

Thank you.

Joe Lynch:

Yup.

Tim Veil:

Thanks for listening to Big Ideas in App Architecture. If you like what you hear and want to watch the episode, head over to our YouTube page linked in the description below. Also, be sure to rate the podcast five stars and we’ll talk to you next time. Thanks. Bye.