Tim Veil:

Welcome to the Big Ideas and App Architecture podcast. I’m your host, Tim Veil, and today I am very excited to be joined by my longtime friend, Paul Lawler, who is head of software at Wahoo Fitness, an Atlanta-based company where I’m currently located as you are too, Paul. Welcome to the show. Before we get kind of into architecture, big ideas, all that stuff, tell us a little bit about Wahoo Fitness, who you are, what you’re doing there. Kind of let everybody know what’s happening at Wahoo.

Paul Lawler:

Well, thanks for having me on today, Tim. So let’s start there. I am head of software at Wahoo Fitness. I have an early tenure there, started in April of ‘22, so starting to close in on one year at Wahoo. Wahoo is a technology company. We service the fitness industry. It’s about a little over 10 years old. Chip Hawkins is our founder, he’s an Atlanta guy as well. Wahoo had a number of ANT+ devices, so ANT+ is a protocol for communicating with independent fitness devices, and that was many years ago. So like a heart rate monitor dongles that would attach to your computer, so you could read information back and forth. But where Wahoo really started to grow and kind of its big splash was what we call the KICKR. So it’s a K-I-C-K-R. The KICKR is an indoor bicycle trainer. So it’s what we call it direct drive trainer where you take your back wheel off your bike and you can put it onto a drive train on the device itself. And what that provides is it’s basically a data platform for broadcasting your power data, cadence, speed, et cetera, et cetera. And the deal with that is that it works with a whole bunch of different software that’s out there, a bunch of third parties that then can create training experiences indoors to help you become a bigger, better, faster athlete. Tim Veil:

Just looking over the website, I mean, y’all have a lot of the kind of products, a lot of offerings obviously, you mentioned the word data is obviously an integral part of what y’all do as a software. Tell us a little bit about how important software data and ultimately where we’re going to go with this, I suppose, is the application architecture itself, but aside from the hardware and all that stuff, how important is software and data to what Wahoo is doing today?

Paul Lawler:

Yeah, it’s foundational. Wahoo’s kind of like I would say when it comes to software, it starts at the machine level, at that device level. So one of my teams is our firmware team and that’s where we get actual information, what you and I would call data that we can actually pull off of what the machine is actually doing. So things like torque, we have a hysteresis brake, we have also, we have- Tim Veil:

A what?

Paul Lawler:

… physics engine. It’s a brake that actually regulates the amount of resistance that you feel. Tim Veil:

Interesting.

Paul Lawler:

… on the bike itself, on the trainer itself, we have a physics engine that runs at the firmware level so that it creates as realistic a ride feel as you can create in indoors. I would argue that we probably have the most realistic ride feel out of anybody in the industry, and that’s what attracts people to Wahoo devices. So it’s a premium experience for athletes and we consider everybody an athlete by the way. So starting at that level and working its way up, what I would call our stack, we are capturing all of that information that’s happening at that physical level and transforming it into a signal or series of signals that can then be represented to the end user. For example, what is the current wattage that you’re doing? What is the current wattage that the workout that you’re doing? If you’re using our SYSTM, S-Y-S-T-M, we don’t like vowels at Wahoo. If you’re using our SYSTM software, training software, how are the watts that you’re producing, what are they relative to the training prescription for that particular activity? And that allows you to say, “Okay, we can set it so that I want you to lock in at 200 watts, Tim, for the next 30 minutes.” And that type of training is designed to elicit response from you physiologically. So that’s going to make you a better endurance athlete. And you do enough of these sessions, which are varied and over time, either with a coach or self-coached, we’re going to turn you into a better athlete and God knows you need all the help you can get. Tim Veil:

You talking to me personally or just in general? Because personally, I do actually need quite a bit of help. Like I said, I think I’ve been on that bike upstairs twice. And the problem is I start … I mean, I have friends who do 30, 45 minutes, I have five minutes in, I’m like, “All right, you know what I’m done with this.” So I don’t suspect I’m generating or we’re generating-

Paul Lawler:

There’s plenty of data in that five minutes. Tim Veil:

… a ton of data or signals.

Paul Lawler:

We’re capturing probably information from the trainer, I would say about every eighth of the second. So there is a true stream of data coming from that device. Now that’s not what you see on the screen, but it allows to zero in on that data and create as quality of data training experience for our athletes. Tim Veil:

Yeah. Talk to me about [inaudible 00:06:11] what is that journey, that data journey? Because again, I’ve never had this conversation with somebody who operates these devices or has built software for these devices. So all this stuff is happening, these signals are going where, I mean, is this all headed up to a centralized repository in the cloud? I mean, walk us through, if you can, just what is overall at a high level, what’s kind of the architecture supporting these devices?

Paul Lawler:

We talk about it at the lower level and then we’ll work our way up to how it gets stored and then reflected back to the athlete. So we talked about the firmware that captures kind of raw information from the device itself and how that gets modulated by our physics engine to provide a realistic feel of being on a bike. We have a software layer, what we call our Wahoo sensor management layer, which then translates that, those signals. So when not only do we communicate with our own trainers, but we communicate with third party trainers. So if you’re using, Garmin has what’s called a tax neo trainer set. SARS has a trainer stages, has a trainer called the SB20. So all of, not only ours, but all these third parties we connect to. In addition, you’ve probably, I don’t know if you’ve ever used your Apple Watch, but a heart rate monitor, a strap based monitor, a risk-based monitor will connect with those devices as well. And so what the Wahoo sensor management protocol does or that layer of the stack is it consolidates all these signals into a way in which our training software can then aggregate that information back to the user. So for example, if you are picking an activity, “Hey, I want to ride on the trainer today, I’m going to go into system, I’m going to pick a workout that’s like let’s say 60 minutes at an easy pace, an easy endurance pace.” We can set the trainer through this stack of signal processing to make the trainer just lock right in at whatever the appropriate wattages for you. So all you have to do is pedal, right? And you’re going to get the benefit of that workout without having to think about it. However, you might also want to do an interval workout. So probably Peloton in your experience, you probably do some high intensity stuff and you go down low intensity, then high intensity and then low intensity and in the five minutes that you last on the pivot. Tim Veil:

Of course. Yeah, A lot of high intensity, very high, extremely high.

Paul Lawler:

So those are what we call intervals. And when using our software or any third party software, we allow for our trainer to be controlled not only for the signal to come up to the user through that software, through that interface, but also the training software can then tell the trainer what to do. So it can control your output and basically elicit a physiological response from you as an athlete to get the benefits of your training program. So that Wahoo sensor management program, that protocol consolidates that information, it then communicates to those signals to our own internal training software if we’re using system or a third party like Training Peaks, trainer Road, Zwift is a very popular platform and we have an analog platform called RGT, which is a virtual experience. So you have an avatar and you’re pedaling in a virtual world. Tim Veil:

Interesting. So tell us a little bit about the … So architecturally, I think I kind of understand, or at least generally the layering and what’s the tech stack look like to do all this? I mean, obviously there’s a lot of moving pieces. I mean, because you’re kind of responsible for the whole thing. I mean, it’s all the way from down to the firmware. I mean, Wahoo in general, whether you personally are or not.

Paul Lawler:

I’m just a one man show. Tim Veil:

You’re doing it all. Yeah, of course. Always have been, which we’ll get into. But what does that tech stack look like to control all this?

Paul Lawler:

So when you’re at the device level, it’s generally C. So a lot of CC++, we have libraries, internal libraries, one of which is called Crux, which is a general purpose library that we use to communicate with all of our devices and in addition, third parties. So this is depending on the type of protocol that the connection is made through. So for example, we can connect through Bluetooth, we can connect through ANT+. Some devices will give more information depending on the protocol than others. So what we try to do is get as much information from the device that we’re connected to as possible and then translate that into more of a consistent set of signals that can then be interpreted by the training software. So CC++ at that kind of lowest level for the most part. Once you get up a level, then you’re on your native device. So you’re talking to the trainer through your phone. So you’re managing your connection, for example, that pairing of your trainer through our native app, we have a couple of native apps, our Wahoo fitness app as well as our Element app. And that is Android and Java on one side. And you’ve got iOS and Zwift on the other side. And we have a bunch of objective C as well, but that’s a lot of our legacy code. We’ve moved as much as we can over to Zwift working our way up. So that’s kind of like you’re pairing on your native device and then you’re going to run that training software on your phone. You can run on a tablet, you can run on a desktop, and that’s where we have a combination of react based applications and then plugins to communicate with those signals through a hybrid application essentially. Does that make sense? Is that a terrible? Tim Veil:

It does, it does. I mean, working for a database company, I’m always kind of curious where is all data stored? So what are you guys do doing for the backend?

Paul Lawler:

Let’s bring this back around to- Tim Veil:

No, no, no, no. Look, I want to hear more about the C layer, but where does all this end up?

Paul Lawler:

Ultimately, it ends up in our Wahoo cloud. So we have a cloud backend and the core, if you had to boil it all down, Tim, the FIT specification, which came out of Garmin years and years ago, that’s the flexible interoperable data transfer specification. Everything gets stored in a FIT file. And that’s a binary specification. And it was developed to be able to have a general purpose way of storing GPS related data. Now you’re like, “Well, wait a minute, indoor trainers, there’s no GPS there.” True. However, all the- Tim Veil:

Why is this thing moving?

Paul Lawler:

Hey, where’s this thing going? So all of the signals that come from the devices gets put into that FIT file as well. And that all gets posted up to our cloud and ultimately stored in S3. So our public API and internal private APIs, it is backed by a relational data store, MySQL running an Amazon RDS based MySQL, but all of the good stuff is stored in the FIT file. And we’ll break that stuff out if we need to do optimize, let’s say, analytics or a review of that athlete’s activity that they just did, and in an optimized way to render that either on the phone screen or on a website, but ultimately, FIT is the way in which that data is stored and transported. Tim Veil:

Interesting. Now, you and I could probably talk about Wahoo Fitness forever, but haven’t always been in the fitness industry. I know you’re a big athlete, have always very active in biking, but for a long time and really where you and I first started working closely together was in education. And so tell us a little bit about some of the things you did prior to joining Wahoo Fitness because I’ve known you for a long time and you’ve always been I think one of the most competent and gifted software architects I’ve ever worked with. And I had the pleasure of working with you many years ago when we were kind of designing some very early systems, which I think we should talk about because I think looking back on the kinds of things we built so many years ago, it’s kind of funny today given the simplicity really that they had. But let’s walk back a little bit. So prior to joining Wahoo, what were you doing? Because it was an education company, I’d love to hear just a little bit about that and some of the software and architecture because you were there a little bit longer than you’ve been at Wahoo.

Paul Lawler:

Yeah, yeah, well, Wahoo had only been there for about a year prior to Wahoo, I was at Echo 360 as a VP of engineering and I was there for about seven years. I joined Echo 360 as a principal architect that Echo 360, still around today. They are an online education video platform. So if you think of you walk into a classroom, your last kind of hardcore education experience is what University of Georgia like- Tim Veil:

Right. Correct. Yeah.

Paul Lawler:

40, 50 years ago. Tim Veil:

It seems that way.

Paul Lawler:

You walk into a classroom, Echo 360 would have devices in Iraq in the classroom somewhere or in some centralized location that university specified. Professor walks in that session, that lecture is timed, I’m going to Comp Sci 101 at 11:00 AM. Those devices come on and start recording video in the classroom, multiple channels to allow for screen sharing, that sort of stuff. But the real fun stuff happens as those video streams go out, get stored into the cloud architecture and captured for on demand or live interaction from the student population. There’s a lot of problems that we’re trying to solve there because it was a fairly Echo 360 went from an on-prem solution to a cloud-based solution, which was a massive undertaking to begin with. You have a fairly stayed conservative market in higher ed or education in general. I was an ed tech for well over a decade and a good part of that was with you. So there that there’s a lot of challenges trying to make that cost-effective. Nobody likes to, it’s a tough market to spend a lot of money in because everybody has pretty tight budgets. So how do you build a platform that can scale out pretty reasonably, but also from a cost per student perspective is reasonable for institutions to take on? Tim Veil:

Well, let’s talk about that journey a little bit if we can because I think it’s one thing in my daily work at Cockroach, I think maybe this is less and less prevalent than it was a couple years ago, maybe certainly when Echo may have been going through this. But this idea that the companies, whether it’s on the app side or the data side are migrating from kind of on-prem to the cloud. I think most people would agree and certainly the vision has been, let’s get everything we can to the cloud in a hurry. But as I’m sure you experienced there and have experienced other places that journey from applications and data living primarily on-prem and data centers that you manage to the cloud, it’s not necessarily always easy or always straightforward. I mean, can you talk a little bit about what that journey was like for y’all or what some of the challenges were?

Paul Lawler:

I would say the social challenges were actually harder than the technical challenges, quite frankly. So you have an industry or these institutions have departments dedicated to managing on-premise solutions and there’s a lot of investment both in people and infrastructure to support that. So when you turn around and say, “Hey, by the way, we’re going to go ahead and move our platform over to a pure cloud-based implementation.” Immediately, they’re like, “Well, wait a minute, I’ve got a whole staff of people here. What do I do with them? I’ve got all of this money invested into servers and network infrastructure to support your solution. What does this mean?” So I think the hard part was really getting past that. And I would argue that there’s probably, if you looked at the wreckage of companies that have tried to make that change, it’s probably significant. There’s a hard transition to make depending on the industry that you’re in. Education being a pretty tough one to convince people to do something different than they’ve normally been doing. So we went through, it was more of a gradual transition. It took probably about, that started before I got to Echo. So it was about a year in before I got there. And a lot of mistakes were made prior to that and a lot of mistakes were made after that. But it took about two, three years to fully get everybody transitioned. But I think Echo had a fairly robust customer service and a robust relationship with all of the institutions that the company served. And so that made it a lot easier than it normally would’ve been. Tim Veil:

Now, you used the word mistake and it’s always interesting I think when we’re talking about software architecture and system design, very rarely do you get it right the first time. And I think one of the things I’ve learned over my career and certainly working with various customers is that just things happen. The designs you lay out at the beginning of some cycle might not always stand the test of time and you end up having to iterate mean, can you think back on … not to put you on the spot, but can you think back to some times where, “Hey, we had this great idea, this architectural design, this concept something, this pattern we wanted to employ. And at some point, for whatever reason, it just never quite turned out the way we’d hoped.” Or maybe another way to say it’s just some of the mistakes maybe along the way, whether it’s an Echo or even Wahoo farther back that you’re like, “Man, learn from that one.”

Paul Lawler:

I’m going to be an open book with you. So when you and I worked together, that was until about, what, 2008 when we worked closely together. Because we’ve been working together for a long time even prior to that. But I want to say when we were at advanced ed, it was 2008 and ultimately left there about 2012, ‘13. Tim Veil:

Yeah, yeah. Yep.

Paul Lawler:

I went to inBloom, which was an EdTech nonprofit startup funded by Gates Foundation and Gates and Carnegie I think or [inaudible 00:22:43], the combination of three. The consulting firms that were involved there before we built the software team at inBloom were, what I like to call, burning innovation tokens. They were burning innovation tokens left and right. We had to clean up a lot of that mess. But my point with all that is when I made my way into Echo 360, one of the things that I had said was to myself was, because I had gone through a number of different companies prior to getting there and being there for a while was I’m going to burn some innovation tokens. We’re going to push the envelope on learning and creating infrastructure that can last. So we’re going to try some stuff that may blow up in our face. And that was, microservices were starting to emerge as a new pattern or at least becoming more of an established pattern in … that would’ve been 2013 to 2015. And so I dabbled a little bit, I was at a startup, a bay area startup in the ed tech space for about six months. And I dabbled a little bit in some of the microservices work there just to get my feet wet. But then got to echo and they were like, “Look, this architecture isn’t working.” They had outsourced the SaaS cloud, the initial SaaS and cloud solution to a third party. And that third party was burning their own innovation tokens because they’re working with Scala and some new tech running on AWS. And I said, “Okay, well, nobody’s against this stuff here.” So it was a concerted decision to say, “Let’s go ahead and not build a … let’s try the microservice thing, not build a monolith, but let’s set up some infrastructure that helps us get there.” And that’s when myself and the team that I started building, we were playing with things like Kafka. I don’t know in terms of the backend Dynamo DB was wasn’t heavy use, but we were trying to get a lot of that transitioned over to managed relational database. I’m not sure if RDS was in play at the time, but it was really about how do we set up a … so docker was a big deal as well for us at that point. So containerized microservices with a backbone supported by Kafka and a pub sub pattern, I think was let’s push that hard. And I think the biggest probably mistake was trying to figure out, trying to get too granular on the microservice side. And so Netflix was doing this stuff for a few years. They were really good at it, but they also had a massive site reliability team and they were able to create an environment where you could deploy and manage thousands of containers. In their case, probably tens of thousands of containers running in an orchestrated manner. And we just didn’t have a lot of experience with that. We’re learning on the fly, but it was a lot of fun. Tim Veil:

Yeah, I think it’s interesting you bring up microservices. I get the sense in talking to a lot of companies and even some of the folks we’ve had on the show, I feel like the pendulum has swung a little bit here where obviously back when we started and it’s fun to think about the kinds of applications we’ve built in those architectures where it’s a single Tomcat instance, or maybe we had a second Tomcat instance. It’s this big monolithic-

Paul Lawler:

Whoa, slow down. Tim Veil:

I know. Hey, wait a minute, wait, wait, you want to have two? But ultimately, obviously you’re right. In that 2013, it’d be interesting to go back and actually kind of research what the timeline was. But at some point that I felt like the pendulum really, really swung hard to microservices. And it’s like if you weren’t doing that, if your application architecture wasn’t broken down into these hundreds of services and each one using its own database, you were somehow doing something wrong. And I think for a long time that’s kind of been the status quo to some extent. But I’m curious what your sense is now about this, because I guess where I was going is I feel like what what’s starting to happen is people are, and I think this is maybe implied in what you were saying, that people are starting to realize that in and of itself brings all sorts of complexity and to some degree uncertainty. I mean, they’re hard systems sometimes to wrap your head around and when things go wrong, it can be very difficult to triage. Is that the feeling you get that like, “Hey …” and there’s a more balanced approach here. Not everything has to have its own database. Not everything has to be this nano service that … I mean, I’ve even heard people talking about, hey, I’m fine with monoliths now. I’m like, I’m back to-

Paul Lawler:

Pendulum’s always going to swing back. Yep, absolutely. I think when we looked at what Netflix was doing, so there was a lot of stuff. Tim, you introduced me to Spring many, many, many years ago. I think that was like ‘04, ‘05. And I was like, “Whoa, what is this? This is awesome.” And Spring has done a really good job of evolving, I think, with the times. They’re always providing some high quality level of support for the patterns of the times that you’re in. And in 20, I would say about 10, 12 years ago, Spring was really the … Netflix was collaborating with the Spring team and building a number of patterns like service location, some stuff around authentication authorization in this distributed environment. And we were looking at that and saying, “Okay, we can kind of piggyback on some of these things, but we weren’t a spring shop.” So we were just replicating some of the patterns that were being built out with the Netflix and Spring collaboration. So in terms of complexity and swinging the pendulum back, we were looking at what are the patterns that these guys are employing? How do you make this actually work? And Netflix was probably not the right template because they were literally able to do things. They were deploying services that were at the function or method level, essentially, kind of a quasi lambda in those days. And so what we found was we tried to … at Echo, we tried to look at things from a domain perspective and reduce the … well, map the microservice functionality to the domain. So for example, you might have a user domain, you might have an off domain, you might have an [inaudible 00:29:26] domains, et cetera, et cetera, et cetera. But what we found was there’s coordination between the services and Kafka helped a lot of that. One of the early mistakes I made in a microservice environment was having services talk directly to one another, and that was just a terrible idea. So failed hard on that. So we should just do a whole podcast on just failures. It would never end. Tim Veil:

Yes, so true.

Paul Lawler:

So Kafka helped mitigate a lot of that, but what we found was in building out the micro, we found that it would’ve been better to look at it from a macro, so maybe not a monolith, but certainly macro services. So there are domain services that work in concert with one another, particularly on the backend from a data standpoint. And can those be grouped together in an orchestrated set of macro services? And I think the first thing you absolutely have to do before you go down that path in terms of distributed service oriented architecture is you better damn well make sure you’ve got a good site reliability team. Otherwise, it’s just disaster. I think you can pawn some of that off today. It’s getting a little bit more sophisticated with managed services with AWS for example, or some of these higher level platform as a service type companies like Heroku and others. But yeah, I think you have to take a real hard look at what your capabilities are as an organization and as a team, and then can you get to delivery in a reasonable manner. If you can’t get to kind of like CD, then you’re in big trouble with that type of architecture. Tim Veil:

That kind of brings up an interesting thought, which is I wonder, and maybe you didn’t mean this, but when you used a term earlier, which I thought was super interesting, which is the kind of burning these innovation tokens, and so maybe we could talk a little bit more about that. But what you just said made me think about, I think one of the dilemmas that folks have, which is this desire to be on the cutting edge of technology to do and use all the stuff that’s out there. And I think we could do a whole podcast just on all the crazy stuff that is emerging day in and day out. And there is a need and a desire I think for all of us in technology to stay up to date, to stay current, to use this stuff. But you also, at the end of the day, you have to deliver, build software that your customers ultimately use. That’s ultimately, I think unless you’re doing it for some other reason, that’s the ultimate measuring stick. Can you push software? Can you get it out the door can and do people use it? I mean, I’m curious your thoughts on that balance because it is a balance. I mean, you don’t want to be stuck 30 years in the past, but you don’t want to be so bleeding edge that you spent all of your time tripping over the challenges that adopting brand new technology introduces. What are your thoughts on that?

Paul Lawler:

A hundred percent. I mean, agree with a lot of what you’re saying in my role now and for the past four or five years is really about how do I get the value stream of the offering to our customers into their hands as quickly and with as high quality as possible. And quite frankly, it doesn’t have that much to do with the software du jour, the innovation du jour, right? It has much less to do with that and it has more to do with starting with high functioning teams, either hiring for that or setting an environment where teams can function in a high performance manner and establishing what the criteria for that is. And then whether it’s CICD, whether it is like bleeding edge pipelines, whether it’s manual, whatever that process is by which you get your service into the hands of your customers, that has to be reproducible and consistent. And if you’re on a two-week schedule, a one-week schedule, a daily schedule, a minute by minute schedule as you commit and approve commits and get into master branches and deploy immediately. To me, that doesn’t matter. As long as it gets into your customer’s hands in a reproducible state, a consistent state, and they’re happy, then that’s ultimately what my job is about these days. So I’ll evaluate in my days as an architect, it was a lot about, “Hey, how can I leverage this technology to not only satisfy what the organization’s trying to do, but also my own personal learning goals.” Don’t let anybody lie to you. That’s something that we all do, right? Let’s find something new and shiny and go play and I love it. That’s a lot of fun until you get from prototype to actual production level implementation. So I think a lot of people, when you’re faced with the reality of operating for a large customer base at scale with high SLAs, in my Echo days, our SLA was five nines in terms of video consistency. If you used echo devices, our hardware devices, you would not lose a video. You would not lose a lecture. That was our commitment to our customers because a professor could go in there for three hours, do a lecture, and then we’re like, “Sorry, we lost it, it doesn’t exist anymore.” That’s just absolutely unacceptable. So then you start moving towards processes and technologies that are just absolutely battle hardened. And what it does is it takes you back probably to some of the technology that we used in our early days in our careers, Tim, that were just tried and true relational databases to some extent, an application architecture, a stack that Java or something that’s fairly battle hardened in the application space and then Agile is pretty much ubiquitous. But how you go about implementing that is where it’s all evaluated in terms of its effectiveness. I think things like if I were to recommend to people, “Hey, how do I make sure that I’m delivering to real world customers in a high uptime, scalable manner?” I would look at things like check out DevOps handbook. I think, what is it? Is it Mark Larson, elegant puzzle like systems engineering? I think that’s his name. Will Larson, sorry, former guy from, I think he was at Stripe, Uber, and a couple. Tim Veil:

Really?

Paul Lawler:

Yeah. Tim Veil:

Interesting. Okay.

Paul Lawler:

So tones like those are ones that give you a- Tim Veil:

Tones.

Paul Lawler:

You like that? That gives you a pretty good perspective on what it would take to build out high quality systems that ultimately serve your customers. That’s really what it meant. Tim Veil:

Yeah, I think that’s the key is that you can’t ever lose sight of that end goal. And it’s interesting, obviously, you kind of mentioned that, I don’t think you used this word exactly, but resilience. I mean, you have to build systems that last, that are available in the olden days or Echo, the video dropped, that’s a big problem.

Paul Lawler:

Well, I will say this, all systems depending on the organization and their level of investment, whether they’re using innovative technologies or not, are resilient. They’re just resilient with human beings. So it just depends on where you want to put your money because that what ultimately happens there now when it comes to video streaming, in the case of Echo 360, if you’re not capturing that stream appropriately, then it could be gone. When it comes to Wahoo, we capture that FIT file, it’s redundant. We have interstitial captures of the activity file. We’ve lost a few, admittedly, but the stakes are much smaller in that case, but we have protocols in place to make sure that that happens very rarely across millions and millions of customers. Tim Veil:

Well, I think one of the things that have evolved just how our view, I think of application architecture kind of continues to evolve over time. I think expectations of software providers and companies is really, really evolved. I mean, again, I think back to when you and I started, we would take this … I think had, what did we have? Basically it felt, whether it was or wasn’t, I don’t recall, but I felt like we had a full day we could take the system down for a full day, couple hours, do whatever we wanted to, and our customer base kind of allowed that. But that was the kind of outage, that was the kind of resiliency that you could afford to have. And in this case, our customers would continue to use the system. Echo 360, you’re providing five nines. I think the expectations for software have and applications and architecture have really changed systems. It can’t go down, shouldn’t go down. Even under strained circumstances, people no longer going to say, “Hey, do you mind if we take this thing offline for a day while we release new software?” I mean-

Paul Lawler:

It’s funny you mention that- Tim Veil:

… you can’t get away with that anymore.

Paul Lawler:

… because when I left Echo, not long after we went through an acquisition, so after the dust settled on the acquisition, I hung out for a little while to restructure the engineering organization to fit the combined companies and all that. But part of the acquisition process and due diligence was literally around that. That was a big piece of, because what we called ourselves a platform, and so the acquiring entities were like, “Okay, well, explain to us how this is a platform.” And a lot of it was in terms of uptime. And I think what they were used to was most companies that they were involved with were like, “Oh yeah, we’re up most of the time.” And we’re like there is no downtime. Literally we have a no downtime policy. Now our SLA is five nines, but from a practical perspective, there was no downtime in the system whatsoever. That was a big selling point because they were like, “Well, how do you do that?” And I’m like, “All right, well, let me show you our container orchestration.” Tim Veil:

Yeah, it’s it. It’s so important. And obviously in the work that we do in the company that I work for, I mean, high availability, resilience, it’s the whole name.

Paul Lawler:

Right. That’s what Cockroach Labs said. Tim Veil:

Yeah.

Paul Lawler:

That’s court of the product. Tim Veil:

That’s why we named it the way we did. What can’t you kill? Well, you can’t kill a cockroach.

Paul Lawler:

Go nuclear. Tim Veil:

Go nuclear.

Paul Lawler:

Tim Veil, Cockroach Labs. Tim Veil:

The only thing remaining, that’s all. One of the things you’ve touched on in the last couple comments and then maybe we’ll kind of wrap up on this theme unless we discover some other interesting avenue to go down is people, right? You’ve mentioned a few times just kind of the teams that you’ve built, the kind of people. I’ve always thought that that’s super important. But as you well know, it hasn’t exactly been easy over the last couple years, kind of building teams, managing teams. Maybe just talk a little bit about your philosophy on team building, developing or building engineering teams and then maybe what has it been like over the last couple years trying to get everybody rallied around whatever your vision or mission is?

Paul Lawler:

So I philosophically, I think the way I would look at it is take your org chart and if I’m at head of software, VP of engineering, whatever it is, CTO, take your org chart, flip it and put me at the bottom. I’m a facilitator, I’m a support entity. I’m overhead at this point in my career, I’m overhead for my engineering teams. My job is to make sure that anything that gets in their way, in terms of what we talked about earlier, delivering that value stream to the customer on a fast, consistent, reliable basis. Anything that gets in their way I need to take care of, I need to help them get around them. That could be technology, it could be human resources, it could be politics of the organization, whatever it is. Yeah, that doesn’t happen. That never happens, never happens. More than two people in a room. You’re going to have politics. So that’s my main function and that’s kind of how I look at it from a role standpoint. So my job is to make sure that I’m competent enough in those various areas to know how to navigate conflict. Whether it could be technology conflict, I want to use X versus Y, or we have a vendor that’s pushing a particular solution. Is it better than homegrown or another vendor? Whatever it is. People conflicts are probably the biggest part in communication. In my mind, software is mainly about communication, less about technology these days. I know that sounds kind of weird, but I believe that. But getting teams effective, I think Agile was kind of a big deal for me back in, let’s see, this is in my early engineering days in, I want to say like ‘03, ‘04 when the manifesto sometime around then when the manifesto came out, early 2000s I believe, if I’m not mistaken. Tim Veil:

I think that’s about right. I don’t recall anymore either. But yeah, I think that’s about right.

Paul Lawler:

And so having a decent Agile process, whether it’s Scrum or Kanban or Safe or whatever you’re doing, people are bought in and then you have sponsors in the organization that are pushing some sort of process. Waterfall, those types of approaches, while it helps senior management get their heads around what it is they’re doing, it just doesn’t serve the organization well. And I’m not trying to use this Agile everywhere, it’s a blanket solution because you could talk most teams, I would say probably if you walk into any organization today and you say, “Yeah, we do Agile.” They’re probably doing some half-baked approach to whatever Agile process they say they signed up for. Scrum is probably one of the worst processes that I see implemented in most organizations. At least- Tim Veil:

You mean the quality of how it’s implemented?

Paul Lawler:

Absolutely, yeah. I mean, it’s just terrible. Tim Veil:

Yeah, I agree with you. A lot of people say they do things that aren’t actually true.

Paul Lawler:

Yeah, I mean, for the automation side of it is generally bailing wire and bubblegum. And that’s why I was saying earlier like your SRE team, your delivery teams, your integration teams, those pipelines that you’re building and establishing and investing in, those are the biggest parts of that Agile experience because it’s all about delivering that value stream that what you do before that doesn’t really matter. So in terms of setting up teams for success, I’m a big believer in the small team approach, highly collaborative. Honestly, Tim, our interview process I think was probably one of the more unique, but I feel like we would always hire for that kind of personality fit, that highly collaborative aptitude. And I feel like we cared less about the technology that those people had come into contact with. And I think we did it naturally when we were together at advanced ed. To some extent we did that at Soul Tech in our consulting days. But I would say now I absolutely look at technology last on the resume. Honestly, if I had to rank everything, I would look for personality fit with the organizational culture, aptitude, curiosity, learning aptitude. Second, a general understanding of computer science, software engineering principles, things that are lasting that don’t pass with the winds of technology. And then specific tech way at the bottom. And I think that’s kind of critical. If you can build your organization around that, then you’re going to be able to handle those changes as technology grows or you get new tech. ChatGPT is a great example. Kind of playing around with that lately, I mean, it doesn’t replace the person that’s got a great cultural fit and a high level of aptitude for learning. Tim Veil:

I couldn’t agree with you more. It’s all about the people dummy. And you just can’t replace wonderful people with the best technology and technical acumen alone is interesting and helps in certain cases, but if you don’t have that cultural fit, that ability to work well together and across teams, I mean, teaming is so important. One of our philosophies here is kind of this one team attitude and something we lean really, really heavily into to break down silos to work really well together across teams, whether it’s software or product or engineering. No more can you just be this narrow focused technical person to be successful. I think in modern organizations and ultimately to deliver the kinds of things we’ve been talking about earlier, value for customers, you have to build teams that are capable of not only being successful in an external facing way, but in an internal facing way to be able to have meaningful conversations, build meaningful relationships internally. It’s just so important.

Paul Lawler:

I’ll say companies like yours, it mitigates the technical kind of depth that an organization may or may not want to get into. So for example, “Hey, look, we’re on a relational data store. I don’t know how to scale this thing out. I’m going to call Tim, he’s going to help me out, he’s going to help, he’s going to take this off my hands.” And I’m sure you guys are very successful in doing that sort of stuff. But what I can’t do is call you and say, “I got a problem with a senior engineer who’s a “brilliant asshole” and he’s corrosive to the team culture. That’s not a phone call I make to Cockroach Labs. I might call you for advice, but I won’t call Cockroach for that. Tim Veil:

No, it’s so true. And those are the kinds of folks that you just have to be so careful of. Because they’re kind of everywhere. So just wrapping up maybe, because this has been great. So enjoyed chatting with you and you’re right, we could talk.

Paul Lawler:

This is long form. Tim Veil:

This is long form, long form podcast. Hopefully people are still listening, but it’s shorter if you listen on what is it, 1.2 or 2.0. I mean, who knows? People may be listening at high speeds, so it’s gone faster for them.

Paul Lawler:

I talk fast too anyways. Tim Veil:

That’s all right.

Paul Lawler:

Comprehensible. Tim Veil:

Two things really I want to end on. So I’m always curious what people are reading just because I’ve kind of been picking up some new books lately. Curious, and you are always a big reader, maybe not now, but certainly back when we were working together, I always looked to you for good advice about things to read, both on the technical sphere and other places. So what are you reading and whether this is personally or professionally at Wahoo, what are you looking for? This is the beginning of our fiscal year. What are you looking forward to for us for the next 12 months, this upcoming fiscal year?

Paul Lawler:

For Wahoo or just in- Tim Veil:

In anything. If it’s Wahoo, great. If it’s personally, what are you looking forward to this year?

Paul Lawler:

Gotcha. Okay, so let’s talk about reading. So I’ve recommended a couple of books for people. I’m always referring to those. So I’m always popping in on Will Larson, anything. He’s got a new one out called Staff Engineer. I like that he’s got a few of those chapters on his website. So just search for Will Larson and Staff Engineer and you’ll see some of that stuff. And that’s kind of cool because at Wahoo, one of the things that I’m working on is what is the professional career development path for my engineering team. And that’s always been pretty, every organization I’ve worked for has been pretty much ad hoc and you guys seem to have a pretty good handle. Because the conversations I’ve had with you and Cockroach, you guys seem to have a pretty good handle on it. But I think for most organizations that are maybe not now what I would call high functioning, typical might not be the right word, but prototypical, internet, cloud-based engineering organizations, it’s just all over the place, right? Tim Veil:

It is. It’s very tough, very tough. You ask our people team, I mean, it’s something we work really, really hard at.

Paul Lawler:

Yeah, so that’s great. And I just think, so getting myself grounded in how to create a professional development environment at a company that is more in that ad hoc, kind of like the culture at Wahoo is very, very casual, very, very ground up kind of self-managing. And that’s great in some cases. And in other cases, I think it’s a barrier to people who need some level of guidance through that career path. So anything from Will is great DevOps handbook. I would always go back to that because again, that is pushing the value stream, that it’s primarily about the value stream and getting that into the hands of the customer and using what people think that DevOps is a noun. It’s not. It’s a verb, it’s a state of mind, it’s a philosophy. Tim Veil:

State of mind. I like that.

Paul Lawler:

There was one other one that I wanted to mention too that was really good. I’m reading anything that comes out of the ThoughtWorks technology radar, I’m all over. That’s always good to read. There was one other, it might come to me, I lost my train of thought there, but- Tim Veil:

That’s all right.

Paul Lawler:

… might pop back in a minute. And then what was the other part of the- Tim Veil:

I mean, what are you looking forward to this year? Yeah, what are you hopeful for? What are you looking forward to?

Paul Lawler:

World peace, definitely like that. So at Wahoo, I think we are doing some pretty interesting stuff. Not a whole bunch that I can talk about, but really, again, refocusing on our … we’ve gone through some economic turmoil as a lot of the tech industry has and particularly the fitness industry. I mean, it’s very well documented, but doubling down on our mission to build the better athlete. And then really focusing on our vision of ecosystem. So I’m working on a few things that is very ecosystem focused. And by that I mean bringing your watch. These are all products. Our bike computer, I had to go and plug that stuff here on the podcast. Our trainers and our software more in line with one another as an ecosystem, as a fitness platform as opposed to a series of devices that you just interact with more of a true fitness platform. So working on some neat stuff there that I’m hoping we’ll really come to fruition as we work our way through 2023 and particularly into 2024. So professionally, that’s where all the oxygen is getting sucked up. Personally, I don’t know. I’m just trying to survive day by day. It was fine. I got a call from my daughter last night, 11:30. So Tim, you’re a UGA grad? Tim Veil:

I am. Proud UGA grad.

Paul Lawler:

There you go. She’s a proud UGA undergrad, a freshman. And I encouraged her to, she’s always been really good at math and science and that sort of stuff. So I encouraged her. I said, “Look, take a look at some of the sciences,” whatever. So she’s now in the science program at UGA, which is a combination of computer science, statistics, all that jazz, right? Get the call at like 11:45 last night. Hey, I got a project due tomorrow, can you help me out with a couple things? So what do I do? I run downstairs, fire up the old, because they’re using Java in this class. Fire up the old editor, but low and behold, she figured it out on her own. So we were on a chat and she managed to … she had some variable initialization that wasn’t quite right, but she got it sorted out on her own. But it was fun. So personally trying to get through kid in college. And then two, trying to get the boys through high school without burning the place now. Tim Veil:

I feel that pain, for sure.

Paul Lawler:

I know you do. Tim Veil:

Well, listen, Paul, it was so great to catch up with you. Hopefully folks who may be listening learned a lot because if it wasn’t clear already, you’ve had such, I think a wonderful experience kind of not only building software, but I think as your career has evolved since we last worked together, really leading such high quality and high functioning teams. I know Echo 360 and certainly Wahoo today are very lucky to have you at the helm of their software. So once again, thank you so much for joining us. It was a real pleasure having you on and I look forward to learning more about Wahoo. Maybe I’ll get back on the bike, one of these bikes-

Paul Lawler:

Let’s get your trainer. Tim Veil:

… one of these days. Get me a trainer, get going. But we’ll be in touch soon, I’m sure.

Paul Lawler:

Tim, appreciate you having me on today. Thanks for great conversation. Probably could do this for another three, four hours. So anybody that’s listened this far is either insane or they’re actually getting something from this conversation. But yeah, it’s been a pleasure and I appreciate all the kind words and right back at you. Tim Veil:

All right, talk to you soon.

Paul Lawler:

All right, man, take care. Tim Veil:

Thanks again for listening to this week’s Big Ideas and App Architecture. Be sure to rate us on your favorite podcast platform and tune in to a new episode every Tuesday. Thanks again. See you soon.