Tim Veil: Welcome, again to another episode of Big Ideas in App Architecture. I’m pleased to be joined today by Systems Engineering Fellow at Paycor, Adam Koch. Adam, welcome to the show. Very glad to have you. Just like we have in past episodes, what I’m interested in hearing about from you first is a little bit about your background. How did you get started in this industry? How did you end up at Paycor? Tell us a little bit about Paycor, and then we’ll see where it goes from there. But again, welcome to the show. Adam Koch: Yeah, I’ll hit the rewind button for one second. My mother worked actually for Playnet, which was an internet provider, online services provider for Commodore, which eventually, after a couple moves became AOL. So, we had a computer in the house really early on. Between that machine and like a spiral bound, basic, how-to book or whatever, you just got to get involved with computers a little bit early. I got really lucky. In my high school, they did the green screen to Wintel, Windows Upgrade while I was there. So, we went from dealing with Apple first generation to Visual Studio my junior year. So, even though I still thought I wanted to be a doctor at that point, I took programming classes, really enjoyed it, or whatever. I enrolled in University of Michigan thinking I was going to be- Tim Veil: Oh, really? Pre-med major, huh? Adam Koch: Yeah. So, I took a 100 level computing course, just because I wanted to do it for fun and I enjoyed it. I really did not enjoy organic chemistry. So, just things kept knocking and knocking and knocking. I’m like, “You know what? This is probably where I need to be.” I worked for the campus computer store. I did onsite computer service. I was the computer guy for a few years at the campus or whatever, doing all the faculty and week house’s computer stuff. It taught me to debug and diagnose everything that can go wrong with computer from A to Z. But then leaving college, I got lucky and found a position over at Nielsen, famous for the TV ratings, working in the Consumer Packaged Goods Research Division down here in Covington, Kentucky. Yeah, started .NET development, focusing on making their systems faster. They were a global company. So, a lot of it was about delivering some of these applications around the world, especially large reports at a time when internet wasn’t the best and things like that. So, offline background copying and things like that. With us being an in-house development team for their HR and financial services, effectively, every couple of months would be some new problem that we had to solve for. So, it was interesting, being thrown different problems at different times. It was that desktop to web transition or whatever. I just got it in a really lucky time. So, pretty much at some point, they decided to stop focusing on in-house software and wanted to convert to SAP. So, I went to a recruiter, and two weeks later, I landed at Paycor. I started there as a dev two and went up from there. Paycor is an HCM provider. So, when I joined, it was a small handful of developers working on three or four systems, largely desktop based. It was interesting, they have this era where they tried to fake the web through like Citrix Remote Desktop-based software. It was better than mailing out CD-ROMs and things like that, but it definitely was not the web experience. So, I got in at a really good time. They were just spinning up their next generation web-delivered payroll HCM solution. So, I was on for the ground floor of that. Tim Veil: I mean, everybody listening to this may know this acronym and maybe I should know this acronym as well, but HCM stands for what? Adam Koch: Human Capital Management. It sounds bad, but thinking of humans as a resource that you have to manage. A lot of people’s major spend in an organization is on their people. Their success is driven by their people. So, it’s really about getting the most out of your workers as possible. Tim Veil: You’ve been at Paycor for about 12 years. Is that right? Adam Koch: Yeah, 2011. Tim Veil: So, I mean, you’ve seen all sorts of change, I would imagine. I mean, you hinting at some of the change that’s happened from a technology perspective, but also, your roles and responsibilities have changed. I mean, I guess, tell us or tell me, systems engineering fellows, where you are right now. What’s your domain? What are you doing for Paycor in that capacity and role? Adam Koch: Yeah. So, I came up to Paycor as a developer working on originally our single sign on, because when I first started, we have three or four passwords to use our suite. That by today’s standards is crazy. If anything, you want to have one ID that’s not even from Paycor, where you want to have a third-party identity provider. But we brought everything together there. Then it was as we acquired software, could we have all that be a single source of truth, because you don’t want to enter your data twice. There was systems where, again, we wanted to go from desktop to pseudo web to web. When I say pseudo web, we got on right when Flash and Silverlight was at its height. So, we designed an entire product around Silverlight, which is beautiful. But when the web browsers decided not to support those SDKs anymore, we had to quickly move over to the more new native web experience and things like that. So, it’s been a journey. As through all of that, Paycor realized that it was growing really fast. Then if it didn’t focus on some of its fundamental architecture, that it wasn’t going to be able to scale. That’s everything from data architecture to moving from a data center based process to the cloud, things about having contracts between the systems that were at the database level. We wanted to move up to [inaudible 00:06:15] services and things like that. Then we wanted to do events to propagate employees being hired, employees being terminated, payrolls being run. So, that way, people didn’t have to pull your database tables anymore. It’s just the classic evolution of software. It just happened very, very quickly. Because Paycor always had a great product, a great customer service. With that came a large increase in user population year over year. So, every year, simply just to keep ourselves afloat, we had to innovate and we had to scale our system. Tim Veil: Yeah. I think that’s one of the reasons I think the conversation you and I started earlier in the week and hopefully, we’ll have today is really an interesting one for me, because in the podcast, we’ve had and certainly in my couple years with Cockroach had the opportunity to talk to all sorts of folks from different backgrounds and different types of industries, for sure. But people in different size companies have been there from the very beginning or have just joined a company or started a company, what I think is really interesting about your background is here, you’ve been with Paycor for 12 years, really gotten to see a front row seat at some really dramatic changes in the industry, right? I think, you can speak to, hopefully, we’ll get in today, some of the real challenges of taking an organization, their infrastructure, their IT, their bread and butter in some ways, and massaging it, shaping it, helping it grow and change, as the industry has really radically changed. I mean, as you mentioned, things like Flash as a way to deliver technology, we don’t think about that these days. Nobody goes out there and says, “Hey, I’d love to go build a real Flash app.” But working with a company through that entire journey, I think, is really a fascinating perspective, one that we don’t get a lot. I mean, people especially today, they jump around a lot. I mean, they’re at a company for a year or two and then move on. I think 12 years and seeing all that is really, really interesting. Adam Koch: Yeah. I would say that one of the reasons I’m still with Paycor is that because it’s grown and innovated itself, there’s been opportunities here. Normally, if you wanted to do distributed systems, if you wanted to do a different delivery platform, if you wanted to go to multiple clouds, and things like that, you may have to do tours of duty at different companies. For example, that we’ve taken a multi-cloud approach between wanting to diversify, acquisitions, landing with whatever technology set it is. Those have opened up opportunities for myself and my co-workers to effectively stay out of cutting edge. Usually, a lot of times, when you interview someone who’s worked somewhere for 10 years, they’ve done the same job for 10 years. That’s even 1 year, 10 years in a row. It’s really great that we have the right people that want to offensively rethink what we’re doing every three to four years. Tim Veil: So the cloud is an interesting topic. I wonder if that’s when we could get into just a little bit, because I think, obviously, to say it’s all the rage is probably an understatement. I mean, I think every company on the planet has either made that transition or thinking about it or in the middle of it, I suppose. Being there 12 years, you’ve seen it all. I mean, you don’t have to get into all the dirty secrets and details, but maybe talk through a little bit about what it was like for you and your teams transitioning, if you will, from older legacy systems and approaches to the cloud, because the cloud offers a lot of stuff. Adam Koch: You’ve probably heard this from people, but there’s the cloud done wrong and the cloud done right. You can have VMs in the data center just as easily as you can have VMs in Azure and AWS and Google. What it really comes down to is, is it affecting your business agility? Can you scale? Can you effectively cost shape your stuff and not have a really high burn rate? So what we did is we went on a journey. Effectively, we started out in the cloud, because we have a time product. There’s this whole idea that you cannot not be able to punch. Whether it’s 3:00, 5:00, 1:00 AM, everyone’s punching in and punching out some point. We wanted to have three or four nine’s ability for people to be able to punch in. So, we just said, “Let’s be cloud first.” At that point, the world was a little more IS. So, we were building what would be almost like a server up in the cloud. But one of the major pivots that we’ve made is moving into platform as a service and then moving platform as a service into serverless. What we want to do is we want to abstract ourselves away from having to do anything with OS patching, because have you ever been through an OS patching cycle, it can take down your company for months, just to certify old applications that don’t have to test automation and things like that. So, that’s the benefit. We partnered effectively to cloud move, which runs through the cloud, with a really strong CI/CD platform. We have a wonderful what we call a DevOps team here. We misuse the term slightly, because it’s effectively the tools that enable DevOps, we call DevOps. But effectively, we can self-service an application all the way from you give it a name, you give it a scale size, you give it a code, is it a node, is it .NET, whatever. We effectively can in a couple of minutes get you a test environment. Once you certify that in a few minutes, we can be in production. That has really been what has allowed us to adopt the cloud well, and quickly, is using platforms as a service. We effectively say punch from the list. Do you want an application? Do you want to write us cash? Do you want a queue? Do you want to be multi-region? One of the topics that your guests have talked about is resiliency and things like that. We definitely have to make sure that everything we build can be up, the three nines plus or whatever. You’re misusing the cloud if you’re not doing multi-region, availability regions, and things like that, because you’re saving some money. But the cloud, everyone tells you, is designed to go down periodically. They cannot keep it up any better than any other server in the world. If anything, it’s shared infrastructure. So, there’s a lot of load on it. So, yeah, so I think it’s half of us go into the cloud, which is guys, check the box, but half of it is us doing it really well. Especially every year, we get back to it to the point where we’re taking our first generation cloud fives and moving them into our latest cloud paradigm. So, that way, we can end of life the first generation cloud as much as we can apply our data center operations. Tim Veil: Now some of this infrastructure, because I think this is where people really get stuck sometimes. It’s that infrastructures code or whatever you want to call it. Was this stuff you guys built at home? Is this you’re using some of the big brands out there, a little combination of both? I don’t know what you can and can’t or would share, but I’m just curious, how did this evolve? What’s it look like today? Adam Koch: Yeah, so I get asked both by people we interview, people who join the organization like, “Oh, TerraForm exists. Why aren’t you using TerraForm? Azure has like Bicep and there’s like CDK and different things over in AWS. So, if things exist, why are you doing anything yourself?” Effectively, what we did is we put a usability layer around everything. We said, “Hey, you know that you need an app, you know that you needed to scale, you needed to have an authenticity, you needed to have access to certain key value pairs for configuration management. That’s all we asked you. Then everything else was abstracted.” We haven’t had to really pivot the next layer down of what we actually spin up just because Azure tends to be backwards compatible for a period of time and things like that. But largely, when you give us those directives, we can create everything you need. A lot of things, if you look at this documentation, they’re like, “Oh, well, first, create this container, then put something in the container, then put a load balancer in front of a container.” The thing is what seems like a really simple application ends up being five or six steps. We have abstracted all of that, and if nothing else, it forces compliance. One of my big things is like you can tell people all day long, “Here’s our standards, do it with this naming convention, do it with whatever.” So we at the front door get you to name things properly, size properly. We only give you a whitelist of what you should be using and things like that. Then worst case is if we ever make a mistake, because humans are fallible, we have everyone doing the same wrong thing. Then we can upgrade everyone at the same time, versus having sprawl of different technologies in different places, things like that. Tim Veil: Just out of curiosity, because you hinted at it in a couple of ways. I mean, certainly, this idea of OS upgrades, potentially taking things. I mean this approach that you all have taken and to me sounds like you’ve really refined over the years. I mean, what has this meant to Paycor? I don’t mean put it in dollars. Obviously, you have an advanced way of provisioning and deploying assets. I would imagine, I don’t want to put words in your mouth. But I would imagine this is actually quite useful and beneficial, but I’m just curious how you and others think about the investment you’ve made here. Adam Koch: Yeah. So, I won’t pretend that for sufficiently complex application, that it’s not going to take weeks or months to get something out the door, because there’s collective business rules, there’s testing and all those things. But the idea here is to effectively hit the floor of getting something out the door. A good example is during COVID, there was this concept of a PPP loan, Payroll Protection Plan or whatever. What happened was there was only a really small window, where the government was going to allow you to say, “Hey, I have this many employees. Here’s two months with a salary cost.” They were effectively going to give you a loan, which could be forgiven if everything fell into place. We were able to within a week, you had a UI, a data extract setup. So, if you were a customer, you effectively picked [inaudible 00:15:54] have more than one business unit with us like a franchise of restaurants, things like that. You had to pick your business, hit a button, and we dumped all the data that you needed to fill out that form in a couple of seconds. So, we believe that the people who effectively did that process gotten in front of the line for those loans, which ran out of funds pretty quickly. They did a second funding cycle, but the idea that we were able to put that in our customers hands, we had all their data, because they’ve been running payroll with us, we just had to make it accessible. The technology, both from our infrastructure code, we have things like reusable UI components. We have a pretty standardized text deck pattern or whatever. So, really all it came down to was the person running the data extract had to put that in there. That was it. Tim Veil: Now, you happen to mention a few times, they’re my favorite four-letter word, which is data. I could guess this, but I also know from talking to you, I mean, Paycor’s a big company, grown certainly through acquisition. I’m sure there’s lots of data in lots of places and probably lots of technologies at play. Again, what I think is so fascinating about your experiences, they’re from very early days and you’ve probably observed. Correct me if I’m wrong, I don’t want to assume too much here, but you observed, I think, how, for example, data architecture has evolved in the last 10 to 15 years from traditional relational systems to no SQL and document stores. Then all of a sudden, we got into big data Hadoop. Then, of course, the world that I occupy today, which is new SQL, distributed SQL, you’ve probably dabbled in all of it. Just curious a little bit about how the data infrastructure and tooling has changed or how you’ve observed it changing over the years. Adam Koch: Yeah, it’s definitely evolved. Early days, you had a transactional database system. The way reporting worked is you pointed to report system to the transactional data system and you very carefully tried to run a big report. That was early days. Then scream to people when they swamped your system, I remember that. Yeah. Then there was some basic phases of like, “Oh, we should have a copy of the data over here. So, that way, that’s the read intent. Then we have our transactional rewrite system over here.” Again, the step, step, step. Then we got to a point where people didn’t want… This is to be very clear. Some businesspeople, all they want is literally a database in Excel format. We try and figure out how to manage that, but other people want business insights. They want to know, “What was my salaries for the month? What was my time-off patterns? These are people who are missing work on a consistent and recurring basis.” So more insights, so even if it’s still classic database base, trying to figure out, “What insightful database patterns can we provide to our people?” Abstracting all of that, of course, to visualizations and reports and widgets, things like that. Then you touched on it. There’s a whole concept of big data. When we think about it, we have a lot of cases, seven years of our customers data, every punch, all these pay stubs that we’ve run, performance use and things like that. What answers can we get out of those things? With cloud scale, there’s no longer a limit on how many bytes you can store it. You may store two or three different times in different shapes or whatever. We’ve leveraged Snowflake, because in some cases, one of my big favorite things is let a tool completely remove a workload off of you. Because in some cases, because we have PhDs working on some of these things, we can’t do that one thing better than them doing. So, then we effectively reinvest that time we saved back into our business layer, making sure that we can make things usable, supportable. The data quality is there because garbage in, garbage out. But yeah, at this point, the intention is cloud scale data both from a horizontally scalable transactional system, having real-time event. So, that way if you need to know about something a moment later, to be able to have both for ourselves and our partners or third-party partners and then of course, the reporting system, all of those things within good SLA. So, we’re not reinventing the world. The major thing is, is that to effectively put that customer service layer around everything that makes it usable, because ultimately, a lot of our customers are, say, a four-person pizza shop or whatever, and we want to be able to make it as accessible to them as someone who might have a data team. Once in a while, someone reach out to us because they want to do a Tableau. They want to do some sort of Power BI on their side. We want to enable that, but at the same time, a lot of our customers just want the answers. They don’t want to have necessary inspection into the actual underlying system. Tim Veil: Awesome. So, just going back into some of the data stuff, in terms of architecture, size, I mean, you could talk a little bit in that. Another topic I want to touch on with you, because I think it’s interesting. Again, we talked to so many companies who have dealt with this is just growth through acquisition and onboarding not only people, but technology and data. Just curious to hear a little bit more about how you’ve tackled some of those challenges and ultimately, the tools you mentioned, Snowflake. I’m sure there are others. What are some of the core pieces of that data infrastructure? Adam Koch: Yeah. So, even though we are classically a Microsoft SQL shop, we’ve acquired people who were like, say, pre-Aurora, relational database in AWS, Mongo, and things like that. The major thing is that, of course, everyone tries to achieve a certain domain driven design or whatever. As we acquire domain, they’re already separate. They’re already adhering to those boundaries and things like that. So, the best we can do is maintain those. The things that they’re missing are our master data, our authentication, our security policies, our navigation and things like that. So, what we’ve tried to do is we’ve tried to provide those teams with the onboarding toolkit and we’re constantly evolving this. So, that way, within, say, a small period of time, they’re recording numbers, that we can effectively say, “Hey, one way to syncing up all the master data, all of their insights come back into our reporting platform. Authentication works both ways and things like that.” Because honestly, one of our first acquisitions, it took months, if not years, to really make it work properly. Now, we’re in a matter of days, the intention there. It really comes down to effectively where they’re at. Some people have done partnerships and integrations as part of their business. Some people were a start-up. Really, all they did is had their own authentication identity provider and they just had to learn to integrate as part of the acquisition pipeline. Tim Veil: That brings me to another, I think, fascinating topic, which we’ve talked to other folks about and it’s really around, I think, broadly, lessons learned. So, you hinted at one, right? Hey, first time we did this, it took years. Now we’ve got it down to a science. Again, talk only what you’re comfortable talking about. I mean, what are some of those lessons learned, having again, this front row seat to some pretty big changes, I think, in a company, where maybe you’ve made some poor technology decisions, architecture decisions? Not necessarily the time but in the rearview mirror, you’re like, “Oh, geez, I wish we would have tackled this or done this differently.” Maybe there aren’t any, but I just curious what your thoughts are there. Adam Koch: One of the funny things depending on your journey in technology is like, “Hey, we want them to feel like they’re part of our application suite or whatever.” So stylesheets, that was a thing or you know what, let’s go ahead and just throw our banner across the top of their application. That’s from an iframe. We don’t have to- Tim Veil: Yeah, iframes. Adam Koch: Yeah. Other than that, that was a thing. Really, our focus on our UI framework has been a lot of React components and things like that. We dabbled in some of the more classic web components or whatever, but we really centered around some React stuff. The idea there is that we’ve been lucky, a lot of our acquisitions have been React based. So, you can provide them with things like navigation bars, and modals, and chat pop-ups and things like that, things that all of a sudden, immediately, give them a toolkit almost to come online with us. In some cases, we just have to rationalize the fact that they’ve done some of the same things as us. Which one wins? Threes were acquired, and there was a lot of times they win the arms race as far as features and things like that. Then at the cloud scale, pushing around a lot of data at the start to simulate how things should be is not off the table and then obviously, we correct that and have true sources of truth and things like that. I would say that overall, the intention is that we want to make sure it’s a stable experience. We want to have it be a very usable experience. Then as things fall into place, we have smoothed the engineering edges off the way, going back to the whole infrastructure as code serverless movement and things like that as time allows. Tim Veil: A lot of the folks we talked to and a lot of the businesses we interact with, I mean, downtime, we talked a little bit about, you’ve mentioned, resiliency and certainly, multiregion plays into this, but downtime can be a big burden to people. You can imagine different industries, where minutes of downtime can cost millions of dollars. What does that stuff mean for Paycor and Paycor’s customers? Do you feel like that is a huge issue? Do you guys have very stringent SLAs? I mean, just curious what the approach is there. Adam Koch: Yeah, the way we look at it is when we’re down, people either can’t punch in, can’t punch out, they can’t process their payrolls. If you’re not familiar how our ACH system works in America, it’s not instant. You a lot of times have to tell people that they’re getting paid on Wednesday, so it can be in their accounts by Friday. There’s modernization working its way through the banking system, but a lot of things are still run off very old processes that effectively have to have a couple of days lead time to get things paid. So, if someone can’t do their payroll the date they’re expecting to, checks may be late. Of course, a lot of people, their check being a day late means a lot of problems in then their lives. Tim Veil: A lot of problems. I can attest to this. Adam Koch: Yeah, there’s situations where we are on the hiring side, making sure that people can get people in the door. We have a lot of different struggles lately, keeping places staffed, so if we are in the way of that stuff. So, we treat it very seriously as far as can people get into our system, is it up? Like you mentioned, we use everything from multi-region to fill your modes and things like that. If we can’t get you maybe some of the insights that we’re trying to deliver to you, we still let you punch, we still let you access emergency phone numbers and things like that. Our intention and hope is that we have to give you the full experience most of the time, but we take it very seriously what we do there. Then of course, our stuff is delivered web, mobile. We have time clocks and things like that. So, if you lost your internet at your facility, it just sends it up when your internet’s restored and things like that, trying to make sure that the whole process of having to walk up to your manager and say, “Hey, I punched at 9:00, but it wasn’t working,” whatever, that adds a bunch of time and effort to a working manager schedule they don’t need to deal with. Tim Veil: Yeah, a whole factory floor of people or something saying, “Hey, I tried to punch it.” Yeah, it could get- Adam Koch: If you’re in a supermarket when the power goes out, they’re not looking to ring people up by paper. Tim Veil: Yeah, no, that’s not good. I didn’t realize that, I mean, maybe I should have about the payroll system that there were so many hard and fast rules, like if you don’t get it in on time. So, if your systems go down the night that that’s supposed to happen, that could cause big issues. One of the thing I wanted to get your perspective on and we talked a little bit about it is building teams, engineering teams, and other teams. Obviously, you’ve been a part at Paycor of a number of different teams and see the organization grow. There’s obviously, all the tech skills required to be a good engineer, but it’s not like soft skills that are important too. Just curious about your perspective and obviously, we’ve been talking a lot about tech, but that’s not the only part of the equation. I don’t think. Adam Koch: Yeah, well, we’ve gone through the journey of establishing an architecture team, establishing an architectural practice? What is EA versus what is application architecture versus what is dated architecture? I believe that there was a certain assumption that as soon as you wrote it down, it was gospel, it’ll just happen. Hey, this is the naming standard. This is a scaling standard. Everything should be done with version API’s or whatever. Really, there’s a few things wrong with that presumption. One, you have to establish the patterns and standards, and then luckily, there’s enough de facto things out there that you can leverage off of those. You have to write them down and there’s a million tools out there for progressing it out there. But one of the things that again, even if you have a wiki, even if you have a good EA tool or things like that is the hearts and minds. Why are we doing this? If you’ve ever been to someplace where they start agile for the first time, you’ve been somewhere where they do unit testing for the first time. It’s amazing, you go to a conference and you’ll see this thing. You can wave like, “Oh, my God. We’re going to do this.” People share the last slide. They share this is the mountain top or whatever, and you sat through an hour and a half about why you’re doing it. You’re trying to convince everyone else with the final slide of this is the promised land. I think if you take teams through it, we talked about the delivery of applications through the CI/CD platform, trying to tell people like, “Hey, test automation is important,” not for correctness. Of course, you want to be correct. Test automation is important for reducing your cycle time to get things out to production. I think people started clicking why we were doing that, because a manual tester can find the bugs over the course of three or four days probably better than an automation can. But what the automation can do is it can find 99% of the problems and get you out in a couple hours. Then yes, we will have 0.1% release defects, whatever. But we have a fast rollback strategy. We have the whole blue-green circle deployments and things like that. That helps your business agility, independent of your tech stack and things like that. What I found is two things. One, having the culture of getting that information out to people, what we got to do, and why we have to do it, but then also making sure that you translate that up to the non-technical folks, because it can seem like… I don’t know how many times that I’ve been stopped and say, “Hey, you guys are doing insert technology going into cloud.” It’s just that simple bread and butter things, just because it’s cool. We just created a Silverlight UI. We just created an angular or knockback JS, whatever. Now you’re saying we want to go to react? It’s just because it’s cool, isn’t it? Well, no, we think that the reusability is there, that the training, the community, the Stack Overflow answers are there. There’s a million reasons that we’re doing what we’re doing. Apparently, I didn’t convey it to you well enough. So, for me, the soft skills, it is showing your work. Why are we doing what we’re doing? I think one of the things that’s completely fair is people say, “Well, can you measure it? Can you do this better thing and show it?” Some things are clear like our deployment pipeline. It went from months down to weeks down to hour down to minutes or whatever. So, clear, clear improvement. When it comes to some of the more application dev cycle stuff, it’s a little bit harder, because every piece of work is different. A deployment is a deployment. It’s bits to a server. When it comes to building three different tools for different areas of the payroll, HR domain, a simple dashboard versus responding to a new law or just very different things. So, yeah, this is 10% faster, is it? But I do respect both of those things. One, why are we doing this? Then two, prove to me that it’s better because I think that one of the reasons I’m in technology is that things are usually provably better, like this iPhone smashed and this iPhone. 4G is faster than 3G. I like someone calling me out and say, “Show me that this is better,” because usually, there’s a way to do it. Tim Veil: Yeah, I think it’s really fascinating. I think one of the things that I’ve observed in working with various companies and you really touched on it is change is hard. If you’re not communicating effectively, it can be enormously difficult. I think this is very true in technology. Especially again, on the vendor side, we’re out there selling a database technology, which is new and different and sometimes scary or at least I think people think it’s so. The hardest part about, for example, adopting a new database oftentimes isn’t actually the data migration, the physical work you have to do. It’s just winning the hearts and minds of people. It’s getting people accustomed to doing something different and adopting change. I think, again, that’s probably something you’ve seen time and time again play out over the years and I’m sure there are at Paycor as there have been everywhere. There’s somebody in the corner that says, “That’s not how you’re supposed to do it here.” Adam Koch: Why do we need JSON? XML is fine. Why would you need YAML? I don’t need that new-fangled- Tim Veil: Yeah. Adam Koch: I think the data platform is a good place, because honestly, applications can come and go. You could completely redo your mobile app in three stacks. The data is probably still the same data. Honestly, you go from self-managed instance to a vendor-managed instance to serverless instance, whatever. We talked about on the other conversation cloud scale, data stuff, or whatever. Honestly, is your business ready for no SQL? Obviously, that’s nebulous. Are you ready for document storage? Are you ready for changing how your data is not all in one place anymore? As you go through iterations of vertical scale, there’s a certain way you can design your system. As soon as you horizontally scale, all of a sudden, you’re talking about Snowflake and things like that. How do you bring everything together at the last moment while making sure that you have true horizontal scale? I think there’s some exciting things happening with… Again, I said before, I want vendors just solve some core fundamental problem with our things, because engineers, I think, largely want to turn on something and use it. Blob storage has been around for a decade, but it still, obviously, blows some people’s minds that you have terabytes, petabytes of storage space available to you, when that used to be a big thing. Oh, our fiscal year is about to end. Can you wait three more months to get more disk? Business agility around those lead times runways is just crazy. I always will defend the cloud in the two major ways. You want something, you acquired them today. Are usually going to wait three, six months to onboard them when we have the stuff that attach them to? Then the resiliency, right now, data centers have to have two data centers. You have to have two redundant things existing. Maybe you have active, but generally, you just have to wreck things twice. With the cloud, you could have 9 to 10, 50/50. You have what you need in both places, and the world effectively is your backup strategy or whatever. Tim Veil: I mean, that touches nerve with the Cockroach folks. I mean, that’s like data everywhere, never go down, data survive architecture, databases survive everything. It’s important. One other thing I wanted to touch on with you, because again, we talked about it briefly before, but I think it’s a nice segue. That is the relationship between an engineering and product teams. I know it’s not necessarily a big idea in app architecture. But I mean, the reality is, those two teams functioning well is a super, super important thing, not only you’re building a SaaS business like Cockroach is, but even at Paycor, where you’ve been doing things a long time. I mean, can you talk a little bit about how that functions at Paycor and what you’ve seen work well, maybe not so well, how it’s changed over time, all those kinds of things? Adam Koch: Yeah, I mean, one of the biggest things is obviously, like any company that sells to customers B2C is we have customer facing product, people who understand. The market understands the business problems. You are constantly evolving, what we’re delivering to our customers. In one of the epiphanies, but one of the big realizations was we have to have internal products. Sufficiently large companies, authentication, storage, messaging, and things like that, I want to write an app that can send an email and SMS, can raise a toast message in an application. I don’t want to write code for that. I just want to hook into the page where we have doing that. I think that’s one of the biggest things we’ve done it on the macro level is we have product owners over those things. Their customers are the internal engineering teams. They understand what their needs are. Will a simple MBI work? Does it have to be a robust offering in order to really service them? Those sorts of things and then the next thing is that’s abstract macro stuff. We talked about patterns and things like that. Okay, you want to go from a single connection string to either a magic data offering in the cloud or you want to go to something that’s more managed charting or something like that. Okay, well, that sounds like work that’s going to take away time from our customer, deliver features, whatever. There’s this new way that we want to be able to support unions in California. It’s a new thing that we want to do that supports specifically healthcare or manufacturing, whatever. We’re not going to be able to do that, because we’re working on your stuff. I think you have to explain to people what we’re doing. One thing is, is that Paycor has a very good problem. We’re a very high growth company. The problem is, is that when you see things going up into the right, we have to have a system that can support that all the way down. Because we talk about the weakest link and things like that. If your authentication doesn’t work, if the file storage underneath something doesn’t work, if the menu doesn’t work, people can’t see which products they have and navigate the solution. So, making sure that the engineers can explain to their product owners what we’re doing, why we’re doing it, why this investment is ready now. I use example sometime of going from a flip phone to a smartphone, you didn’t really have to sell people on it. Once they saw, they knew it. Sometimes things aren’t as easy as that. Sometimes it’s like your furnace is on its last leg, is it? It’s still blowing hot air. I don’t quite know what you mean by it’s almost dead. It sounds like it’s still working. So, yeah. So, the strongest teams at Paycor that I’ve seen have built trust effectively. Again, going back to the sticking to the wall and it’s the rule, 20% of your time needs to be spent on just clearly stuff and then for feedback as well. If it’s always 20%, what if I need lessons and it feels like I’m wasting. I’m leaving your time on the table. Again, that does not work. What you can say is, is that if something’s going to affect your customer now or impact your customer in 90 days, whatever, and it’s going to take, say, 60 days to fix it, we shouldn’t wait until it’s broken, until it’s slower than we want it to be until you address it and things like that. Again, the best teams are the ones who have built that trust. The other thing, I hear terms like gold plating. Oh, you’re going to build it, but you’re going to build it so cool and using the latest technologies and whatever. Well, no, if I bring a new data platform in, it is to solve a problem. If I bring a new type of compute, because over the years, we’ve changed, for example, in Azure, the type of hosts that we’ve targeted and each one brought something to the table. It wasn’t like, “Oh, this is cool.” It was more like, “Hey, do you realize that this auto scales? Do you realize that this takes away the need of having to have a VPN or just something about the prior solution that we can extend?” Tim Veil: It’s funny, this term gold plating, I’ve not heard. Is that analogous to putting lipstick on a pig? Adam Koch: Yeah, I mean, I’m sure that there’s different phrases that have been used, but ultimately, the idea is imagine if you gave an engineer unlimited time. They would add every library, every automation, every extensibility thing, whatever. Ultimately, this is nothing that we invented. It’s like get something to production as quickly as possible, have a few beta users tell it. They’re going to tell you pretty quickly, if it’s landing the mark, whatever. We did a lean startup methodology while back, if you know the book. The intention there is, again, you’re a little a lot more by having software customers' hands than by doing things. The other thing we know about your reusable modules and patterns and things like that is like if we can get at 60% right before you even touch the code, to have authentication and all these things baked in and really all you have to focus on is business logic. So, again, when we’re arguing or negotiating with somebody on what we should spend our time on, it really comes down to this is the most important thing. Again, when they see their application respond faster, when they see their uptime at a nine, whatever, that I think is measurable. Tim Veil: You mentioned a book that you liked. Are there any other industry books that you have read or you or your team’s read? I mean, some things out there that you find interesting and relevant. Adam Koch: The main thing I like about the books is that sometimes it injects language. Obviously, there’s the domain driven books and things like that. There’s some leadership books like Who Moved My Cheese and whatever, steering the ship or change the course, whatever. I personally am more of a wiki, podcast person. I think that there’s some classic things out there. Again, we want to get the vernacular out there. We want to be able to talk about things and we can think about where agile came from. It’s like hey, there’s these sticky notes and we have strong and steady, poker and all that stuff. Yeah, like Jeff Atwood, back before he blew up in the Stack Overflow game, he had a blog called Coding Horror. It was great. It’s just like little nuggets of wisdom and Scott Hanselman, things like that, where it’s real life things hitting you. Tim Veil: Is Coding Horror still out there? Because Jeff Atwood stuff was amazing, man. He has had Stack Overflow, isn’t it? Adam Koch: Yeah, he was one of the founders, I think. I think again, you invent the pet rock, whatever type thing. It was the right thing at the right time. Tim Veil: I remember the Coding Horror logo, which was from a book or something. I can’t remember the details. Yeah, that was a great place. Great resource. Adam Koch: Yeah. I mean, one of my most cited things from him was reusability. Going back to the whole gold plating, are you going too much? Is it mainly a viable thing that’s better off? He talked about reusability, requiring three instances. You make something. You’re like, “Oh, this is the new standard, everyone should use it.” So you actually build a lot of stuff in there, design around usability, but honestly, unless you go out and you promote this thing that you did, no one’s going to reuse it because they don’t even know about it. So, then you say, “Okay, I have two. I’m going to effectively partner with somebody and we’re both using it.” Okay, but you’re both on the payroll team. You’re both on the HR team. The other domains or subdomains have completely different needs as far as data access or just something that’s not going to work. So, again, you meet through usable that’s fully reusable. Yeah, he hashed it out and I’ve used that many times to say, “You come to me and you show me two other teams that you’ve reached out to who are willing to adopt this in the first three months, then I’m going to believe it’s worth reusing. If not, you’re a smart person, but let’s build something for what you need and focus on reusability at a different level.” Tim Veil: That’s so true. Well, listen, I’ve already probably taken too much of your time. But before I let you go, maybe let’s wind down, like we have with many other episodes. Just looking forward to some things, whether it’s at Paycor or personally, whatever it is, that you’re looking forward to over the next 6, 12, 18 months. Adam Koch: Yeah, I mean, personally, I have a son and he’s about to go into kindergarten. That’s really exciting. It’s effectively, that next phase of life, so, that’s really amazing to see. Professionally, technology wise, I think a lot of movements have been off of things becoming ubiquitous. There was the idea of like the smartphone, ones that went out of smartphone, that enable things rideshare to exist, because everyone was able to do GPS and map and mobile pay. There’s things like cloud computing, whatever. I think the ubiquity of AI processing, the OpenAI stuff is going to be amazing to see, effectively, what that allows. Us like anyone, we’re trying to figure out, because you can do a lot of technology demos that look really cool. I was actually in a conference recently. The presenter was being a little bit frustrated. He kept showing all these things that the different frameworks like DALL·E could do and no one was impressed. He’s like, “Why aren’t you impressed?” It’s hard to be impressed anymore by what ChatGPT can do. Okay, so the luster goes off of it can just do a bunch of cool stuff. What problems are going to go away? Again, the rideshare thing and things like that, it made something possible that wasn’t possible anymore. So, I don’t know. There’s some, again, pet projects that we’re working on and things like that. But what is maybe that watershed, oh, my goodness event or whatever? I do laugh at the whole well, it’s going to write all the code for you. You guys haven’t used it. It definitely will write a block of code for you. It will definitely, stub something out for you, you probably want to ship in production, but it is definitely there between that and GitHub Copilot, things like that. Amazing. But again, what I’m really interested to see is, what are they going to be these society level changes that are going to come out of this that are more than, “Oh, look, I put your face on a picture”? Tim Veil: Yeah. I agree with you. It’s been a fascinating six months of technology, hasn’t it? I mean, the amount of time we’ve been spending talking about large language models and generative AI and all this stuff is just hockey stick recently, but it is. You’re right. I mean, there are so many neat possibilities. I don’t know that the matter has been settled or it’s all been figured out, but is certainly going to transform our industry, our work, et cetera soon. Well, Adam, it has been great chatting with you. I think it’s one of those things we could probably talk for quite a bit longer, but I want to be respectful of your time. So, again, thank you so much for joining Big Ideas in App Architecture. Hopefully, we’ll have an opportunity to speak again soon. Thank you. Adam Koch: It was really fun. Thank you. Tim Veil: Thanks for listening to the Big Ideas in App Architecture Podcast. As always, please rate and review the podcast five stars on your favorite platform. If you like what you’ve heard, you can watch every episode on our YouTube channel linked in the description below. Until next time. Thanks, bye.