David Joy:

Welcome to the podcast. When you were coming, I was seeing if I could find a Tears of the Kingdom t-shirt. I couldn’t find that. Interestingly, I was talking to a friend of mine who just said to me 15 minutes before I got on the podcast that he completed 120 shrines on the ground. He is kicking it off, right? For context, for folks who are listening, me and Sid, when we first met, I looked at Sid and I’m like, “This looks like a gamer,” right? He’s like, “Yes, I am.” He’s a huge fan of the Zelda series, apart from being an engineering manager at Route. Welcome to the podcast, Sid.

Siddhartha Sandhu:

Thank you, thank you, David. I’m happy to be here.

David Joy:

Yeah, and I think we want to stick to what this podcast is about, which is big ideas in app architecture. We’ll stick to talking about what do you do at Route as part of the conversation, but I do want to talk to you about where you are in Tears of the Kingdom. 

Siddhartha Sandhu:

Without spoiling any fun, I’m around, I think, 135 shrines now. Yes, yes, and I’ve collected most of the armor sets that the game has to offer, which is kind of fun. I missed out on that in Breath of the Wild, but this time I’m going to …

David Joy:

How many are there in total? Because somebody told me you have 36 in the sky and about 120 on the ground, and then you have a bunch of them underground as well.

Siddhartha Sandhu:

Yes. Yes. I believe there are 152 shrines in total, so quite a few shrines.

David Joy:

Oh, wow, so you’re pretty close to finishing the game up.

Siddhartha Sandhu:

Yeah. I’m kind of taking it slow because I want to do justice to the developers who made the game and made it look so beautiful, so I want to explore everything before I go into the arc which actually closes the story for the game. Yeah, yeah.

David Joy:

Oh, mate, you are way ahead of me, because I am, right now, at probably 20 shrines, 25 shrines. That’s where I am, right? I’m way behind, and I think I’m also going to take my sweet time to complete them. Again, it’s always fun to have people who are Zelda fans on the podcast, so you’re actually one of the first ones I’m talking to. I just realized I should have asked other people about that too, but anyways. It’s a weird segue. Let’s jump into the podcast and kind of talk to you about what you do at Route. I know your role at Route is engineering manager, and you lead a very important team at Route. As we begin, for folks who are listening, why don’t you help us understand? Kind of level-set, what does Route do, and what it brings out to the industry, why is it important?

Siddhartha Sandhu:

Route as a company is focused on the post-purchasing journey of the user. What that means it that once you actually check out a product, after that you have a seamless experience. I’ll go into the various product offerings that we have. We have protection, which is … You can buy kind of an insurance on your package, so if it gets lost in transit or you have post thieves or something, you don’t need to worry about it. Then, the second piece of the product is tracking. That’s while the package is out, you want to track it. It’s kind of like an Amazon experience but for the merchants as well, and that’s the second product. Then the third product that we have is resolution. If your package does get lost, how do you go about kind of getting in touch with the consumer? This is so that you can get that situation solved. The fourth part is engage, which is a newer product that we’ve got coming out. What it does is, during your post-purchase, that’s while you check out, it recommends products to you that would either be complimentary or similar to things that you already have in the cart and would make sense to you, and you can engage with those products.

We extend that product by including it into your email campaigns or your notification campaigns, which allows the user, the consumer of the product, options on what else is available in the market, which might align with my interests. That’s Route, as such, yeah. Yeah.

David Joy:

That’s awesome, so is it, the business is more B2B, or it’s B2C?

Siddhartha Sandhu:

That’s a good question. I would say half of the business is B2B and the other half is B2C. The part where you purchase protection, that’s kind of part of the B2B chain, as well as the engage product. Then we have tracking, which is a B2C product. We have an app called Route. Do check it out. You can go into it and track. All your packages would be at one place, and you’d be able to get tracking information on it rather than you having to go check your email, go into the actual orders, carrier’s page, and checking where it is in the journey. You can just go to the app, and that’s got everything consolidated, and check where your package is on that journey and just move on, call it a day.

David Joy:

Yeah, which is what I love to do. I’m restless when I’ve ordered something and it hasn’t come home, right? For example, Amazon app has the tracking information. Then, as soon as I see 10 stops away, I’m excited. Like, “Oh, yeah.” I’m getting ready. Do you guys integrate with Amazon as well as other services like Shop, that provide you to purchase things but then also you can do the route tracking for them?

Siddhartha Sandhu:

Yes, we do. Our main focus is Shopify, and the merchant platforms. There’s quite a few. Shopify, BigCommerce, Magento. Then we’ve got Salesforce Cloud and a number of platforms where you can go and set up your merchant store, and we have app integration onto that, so you basically add the app into your store, and at checkout it will give you an option like, “Hey, do you want to purchase the Route insurance?” At that point you can purchase your Route insurance and rest assured that Route will handle the rest for you if something happens with the package. Yeah.

David Joy:

Got it. Oh, that’s awesome. I mean, especially for small businesses that are running on Shopify, providing some sort of an insurance opportunity for them to protect whatever they’re sending is I think a pretty interesting value proposition. Is that one of the biggest selling points that you have, when you go after users and customers about what Route offers? What are the three things that you kind of tell them, like, “Hey, use Route because we do these three things”?

Siddhartha Sandhu:

Usually, the user as such, the consumer as such, is more interested in the tracking capability, I would say. The protect offering, as such, just ties into it. If you have the Protect offering on it, then we take care of the resolution if your package gets lost, but generally, anybody who installs the Route app does get the track offering as it is. Any user, you don’t have to explicitly purchase Route insurance, and they go through that flow and we provide the tracking for you. Yeah.

David Joy:

Got it. Yeah, and I’m just curious. Maybe I’m getting a little bit ahead of myself, but when you were sharing, I was thinking. Do you also have to start thinking about … When you have this Route stuff that you’re sending to different people, packages going, do you have to think about optimizing which route the, I would say, the companies, like FedEx or others who are delivering it have to take? Do you engage with them as well in that capacity?

Siddhartha Sandhu:

Yes. We have carrier integrations which engage with them. Generally, we use some of our own carrier integrations and third party aggregators to go provide the tracking data to our users. Yeah.

David Joy:

Got it. Well, that’s brilliant. I mean, there’s a lot of fun things that you’re cooking. Let’s dive into a little bit more about what you do at Route, right? One thing that’s clear is that you are in a business that’s focused on B2B and B2C, and then you enable this experience that you just shared, but what does your role entail? Expand a little bit about what you do and the team that you take care of and different functions within it, for everyone who’s listening, say.

Siddhartha Sandhu:

Perfect. My team, I would say, is literally the big back end. If there was a back end as deep as the Mariana Trench, it would probably be that. We have the team sitting over in the orders infrastructure. That’s basically ingesting our packages, and post that, once the package is there, what kind of notifications we want to send. That’s also owned by us. In addition to integrating directly with platforms, we also allow the users to allow us to hook up emails directly to us, and we parse their emails to figure out, which packages are they tracking in the app? That parsing capability, as well as connection capability, also sits with my team. The fourth thing that we have, that’s the engage product. That’s the product recommendations that we make during the post-purchase checkout or post-cart email campaigns or notifications. That also sits with my team, so literally any product that Route has somehow interfaces with the team’s general set of responsibilities. You could say that I’m pretty hammered by requests on what’s important at the time. Yeah, yeah, yeah.

David Joy:

Got it. Yeah, so you and your team set the deep architecture, but you kind of are serving a very, very critical function for the company, right?

Siddhartha Sandhu:

The company, for us, this team has such … It is critical in its capacity to run most of the business operations that will reach out to the consumer, and store in the knowledge that’s coming in from the B2B side and make that connection. We are kind of, I guess, like that routing box which would get the information from the business and to the consumer, as well, and of course the information is also used to kind of give good insights to the business as well, on how-

David Joy:

It seems like you’re looking at data as well. Expand for me the different teams that you have. Do you have a database team? Do you have an infrastructure team? Developer team? How does that work, I mean, for you?

Siddhartha Sandhu:

I’ll expand on, yeah, that. The idea is that right now, how our teams are structured, most of the teams operate on an operation model that we are end-to-end, so that means we generally take things from development to deployment, and I would say a bit of QA, also, after that. We tend to keep our deployment pipeline such that at least the obvious things get checked. Now, how our teams as such in Route are structured, just to get something deployed, there’s a lot of things involved, so we do have a person who’s in the … I would say a development enablement, slash, sorry, slash platform engineered, who are empowering us to let us focus on actual the business use cases. They provide the bare bones of pipelines as well as infrastructure that is required to actually make us operation into the cloud. Yeah, yeah, so that person is generally not sitting with our team. They’re its own team, and they empower us to make decisions independently so that we can move pretty quickly. Yeah.

David Joy:

Got it. Yeah, so you have a set of folks who are saying, okay, they’ll provide you what you need. They’re basically bringing you the world. They’re bringing you everything that you need. I mean, going back this Zelda reference, they provide all the things, but you essentially put it all together based on what your business requirements are.

Siddhartha Sandhu:

An example would be, for example, GitLab pipelines. My team does not have to invest too much time into understanding GitLab pipelines and bringing that up. We do have a bare-bones structure in place, which, it is plugged into the architecture, which is minimal, but it gets you there very quickly. Then, after that, if you want to append stuff to it, that’s kind of the team’s responsibility, as such. For example, if you want to build in integration tests or functional tests, as such, so those kind of things are built in by the team themselves, because each team has a different way and style of going about their testing requirement. Route gives you the option to kind of work against the things that you’re comfortable and the choices that best reflect the ecosystem of the software that you’re working with.

David Joy:

Right, yeah. This new exciting product that you’re working on, the engage, it’s basically a totally different product, different from the Route app, or is it part of the Route app itself?

Siddhartha Sandhu:

The Route app itself gets the feedback from the engage product. If you’ve got packages which have recommendations against them, those are usually the merchants that have integrated with the Route ecosystem or have installed the Route app. They would carry recommendations, if they’re on the appropriate plan.

David Joy:

Got it, yeah. I’m intrigued, always. Whenever you start companies, right, typically you have this one idea or you feel like, “Well, I want to build something, and it should look like this.” The first year of that company, first few years, they’re focused on building that one service, right, but over a period they realize, “Well, there are other things that we can do with it.” Let’s say, take the classic example of Amazon, which started off as selling books. Then they expanded to do other things, and now they are selling Prime, which is video, and then they’re saying, “Okay, we’ll sell you now a product,” and so it’s a classic example where you start with an idea and then you look at the market and you realize that, “Oh, there is a need for something else that we can actually bring.” This engage kind of feels like is an extension of how Route has expanded and the use cases that it’s providing to the users.

Tell me a little bit about, if you were involved in it, as to what goes behind the scenes as to thinking about, “Oh, well, we feel like we need something like engage for our users.”

Siddhartha Sandhu:

Generally, the product itself was, I’d have to say, a brainchild of our product, as such, and we’ve got an amazing team looking at what merchants are wanting. We have an ecosystem right there. Account executives, and there are CSMs, and unlike engineers, they’re interacting with the merchants and prospective merchants on a daily basis. They understand that, “Hey, these are the things that merchants are interested in, and these are the things that we want to develop to keep the merchants who are with us interested in this platform itself.” That feedback is kind of funneled into our product team, and that product team kind of looks at that feedback, understands what merchants are kind of moving towards, without hyperbolically discounting any future possibilities that the company wants to go explore. Then, that feedback is kind of put on the plate of the engineer to develop features that would at least address feedback that would help retain our merchants, which is amazing, because we have 98.1% retention at this point. We also prepare for any enterprise merchants or big-ticket merchants that want to come into the Route ecosystem over the next six months, and if they requested for a specific feature which is specific to their ecosystem.

That’s how we, on the engineering side, can get that feedback. We funnel it through, and then after that, kind of pull it together data with the data analyst team and the product team to kind of verify and justify the product capability as such. Yeah.

David Joy:

Right. What you’re essentially saying is that there is no bad idea that exists out there, until you … Yeah, right? Which is fascinating, because I feel like folks don’t realize that. Especially, one of the things, what you said, I really like, is that you’re getting active feedback from the customer success team and people who are talking to these users, and merchants who are on the field. That active feedback comes back into the company’s tracker, and then from there, you understand product requirements and capabilities. There essentially is no bad idea. There can be a merchant who might give you … One person might feel it’s the silliest idea, but then you consider that. You’re talking about that, so you’ve got a feedback. From that feedback, I wanted to understand, you’ve gone through this process of talking to the data team and understanding if there is some substance to this. Okay, there is a viable product or a service or something that you can enhance in the product. How does your team get engaged after that? As an engineering manager, how do you get involved in the strategy for that, and kind of laying out what needs to be built? Expand on that a little bit.

Siddhartha Sandhu:

That’s a good question. What we’ve done is, the feedback, once it’s funneled in, is not only for a certain set of eyes. It’s wide transferred, the mechanism. We work with Tableau dashboards. We’ll go and check that out, and there are multiple Google sheets that are running around, yeah, that I keep bookmarked to make sure that there’s no surprises as far as product features are concerned, so it’s not a complete pivot. If we are going into a product feature which would … If you think about it, engage itself is kind of like a pivot from our traditional offering, which is protect, track, and resolve. It kind of fits into the build, really, but when engage kind of came into place, because platforms like Shopify introduced extensibility that allows you to plug in things during their checkout flow. Yeah, yeah. These sheets are there, and generally we end up reviewing these on a weekly basis, at least, to make sure that at least the things coming into the pipeline are the things that we already have secured or now are kind of getting the value add that we wanted, and we promised them, against Route.

It’s quite a transferring process, and like I said, our product team kind of is amazing. They kind of keep us on our toes as well, on the SDMs as well, and they make sure that if they are planning on going a certain route, the SDM is generally engaged at the get-go to talk through the challenges that we could face before we actually hit the engineering team and introduce the idea to them, and go into a planning phase of how we want to execute against this in a timely manner. Yeah.

David Joy:

Amazing. No, that’s great to know. Were you always … I’m just curious. Were you always passionate about this line of work and doing or building applications and helping built applications for companies and users? Let’s dive into a little bit … Maybe this is the point where we go back a little bit into Sid’s story as to what led you to do what you’re saying that you’re doing right now.

Siddhartha Sandhu:

Yeah. I remember when … I bought my first computer when I was in seventh grade in India, and it was back in, I think, 2001 or something. I was so, so surprised with this new thing. We had the TV, and now I can do things a bit. For me, as a child, it was quite interesting. Growing up, I went into computer science because of that, because it kind of felt like a good solution in a space where I was comfortable, and yeah. Yeah, that kind of led my computer science background. Then I got into companies, and I am inherently more interesting in consumer facing products right now, rather than much more deeper. Though, with recent changes, that can change. Yeah, it’s incredible when you see and develop something and you put it out there, and then users interact with it, and they’re like, “Hey, I was looking for something of the sort to be in the market, and this app does a really good job of it.” That kind of really motivates me to kind of build apps that actually drive value towards the consumer, and that’s kind of the motivation behind why I like to do a lot of apps and build things that people actually use. Yeah.

David Joy:

Yeah, it’s amazing, because I mean, I see a similarity. I have another friend, actually, who used to work at FedEx. I mean, every time we would try to connect with them, he had such a busy schedule, for almost a year, and I would meet him and it’s like, “Where were you?” He’s like, “I’m working on something really important,” all of that, so I’m like, “Okay, no problem.” Then finally, we meet him after a year and he is really excited about something. He’s like, “We finished this project.” I’m like, “Great.” Then, we’re driving together and he sees the FedEx office. He used to work with FedEx, and he’s like, “Well, you see this new FedEx office? They have a new system right now. If you go and kind of use it, that system is what we were working on.” I was like, “Dude, that’s fascinating.” There is this joy that I’ve seen when you complete a project, but there is more joy when you see your product being used by other people. He literally was like, “Let’s go inside and go check it out, check out the system.”

I’m like … We really went in and we saw the system that he was working on, and the funny thing is, while they were doing … While we were interacting with the system, he found a bug in it, and then he called his team, like, “Oh, seems like thing is not coming out,” some receipt or something like that. It was really interesting to see that, so to your point, right, when you build applications, there is this joy that you get, that you’re talking about, when you build applications that folks use, and that kind of makes a difference, which is really exciting.

Siddhartha Sandhu:

Yeah, that reminds me of my own anecdotes. While I’m using the Route app myself, if something goes wrong with my parsing of emails or something, then I raise a bug from my own form. “Hey, guys, you need to go fix it,” and then kind of transfer it into our queue, without having it to kind of go through the whole route of reproducing the bug, because as such, it’s right in front of you. There’s nothing that you can … There’s no need to reproduce it. Yeah, yeah.

David Joy:

Yeah. Yeah. I think that’s the best way to improve your own product, right? The use it as much as possible. I mean, I think there’s this whole story of Steve Jobs, when the first iPhone was coming out. I think they made a prototype, and then he kept the iPhone in his pocket and had the car keys, and would drive around and use the phone for a certain amount of time, and saw scratches on it, and felt like they needed a different kind of glass that didn’t have scratches on it. I think that led to the new form of the phone glass that you keep on top of the phone, so that led to the development of that. I feel like what you’re talking about is, in a sense, what Steve Jobs did for the iPhone, which is pretty interesting.

Siddhartha Sandhu:

Yeah, generally when we come to the table, at least through my own experience, I’ve seen that the devil is in the detail, and these small things which often get ignored actually make for what builds a great consumer experience. Yeah, yeah, so sometimes you’re more tempted to go after the bigger fish, but something it’s like, “Hey, this idea was so easy to implement and it grows so much value for the consumer.” Those ones, I wait for those ideas to pop up, and when we see the KPIs against it, it just blows your mind. Like, “Oh, wow, this was amazing.”

David Joy:

Yeah. I think great customer experience being built into the app is so critical to the overall experience of what the company is trying to drive, right? I mean, many people feel like customer experience is driven by a UI experience, right? I mean, of course it is true that a UI experience is important, but the way it performs, the way the app is always available, how it doesn’t go down, how it’s able to scale, I think those are so much more important variables that many people who are developing are not thinking about, but you in engineering positions are always thinking about. “Hey, we need to consider all these important functions that are truly important for a good customer experience.” Right? From that, I wanted to segue into this question that I was meaning to ask you, is, these things are important, so how do you consider that into the architecture, and how do you think about the engineering decisions that you have to take when you’re choosing that tech stack for that experience?

Siddhartha Sandhu:

That’s a heavy question. Yes. Choice of tech stack-

David Joy:

Yeah, it’s a broad question. We can break it down and go into that. Yeah.

Siddhartha Sandhu:

In Route, we have quite the consistent tech stack. We generally prefer to write back end sources in Golang. It’s quite an opinionated language in itself, which keeps the opinion of the developers on the lower side while you’re reviewing code, so it minimizes the amount of ways that you should be able to do stuff, compared to languages like JavaScript and Java, in which you can do one thing 1000 ways. Opinions start to clash in those languages. Golang kind of reduces that for us. Now, in terms of our infrastructure, we generally choose … The first choice is going with AWS. If AWS does not have the thing that we require, then we tend to be … We look for third party providers, after. That’s always one of those, will be for convenience. Databricks would be one of those services that we’ve chosen, as well as CockroachDB, which is kind of a high availability database that you all provide. Those are the major tech stack components that we have, so we default to AWS, and then we have CockroachDB and Databricks for our data jobs.

As such, choice of new infrastructure is decided through a process of review, which is quite simple, I would say, in a team the size of Route. Generally each team has a very senior engineer. In my team, I have a consumer engineer. Before we make a choice in moving along with a certain piece of infrastructure, for example … An example being, “Hey, we want to use Kafka instead of Kinesis,” or, “I want to use Kinesis instead of SNS.” Then we ask basic questions. At least that decision and that questioning is localized into the team, that, are we going to be seeing packets come in with great 256 KB? Do we need this? What is the additional functionality that we’d have to build to operationalize something like Kinesis? Just an example of that would be, “Hey, on SNS, you can go click in the AWS console to redirect stuff. Does Kinesis provide the same kind of functionality? If it doesn’t how do we want to operationalize re-running?” Our cues, our topics, are, how do we want to operationalize re-running that payload on Kafka?

Once those questions are localized, as such, we put together a proposal. We have, at the highest level, a technical architect, and he generally looks through the proposal to make sure that everything is sound. Yeah, yeah, and people are not getting too ambitious with their designs.

David Joy:

I mean, it’s important. I mean, in my capacity at CockroachDB or Cockroach Labs, I work primarily with AWS, and I’ve really enjoyed the AWS platform, and just the multifaceted solutions that they have, but not every solution really works for me and my requirements. I think I’ve noticed, in my conversations with other folks, even you, that sometimes not everything that they offer works as you expected, and then you go and pick something else. The driver for picker something else is usually a requirement that you’re trying to build out, right? A high availability system, or you need multi-region capability, or you’re like, “I want all this on.” You’re like, “Well, I want to scale,” so you’re like, “Well, let’s use EKS, and start … " When we were talking last time, you were talking about something that you were trying to explore with EKS. Tell me a little bit more about that driver, as to what let to you thinking that, “We need to move to something like this,” and then, why you were thinking about EKS.

Siddhartha Sandhu:

At the moment, we use ECS, which is a great control plane. I have no complaints against it. It’s quite easy to use, and we use, as such, target profiles in the back, so we don’t host it on EC2 servers, so our computer is pretty optimized when it comes to costing. There are a few things that the EKS domain kind of provide, and that’s kind of the trend of the industry, that’s shifting left into security and other stuff. That gives you a more fine-grained control over your traffic, how you do the metrics, or even how you do your logs. That’s something that has at least sparked the debate in our company on looking at more simple solutions towards that. AWS itself is keeping up with that ecosystem by providing their own service discovery mechanisms, because one of the things that I feel folks don’t realize is that you’re not really investing in EKS from the standpoint that you stand up EKS and that’s that. You’re actually investing in the tooling that the Kubernetes infrastructure as such provides, which is separate from the control plan itself. While we invest in this, we have to be cautious about which tooling we want to go about using in that ecosystem, so as not to burden the team with exponential learning curves, which would essentially slow down our product growth.

Being specific about what things we want and what would drive the most value for us. Something like, yeah, service discovery kind of drives automatic value for out of the box, or even SDO. SDO itself is service discovery. Yeah, service mesh. That’s amazing. You just stand it up and all your metrics, all your traffic gets routes through the Envoy L3/L4 proxy, and that gives you metrics even for the when the call hits your application. It removes kind of work off the plate of your own team. My own team, or anybody’s team won’t have to be directly involved in putting in specific sidecars in the ECS mechanism to go and scrape and get Prometheus metrics, or a StatsD sidecar to push those metrics. That’s kind of controlled by the control plane itself, and you can upload something of that sort through a … I believe it’s called a daemon thread, or something in EKS which appends that sidecar onto your already deployed application and makes it seamless on that routing mechanism. Yeah, yeah.

David Joy:

Yeah, that’s interesting. I mean, so you have to think about the efficiency that a new system brings to what your existing tool is, versus the effort your team has to put into it, right? How do you go about making that decision, right? Do you also have a process? As an engineering manager, how do you research and kind of define that, “Okay, this is something really important and we need to build a case around why we need to use it”? How do you do that?

Siddhartha Sandhu:

Generally, what I do is I take a dig at it myself first to see how bad it is. You can say it’s … You’re going to ask somebody to do this in the future, right, or at least probably something of the sort. Having an understanding of the ecosystem kind of helps talk through it, so that’s how I start. I take a dig at it myself. I see how … Up to now, this is my first time being an engineering manager. Up to now, I’ve been an engineer, so my problem-solving training book is still the same. Just learn the things that are important to you at this moment, and see how quickly I can go from a zero to at least 95. Then, I say, “Hey, I was able to stand up on EKS faster, while an infrastructure as code mechanism that’s like Terraform or something, through my pipelines, in a three days framework,” or something. Based on the experience that I’ve had in the industry and how quickly I’ve been able to stand up this, how would my own team fare against getting apps deployed into the system? Can I do something so that they’re not involved at all and this process is seamless?

I tend to kind of take some time and answer those questions, and go through those internally with myself, and then I can talk to somebody who already has experience on those and see how their experience has gone. AWS does, not only AWS. I think most of the companies who’ve built architecture on top of control planes, like ECS or EKS or the one, I think Google’s is GKS or something, and … GKE, yeah, and they’ve written different blog posts which kind of guide you in the right direction, so you’re not the first one who is going into this problem. The ecosystem is quite established, and it might be worthwhile to gather that knowledge before you actually go ahead and funnel that change. Gather that knowledge, what are the pain points that companies of our sizes have faced?

Now, tying into your earlier question here, how many regions is our system deployed in? If our region, our business is more specific to North America, then we would kind of keep most of our compute inside of North America, US Eastern or US Western. We are not in the place where we are deploying across regions, where we’ve got it in Singapore or Japan or even in Germany. We’re focused on, yeah, and so we can kind of focus our knowledge building on top of companies who already faced these challenges, and there are enough startups kind of adopting this technology that they’ve written enough and there’s enough knowledge floating around to guide us into a direction on the early challenges that we might face while interfacing with this technology and could be adopting this technology. Generally, my idea is to kind of gather that information, form a framework of, “Hey, these are the problems that we are going to face,” and yeah. Yeah, and then we kind of delve into that discussion, where, “Hey, how would this create value for the company?” That’s kind of the framework I ask my team as well, to use. To be honest, they are the ones who kind of influence this framework into me. This is something that I train the newer people who join the team into. Yeah, yeah.

David Joy:

That’s brilliant. I mean, so there is also this feedback mechanism where, once you’ve kind of assessed something, you go to your developer teams or development teams and they kind of give you feedback on what they feel about it, and there’s this collaborative process of decision-making, right? That’s pretty interesting. One of the thoughts I had when you were saying, was especially around Kubernetes. It’s a brilliant project that came out of Google right? The ability that it gave to enterprises to, of course, not just containerize use containers, but kind of scale things out in a pretty good fashion. For where Route is right now, it definitely feels like it’s a great thought leadership, fundamental, to think about scale in everything that you’re building, right? From that, I wanted to go back to a question around AWS itself and the journey to the cloud, right? Was Route always, from the very beginning, a start on cloud kind of company, or was it like they had their own data centers and then they moved to the cloud? Tell me a little bit about the cloud journey or how it all began, if you know that.

Siddhartha Sandhu:

Route is a fairly new company. We’ve been in operations around four and a half years now, and this company came out a little bit pre-COVID. You can see why COVID would boom the Route business, because basically folks are ordering packages at home, and that means there are more tech start-ups for this, and protect/track/resolve kind of worked really well, and it kind of made the business use case for the company itself. We came through that, so it’s fairly young, and AWS by that time, I think it was fairly, I would say, mature ecosystem. Route has always been on AWS, and Route from the get-go adopted GitLab as the hosting platform. GitLab provides pipelines inherent, native to the platform, which played into the philosophy of having software to deploy multiple times a day, rather than just the once.

In the beginning, I believe … This is before me, but I’m still owning some of that infrastructure, so I can talk about it. In the beginning, I think the deployment mechanism was more manual, but as the teams kind of grew, it was great that the leaders invested in developer enablement as well as site ability engineering teams that allowed for us to reach a place where, by the time I joined, in a couple of months only from the time I joined, we were in a continuous integration mode. That’s, the only time we have to click a button is when we deploy to code. It was a fairly good mechanism, so we were deploying at, say, a velocity of at least four or five deployments, big deployments a week, and then smaller deployments just going through the whole day.

David Joy:

Right, right. That’s interesting, because I think my whole experience too on AWS was initially, I would get those free credits, 300 dollars of credits, and start using it. Then I started loving the fact that pretty much everything I need is available there. Then, again, I reached a point where I needed to figure out, “If I have to build so much infrastructure, I have to build route tables and I have to put the security groups everything, I can’t do this manually.” Is started using Terraform, and I’m a huge Terraform advocate. I feel like it’s pretty good infrastructure as a code solution that works in multiple environments, obviously. It really helps me map out how I want to put my infrastructure together, so that maturity and building that into a company, especially, is really critical, I feel, as you try to scale.

Siddhartha Sandhu:

Yeah. There’s been a lot of conversation around the operating models of teams on how you can deliver product really quickly, and I don’t know if you read this. There’s this book that came out, I think, in 2019. It’s called Accelerate, and it basically talks about … It’s a fact analysis of what a high performing team is, and one of the things that they noticed as a metric of a high performing team is how quickly it deploys, getting in front of the customer. Right now, the market that we are in, you’re running against time, because everything has access to the technology that you have access to. If you’re thinking of … If you have just started kind of a unique space, which Route did, like the package protection, then the competitors are going to show up sooner or later. How can we keep ourselves at the, I would say, the cutting edge of our offering to them, so that merchants don’t turn away to other folks, is by keeping our technology stack in a manner that allows us to innovate much more quickly and be ahead of the game. Yeah, and your comment around infrastructure as code kind of started that thought. Yes, yes, yes. Trail of thought, yeah.

David Joy:

Yeah. I was going to say, when you were saying … I mean, it’s so critical for whatever your teams are building, for it to have feedback right from users, right? I’ve seen engineering teams that go back and have this brilliant idea, and they work on that idea for almost a year. Sometimes it takes 18 months, they’re making changes, and eventually it goes into production, and then it’s died. In the sense, they’ve got the feedback so late, by that time, things are moved on. Today’s day and age, when people want quick experience, quick feedback, everything has to move quickly. Having things planned out in micro-services and infrastructure kind of really helps out. I think that allows you to look at everything that you’re building in a silo, and kind of get feedback quickly. It seems like you have that kind of an approach at Route.

Siddhartha Sandhu:

Yeah. What we tend to do at Route during our deployments is, once we’ve deployed the product, we definitely track the success KPIs behind it, and sometimes things work out. Sometimes things don’t. Sometimes the product is adopted really quickly, and sometimes there are challenges that are not essentially with the technology, but how the technology is communicated to people for whom the value would be created. For example, ChatGPT itself, it probably creates a lot of value for me and you, being in the tech sector, but for a plumber, I don’t think it could create that much value. Trying to sell a technology solution which probably utilizes AI to make their work easier would be a harder sell. That’s kind of the balancing act. Even if you have a great product, it’s a whole team that comes together to actually get it into the hands of the consumers, and I feel like the consumer feedback, when it comes, when somebody already has that product in hand, is super critical at that point.

Not only were you able to build a good product, people adopted it, which is kind of the biggest thing for any startup, kind of moving out there and becoming successful. On adoption, what’s the feedback that you get, and how quickly can you rally behind the feedback that you get and fix the problems, or at least extend the solution in a manner that allows for the consumer to make it more, I would say, natural of an experience for them?

David Joy:

I think that’s the … From where you are as a startup, and where I feel a lot of folks are, certain risks are important. You have to go after certain things that you believe in, that can enhance your overall portfolio. I think what Route kind of offers is this very whole 360 experience for a merchant, where, one-stop shop where you have everything that is always available, and is providing you with the best information that they can look at to provide good experience to their users who are eventually buying their product. I think that it’s good, and I’m super glad that we have companies like Route that exist, and engineering managers and engineering teams who are trying to solve these problems, such as you. It’s brilliant.

I wanted to ask you this, just out of curiosity, but usually I don’t go into Cockroach related stuff or CockroachDB stuff. I wanted to ask you, you have an infrastructure designed right now where you basically operate within the US, so you have US East One, US West Two. Those are your two territories. What led you to pick CockroachDB?

Siddhartha Sandhu:

The decision pre-dates me, so our CTO, previous CTO, he was coming from Qualtrics. Cockroach was quite successful and has been. I don’t know if they’re still adopting it, but yeah, but was quite successfully adopted into Qualtrics, and when he came in here, they had to make a choice on the database where the orders would be stored. Orders are right now the crux of everything. Yeah, yeah, yeah. That’s the bread and butter. If that goes down, yeah, we’re not in a good spot. In those early phases of the company, the general idea is, how can I adopt something which does not require my developers to spend a considerable amount of time learning? Cockroach, with it’s easy-to-use SQL interface, kind of fit that bill. “Hey, we’ve got distributed compute. We’ve got relational matching in here, and we’ve got a language that most of the engineers know. Let’s kind of put ourselves in a position which will allow us to scale for the loads that we are going to see.” Right now, I remember my current … My manager, he told me that at one time, he had to drop 11 million rows. 1.8 million rows, yeah, from Cockroach. This was in the early, early days, and compare and contrast, now we have 12.1 terabytes sitting inside of Cockroach, all that good stuff.

Yeah, whoever made that choice kind of understood that Cockroach kind of provides a mechanism for you to scale, where … Without having to really, really affect the team’s velocity while the adoption is going on. I feel like that’s the biggest, I feel like, the biggest selling point for at least Cockroach for me. It’s, yeah, it allows you to use the friendly language, which everybody … People out of school, people are taught relational databases in school, and this is SQL, so when they come into a job, it’s not like, “Hey, this is MongoDB, this has got a separate query, and then this is DynamoDB, it’s got a separate thing.” It’s relatively easier to just adopt this technology, rather than having to spend too much time. Yeah.

David Joy:

No, I love that too. I mean, interesting that you brought up the fundamental idea of SQL being the language to talk to data, because it’s still getting taught in universities and everybody’s still … I mean, I know there are different databases that have done different directions with how they want folks to interact with, but distributed systems and having that layer of SQL I feel is amazing, because it allows you to do pretty much everything that you’re used to doing, but now at a distributed scale. The other thing I was just fascinated by, what you just said, was, for you, the most critical system is orders, right? Orders have to be very consistent, because if you have to show those orders out to different people or merchants, right, it has to be strongly consistent. I think across multiple regions, it’s very valuable, so it’s great that we see value of the product being used in that way at Route as well. It’s amazing.

Yeah, I mean, I know we are almost hitting our limit. The way I see this is usually, this is one of our first conversations, and we have so much to unravel, but usually an hour is great for a beginning. I wanted to ask you another question as we kind of close things out. Working as an engineering manager, what else do you do? How do you kind of stay on top of what’s happening in technology, as well as focus on playing Zelda and keeping up with all the 120, 130 shrines that you have to unlock? How do you do that? How do you manage all that?

Siddhartha Sandhu:

Around five years ago, I was pretty overwhelmed with what’s happening in technology. It’s just too hard to keep up. There’s so many things coming left, right, and center, and so I had to kind of identify, how do I want to start learning about things? Do I want to focus on only one thing, or do I want to focus on everything? After a little bit of soul-searching, what I started doing was I started reading research papers. This is how … I started reading research papers, and when I read a research paper the first time, it took me at least a month to understand, what did this person say? It was a very heavy literature, and so I kept doing that, and eventually I reached a point where difficult concepts became easier to understand, and that time period started shrinking for me. Now I’m at a point where I can look at a documentation, and just because my basics, like distributed systems and everything, are pretty fleshed out thanks to all those research papers, it’s easier for me to, I would say, grab onto concept quickly. I don’t have to spend a large amount of time trying to understand it. Feel like there’s a lot of solutions in the market, and understanding the value benefit of each of them much more quickly would help you kind of execute on things that would drive value for yourself.

An example would be, hey, trunk-based development is quite the industry standard right now, but there’s still products out there which work on feature-based development, feature branch-based development. I probably want to focus more on products that are trunk-based rather than on feature branch-based, because that introduces a lot of complexity. That initial process that I followed, “Hey, let me read about hard things so that I can get my basics straight,” when I got through that, I also realized that I need not practice everything. I just need to practice things that drive value for me. Then, I practiced those, and now I’m at a place where it allows me to take decisions quickly, kind of bring up … If I want to try out something new, then, yeah. I would say I don’t have that feeling, that, “Oh, shit, I’m going to fail,” all right, at this point, because it’s so tough and hard. Now I’m at a point where I’m like, “Something is going to happen. I don’t know what’s going to happen, but whatever happens, I’m going to be able to deal with it.” Yeah, so that’s kind of the mindset that I’ve evolved.

David Joy:

I was going to ask you a question after this, right, your advice to folks, but you kind of alluded to what your recommendation is for people, right? One of the things is, the fundamental idea, see, go, read research papers. Right? I also had this fear, right? It was always intimidating to go read a dissertation or a research paper that somebody has written. They’ve spent three years onto it, and then you’re trying to digest it, understand it. It feels a little intimidating and you have imposter syndrome sometimes, which I have all the time. There is a lot of humility in what you said, that, “Hey, go read something, and it might be complex for you in the beginning, but as you do something more, it’ll get better, and towards the end it’s just going to benefit you and the teams that you work with.”

Siddhartha Sandhu:

Yeah. The mind itself is a muscle. You exercise it, it’ll become better and better, just like the body. I got into the habit of just exercising it a little bit from time to time, and generally, now that you asked for the advice, it’s not only important … At least in the modern industry, it’s not only important that your architecture be sound, but it’s even more critical that you’re able to communicate that in a fashion that is digestible by other folks. This is where kind of Richard Feynman’s philosophy of learning plays in, too. If you’re not able to explain something to another person in a period of two minutes, then probably you’ve not understood it well enough yourself. Hey, it might be time to kind of go back, see, and understand it a little bit better, so that you can explain that thing to a layman in a period of two minutes, and they’re able to ask questions on that two-minute statement that you made, too. That’s kind of been my litmus test. I’m sorry to say, my wife’s been my guinea pig for the longest time.

David Joy:

Which wife is not? I always go bounce my ideas with my wife, and she’s like, “I don’t get it, man.” Yeah, I mean, but what you said is words to live by, right? If you can’t explain something to somebody in two minutes, I mean, there’s an art to pitching. There’s an art to being able to articulate ideas, and especially in positions where you are, you have to come up with … You have to digest so much information and be able to bring it in front of people and say, “Okay, this is what this helps us do,” and kind of summarize all of that. That, I feel, is great advice and words to live by. Amazing. Siddhartha, or Sid, and a fellow Tears of the Kingdom player, I would say it’s been an absolute pleasure to have you here on the podcast, and it’s been a great conversation. I’m going to let you go and help you finish up all those shrines that you’re finishing up and that you need to find.

Siddhartha Sandhu:

Thank you so much, David. This was a good talk. Yeah, I had fun.

David Joy:

All right, thank you so much.