From FedEx to Five Guys: Designing digital experiences with Yext’s VP of Software Engineering

Matt Bowman

VP of Software Engineering at Yext

Never miss an episode

Spotify
itunes
google

If you have ever tried to find a local branch of a major retailer, left a review on a product page, or talked to an on-page chatbot, you have likely interacted with the frontend of Yext’s Digital Experience Platform. But what goes into the backend for a modular and composable platform that can deliver for enterprises of all sizes? To answer this question, we sat down with Matt Bowman, VP of Software Engineering at Yext.

Join as we discuss:

  • The biggest changes in the technical landscape over the last 20 years from cloud adoption to application development.
  • Leadership principles, hiring strategies, and advice for fostering innovation.
  • The challenges and opportunities that come with AI.

Matt Bowman:

We’ve got developers who are learning, we’ve got developers who are growing, we’ve got developers who are becoming more marketable and they’re becoming excellent at what they do. And for me, that’s really the rewarding part is seeing people who came in not asking questions because they’re used to school where you’re not supposed to work with others, you’re supposed to solve the problems yourself. And seeing them grow into either managers or into super high output ICs or into architects.

David Joy:

What is up everyone? And thanks for tuning in. In today’s episode of the Big Ideas in App Architecture podcast, we speak to Matt Bowman, who is the VP of Engineering at Yext. Matt and I dive into some amazing things that he and his team at Yext are working on and how they design products for their B2B customers. We get into Matt’s leadership principles and how he and his team are building an ecosystem for innovation and drive their software teams to optimize for outcomes. It’s a fascinating conversation where we get into cloud infrastructure and data sovereignty as well as how AI is shaping up the world of software engineering. So pump up that volume and get ready for an intriguing conversation with Matt Bowman. All right, Matt, we finally made it to the podcast.

Matt Bowman:

Yeah, thanks for having me. Took a little bit, but we’re here.

David Joy:

Yeah, I mean, what I’ve experienced is usually… I mean, schedules are all over the place. Something keeps happening, but we’ve never had situations where we have to reschedule more than once or twice. So I think we are perfect. We’ve made it and I’m glad that you’re here on the podcast. And while we were getting ready, you told me last week that you went and played some golf at Pinehurst. Tell me a little bit more about that.

Matt Bowman:

It was a lot of fun. There were 12 of us that went down there. We played courses eight, four, three, and then on the final day we played number two. So it was a lot of fun challenging courses. Playing number two after being able to play three or four rounds was really nice because you don’t start with the toughest one at the outset. And I actually had my best round there, so really, really good trip and a lot of fun overall.

David Joy:

Do you like golf often nowadays or do you take the time to get out?

Matt Bowman:

This year’s been a little bit crazy, so it’s been more challenging. I saw some escalation in my handicap, but yeah, I try and get out there during the summer once or twice a week when I can find the time. It’s tough with kids though. And this fall, we had travel baseball, so every Sunday was fully booked.

David Joy:

I’d never gone into golf until a few years ago when I had nothing else to do and everybody was stuck at home. So golf course seemed like a place where you could get out to. So I went and started playing some golf and now I love it because I just feel like you’re playing against yourself more than you’re playing against other people, so it kind of helps you to keep getting better at the next shot and things like that. So also I’ve realized extremely frustrating game because… Any who. But I wanted to say thank you so much for taking the time to come on the podcast. It’s a real honor to have you on. For everyone listening, why don’t you let the people know of what your role at Yext is and what you’re doing that’s super impactful right now?

Matt Bowman:

Vice President of software engineering here at Yext, so I kind of oversee all of our product engineering. Pretty much anything that our customers use that’s not bespoke is developed by folks on my team. There’s been a lot of movement. We kind of are trying to see where our customers want us to provide more support and what we’re really seeing is a return to our core focus, the areas that customers came to us originally for. So our listings' product, our reviews' product. We’re going to be bolstering the development on those and we’re really looking forward to it. We just moved around some folks to try and add some additional horsepower to those areas. But yeah, that’s kind of the big area that we’re making an impact today.

David Joy:

You’ve been with Yext for almost 10 years now. I know you went from a software engineer director and now you’re leading this team. So we’ll dive more into how that experience has been for you. But describe for me what does Yext do actually as a company? What are the products like and how are people or companies… Is it like a B2B product right now or is it a B2C? Kind of open that up for us.

Matt Bowman:

Yeah, so Yext is fully B2B. We work with companies of all sizes, so all the way down to just single person SMBs, all the way up to companies like Verizon or Samsung that are using us for a handful of different reasons. The thing that we started out doing was our listings' product. And our listings' product was all about… It was called power listings. And what that allowed companies to do was they’d sign up with the X, they’d input all of their information about their different locations. So let’s say it’s like Seven Eleven or something like that. They’ve got thousands and thousands of locations. And then on top of that you have thousands of sites where people can find that information. And the way that that information is generally brought to those different sites is through aggregators. The problem with aggregators is that you don’t really control that information.

So what we did is we gave control back to the business so that what they could do is they put their information to our system and then we relay that out to all the different publishers in our network. In addition to that, what we started doing was using that to pull reviews because a lot of these listing sites, what they have is they have reviews. So we started pulling that in and allowing people to respond to them. And we’ve kind of organically grown and added new products that were complimentary to this set of products. But really if you look at what drives the majority of our revenue, it’s our listings' product.

David Joy:

And I know you guys have expanded your portfolio to have now reviews obviously, but you also have platforms and pages and analytics and content and you’re trying to do some cool stuff with AI as well, which I’m curious to know. So is your software engineer team focused on all these products across the board?

Matt Bowman:

So most recently I was mostly focused on our web publishing business, which really was about the pages' product. But through our pages' product, we would also host our search product, which was kind of a first in market AI driven search product. And more recently, we’ve started adding in our chat agent, which I think you’ve seen a lot of companies doing because it does help lower support costs, increase customer satisfaction, those sorts of things. So we have a fully dedicated data science team that does all the research. They build models. Sometimes we work with external models. It’s really just a question of what problem we’re trying to solve, but they’re kind of always on the cutting edge. And then we’re trying to kind of integrate that into our system, make sure that we keep a continuous feedback of information back into them so they can retrain. We’ve got data labeling, all sorts of fun ML engineering type problems to solve. So that’s relatively new for us, but I still think we were pretty far ahead of things. I think we started that in about 2019.

David Joy:

Right. Yeah, which is kind of a sweet spot of when everybody was getting into this with data and everything. Amazing. So it’s been 10 years at Yext. And tell us a little bit about… Even before we talk about Yext a little bit, how did a young Matt Bowman decide this is what he wants to do for life and play golf? So yeah.

Matt Bowman:

I like hearkening back to the original computer that we had. It was a Commodore 64 and I would just play around with that. We bought it from a neighbor that came with a whole box full of five and a quarter floppy disc games. And so I was busy with those and then I realized that, oh, this is actually pretty cool. I wonder if I could make something that did something on the computer. So I messed around with that a bit. We got a 386, I started doing QBasic. I’ve been interested in computers since I was about eight or nine. It’s kind of been a lifelong thing. I developed some software that I ended up selling to local swim clubs, so it was kind of just a lot of fun. I would just sit and program all day. Outside of swimming, which was a full-time activity. That was what I was doing.

David Joy:

I’ve spoken to so many people and it’s so amazing to know different stories and beginnings, right? And especially when somebody says I build a software and that’s what I loved, that was programming. So at the core of it, even if you’re a leader today, at the heart of it, you’re still a programmer. So do you miss working on hands-on code? Do you get to do that nowadays?

Matt Bowman:

Oh, man, not very much. No. I definitely miss it a lot. It was a very difficult… I would say that was probably the most challenging transition was knowing that I could participate in the actual coding, but knowing I wasn’t supposed to. And so it was kind of pulling yourself back, letting others make decisions, letting others kind of drive how we’re going to architect things. And really I’m there to facilitate things and make sure that things go smoothly and basically deal with the stuff that no one who wants to write code wants to deal with.

David Joy:

The landscape of technology has changed so much in the last 20 years, right? The stuff that you were working on basic… At that time we were building websites, say on HTML and then CSS was there and now we’ve moved to React and you have this whole idea of building applications with components and next year there’s lots of changes in at least front-end development and then mad amount of changes in backend with databases and cloud and everything. What’s been your take on all of this?

Matt Bowman:

I remember a lot of the software we wrote early on in my career was dedicated toward Windows machines and Internet Explorer and really just making sure we had good support there. And then I remember people up in arms about how Internet Explorer was so non-standard, and then you started seeing buildup on Firefox and then Chrome came out and there was just all these changes there that were great developments because it did provide some level of standardization across all of those technologies and made it easier to develop. I can’t tell you the number of shims that we always had to introduce into all of our code in order to deal with all these front-end bugaboos. We used to be all on-premise or all managed data center. We would drive there, we’d scan the back of our hand to get in, there’d be a human trap in order to make sure that only one person was going in and out, really cool data centers, but we used to be very much inclined to do that sort of thing.

Because it was like, well, you got to manage your own hardware, because you got to know what kind of power you got. You don’t want all this noisy neighbor stuff. And then we started experimenting a little bit more like I left that company and I was working at a small software consulting firm, and our clients really needed us to stand things up cheap and quickly. And really the only option there was to go with the cloud. You could stand up a full website, you could stand up a whole system for hundreds of dollars a year versus tens of thousands, which is what you’d probably be paying in a managed data center. So the innovation kind of skyrocketed and when I was originally skeptical and totally wrong, I am all in now. We love going to the cloud because it allows us to control costs. It also allows us to expand when we need to without having to be like, all right, let’s go rack some more servers and accept this massive capital expenditure.

David Joy:

Do you have a multi-cloud strategy or all in on say one particular cloud like AWS or Azure?

Matt Bowman:

The team that I originally joined with here, I had worked with it at the previous company and so we were very much about our own hardware. And so we had a big data center in Secaucus, New Jersey, we still do. And we’ve got one, a managed data center in Texas, but we just opened up a second mothership in Europe and that is entirely in the cloud. And so we do have support right now for GCP as well as AWS. And we were in Azure for a little bit and we do plan to return to that, but we now have… Depending on the product, we have data centers kind of all over the place. Cloud data centers all over the place.

David Joy:

What’s your personal preference when you think about these deployments and of course the scale and the robustness and the elasticity that you get with cloud? Do you feel like being all cloud makes sense or do you think some sort of a hybrid makes sense just to make sure that you have the freedom to run your own stuff also and control it? Is that where you lean?

Matt Bowman:

I mean, I definitely would lean to cloud first and then once you realize or have stabilized and once you’ve decided we understand the workload, we understand this domain really well, we know what to expect from traffic, we know what to expect from all these different aspects of how the system operates, how our customers interact with it, it can make sense to do an on-premise or a managed data center cloud of your own running with one of these great orchestration frameworks like a Kubernetes or a Nomad, which is what we use.

But I think generally, I would always start out in the cloud. I think it just makes so much sense. It’s so easy and it is so inexpensive and you can experiment and you can tune and then once you’re like, we’re spending too much in the cloud and we could easily take this in-house, we understand it, we can deploy it. As long as you’re using a lot of these sort of modular components like the orchestration layer, like I mentioned, you can pretty easily just stand up a bunch of machines in a data center, deploy that to it and get your code running out there.

David Joy:

I spoke to a bunch of people and one of the person said, “In 1999, I was writing my own database because Oracle was expensive.” And then today he was like, “We have so many solutions, we can go to the cloud, we can run [inaudible 00:14:36] on the cloud, or we can select a Postgres database.” There is a plethora of technology available and Kubernetes definitely is something that changed the game. So do you feel like this is… If you were born or you were getting into tech today, do you think that’s more exciting or is it exciting 20 years ago when you had no documentation, you have to go to so many different forums and places like finances to it? No Stack Overflow. So what do you think about that?

Matt Bowman:

Honestly, I don’t even remember where I would go to get my information. I remember building my first Linux system and all I could get it to do was gray scale. And I was like, “This is so frustrating.” And you’re looking at your drivers, you’re looking at what graphics card you have. It was so challenging. I would say it was more of a slog and you were far less productive. I’d say today is probably a lot more exciting because you literally can stand something up. My son plays Minecraft and we stood up a Minecraft server. It took no time. It was amazing. I mean, it’s the sort of thing that you just… I don’t know, so cool. The other problem that you would have with your Linux machine is that you’d get it running and then you couldn’t connect to the internet and you’re like, “Oh no. Now what am I going to do?” You have to go get a second computer to try and find out what’s wrong with your driver for your ethernet card.

David Joy:

Oh, yeah. I have done stuff where I would have the same program file running somewhere else, and then I would go line by line, compare where did I mess up the code and then I would realize that I missed a bracket. Debug was so difficult at that time, but those were the days. I think folks have gone through an experience like that. And now with all of this, it’s amazing, but at the same time, I think it just makes us so much more confident about what we are building, right? And so your software engineering teams, the teams that you manage, what are the kinds of things they’re working on right now? In the sense, can you expand on that a little bit?

Matt Bowman:

So when I originally started working on our web publishing platform, our pages' product, we were very early to the static generation game. I mean, this is something that more recently has become kind of an industry standard, this sort of… I think they’ve retired the term, but it was like the Jamstack. But the static generation, people were like, this is really great for sites that don’t have tons of dynamic data, right? It allows things to load up really quickly. You can get everything out to a CDN. It’s just like the performance and the customer experience. They’re not experiencing five-second delays loading up a page, right? Because anyone who’s seen a five-second delay is back and clicking on the next link, right? Yeah. They’re gone. So we had a system that did that. It was our classic pages system, but it was entirely internally managed. It was entirely serviced by our services team, which kind of limited I think our ability to use it.

And we also kind of had some plans for this new system that was going to be based off of streaming data. So you define your stream, what are the linkages, what are the data from this graph of information? And anytime something changes across that graph, we want to generate a new page or an update to that page, right? So you have to understand the linkages between all of the data points in order to do that. And so what we built was the stream system. Well, the stream system wasn’t compatible with the old way pages were generated. So we ended up building this new thing that we had called sites that allowed you to use this streaming concept and generate pages on demand using your own code, running in serverless functions. We got a whole CI that we built for it. It was a really cool and fun project to work on.

So we did that. Our streams project was really big and it’s really mostly an internal thing, but in terms of the power that it’s going to give us in the future, I think it’s pretty incredible. We’ve been doing the big move to standing up a European mothership. So a secondary mothership was a massive undertaking, and really it forced us to think about how can we deploy our code in a much more modular fashion. Because when you build something organically, you end up with network routes and firewall rules and all sorts of crazy stuff that you try to deploy it to a fresh cluster and it just doesn’t work. Nothing can communicate, everything’s not working. So that was a really big undertaking, the team recently wrapped up.

David Joy:

I mean, and setting up anything in Europe is becoming more and more interesting, right? Because of all the new regulations that they have and compliances that you have to follow. So does your product have to adhere to the sovereignty laws and data sovereignty goals that Europe wants to maintain?

Matt Bowman:

Yeah, so GDPR was definitely a major driver for our decision to do this. I mean, as I mentioned, it was a massive undertaking and there wasn’t incredible demand from the European clients because what they were seeing with the performance of the system was generally fine. They didn’t really have any issues with it. But once the sovereignty stuff started coming in, they started getting concerned and they were like, “Look, we don’t want to deal with all this silly stuff. Can we get migrated over to Europe?” And we’re like, “Well, we can’t really migrate you to Europe until we build an entirely new Yext system over there.” So yeah, it was largely driven by that.

David Joy:

If you’re following AWS last week announced that they are building an AWS cloud that is only for Europe, not just region, it’s just an entire cloud environment that will work only in Europe, which is pretty interesting. I know Google’s committed to that as well, but I think they kind of announced it. So I think the whole idea over there also is that only European employees are going to be working in those data centers. So they’re very strict laws and it’s pretty interesting how that’s shaping up.

Matt Bowman:

I wasn’t aware of that in particular, but it sounds almost like a FedRAMP. It’s something that’s totally isolated, follows a very strict set of rules and as you mentioned, who’s working there, who’s accessing the data there, where are they located? There’s a lot of stuff there that we’re trying to figure out. And I think the lawyers are even struggling to figure out what exactly to do.

David Joy:

Yeah, I think it’s good value prop, right? If you’re being pushed in a position where you have to contain everything. And I think the best way is what you put. It’s like a FedRAMP for Europe, but it only works in Europe, kind of a cloud environment. What else does your team use in terms of technology? You mentioned Kubernetes, obviously a Nomad, something that you’ve… Tell me a little bit more about what do you guys use for your front end applications? And I mean, you don’t have to give any secret sauce if you have any, but just generally for folks to know, how are companies like yours that are leading the path in this space, what kind of stuff that you use to build this kind of an experience?

Matt Bowman:

The challenge with leading the pack is that you’re kind of constantly in a state of migration. And so if you looked at our stack five, six years ago, it would look very different from the stack you see today. But when it comes to our primary stack, we’re using React for our front ends. We still have like Java play servers kind hosting our web stuff. Because there is just a massive amount of stuff that we’re running on there that is something that we’re kind of in the midst of a bit of a migration, just trying to feel it out, see what we can do, see what makes sense. We’re using Google’s closure compiler in order to compile our JavaScript, which is kind of like a throwback. A lot of people aren’t even familiar with that, but it was kind of a big deal at the time.

It does tree shaking. You kind of get these small compact binaries, which was a really big deal, especially if we were trying to service low internet areas. But generally speaking, we’re dealing with business customers who have a business internet connection that are using their computer to do the work that they’re doing. So for us, it’s not as much of a differentiator to have these micro binaries. So kind of moving away from that, because there are some kind of [inaudible 00:22:54]. They’ve gotten a thing called advanced compilation, which restricts some of the things that you can do with JavaScript, but it also helps it trace through and see what’s being used and what’s not being used so that it can actually chop out whole functions or hold files that aren’t ever being touched by the code. But it’s a little bit challenging because if you compile it not in advanced locally, because advances can take some time, there’s a chance that when you do compile in an advanced mode, it won’t work.

David Joy:

Interesting.

Matt Bowman:

I don’t that don’t like that from a developer standpoint.

David Joy:

Unpredictability. Yeah, yeah.

Matt Bowman:

Exactly, exactly. So we’re kind of moving away from that. But yeah, React front, Java play, but we are using a lot of Go, now. We’ve got a significant amount of our system running in Go and it’s a very slim HTP server that’s doing all the Go web serving.

David Joy:

We can come back to this. I mean, I want you to continue obviously, but is go because you have a lot of developers who like Go? They’re saying that this is the right place because I’ve seen a lot of developers who love Go, just the way it is. Is it because of that or is it you just… Company felt like Go is a language that you need to get to?

Matt Bowman:

Yeah, I’ve got some insider information on this one. A good friend and a former colleague who’s now working with other former colleagues at a startup, he kind of fell in love with Go and he at the time was a very influential member of our technology team and he was like, “Guys, we should use Go. I really like it.” He ended up doing a ton of work to make it so that the entire team could participate and use Go within our ecosystem. It’s an unbelievable undertaking to actually do something like that. And he did it all on his own and kind of got us to the point where we could use it. So yeah, it was kind of a personal preference thing for him, but it’s spidered out into different parts of the organization. I think a lot of people really do appreciate the simplicity.

David Joy:

Yeah. No, I agree. All right. Yeah, I know I stopped you, but yeah, go ahead. Continue. So you use Go, React and what else do you guys use? Yeah.

Matt Bowman:

Yeah. So depending on where… If we’re in our managed data centers, then you’re going to probably see a bunch of MySQL servers. We’re like a 17-year-old company at this point, like 2006. So we were using MySQL a long time ago and we really haven’t seen any compelling reason to leave it. I think if we were, I’d probably say Postgres or something like that, it’s a little bit more fun. But anyway, so that’s our primary relational database kind of communication between our different systems. We wrote a homegrown RPC library and it has served us really well. It’s actually really great.

We’re trying to move over to GRPC to kind of replace it but you kind of see where we were really strong with that system. It’s able to handle just an unbelievable amount of traffic and GRPC is kind of choking under the load in some cases. So it’s kind of funny to see that we definitely wrote something that solved the problems that we were facing as opposed to kind of a general system that solves a lot more problems, but maybe not necessarily as narrow as what we’re looking for. So we’re moving to GRPC, but currently we’re still on what we call Yext RPC. For message handling, we use RabbitMQ internally, and that’s… I mean, it’s solid as a rock, as long as you don’t abuse it, it is just unbelievably reliable.

David Joy:

I think your use cases have a certain scale and it works for you. I’ve seen experiences where trying RabbitMQ and just at some point I’m like, “This doesn’t scale and I cannot live with it.” And my experiences is working with use cases that require scale and resiliency, and there is this need for terabytes of data to flow between systems and stream and real time. So I think in those cases, I’ve seen KFKA along with a great CDC on a database work really well. But I agree with you that RabbitMQ in a use case where it fits the bill, it fits the bill really well and it’s been around for such a long time, right? So…

Matt Bowman:

It has, and it was written in Erlang, which is probably not a lot of people know that, but that ran the entire European telecommunications system for… I’m not sure if it still is, but it might be. Yeah, so we are using KFKA as well. I think one of the challenges with RMQ… At least from my perspective, is that it can hold so much data and you don’t know what’s in there. So you have to figure out ways to inspect it. How do I know what’s in this queue? What’s backed up? When is this customer’s problem going to be solved? You just don’t know because it’s opaque. You’re like, well, it’s somewhere in the queue. It’s somewhere in there. And then you can also run into issues where you have these sort of massive fan outs where a single message will go into a queue and all of a sudden you have thousands of outbound messages as a result because everyone subscribed or we’ve got like 50,000 exchanges because when we wrote it at the beginning, only five customers were using this, but now we’ve got everyone using it.

And so yeah, we’re using KFKA though, as well as MirrorMaker in order to transfer data to our different… We’ve got our source data region, which is kind of like the mothership, and then we have our consumer serving regions, and those are kind of scattered around the globe. And that’s where kind of all the customer facing stuff. So when I say customer, I mean, it’s really more our customer’s customer. So someone who visits a website that’s hosted by our pages' product, that’s where it’s going to be served from.

David Joy:

Do you have all of this technology available? So your software engineers, if they have to do any sort of R&D, do you have any protocols around… Not protocols, but more process around how do you foster research and development and how does that flow into your product lifecycle, stuff like that, that you manage?

Matt Bowman:

I probably tend to be too open to new technologies and would kind of have us on 50 different stacks because it’s so fun to see them and you’re like, oh, this actually solves this problem really well. But at scale… I mean, and in scales relative, we’ve got 170 engineers now, and when you have that many people and you have a massively diverse set of technologies, it can slow people down and it can inhibit movement across the organization. If we need… Like what we just did. We just moved a handful of people from a bunch of different teams over to a different area.

If we were all using disparate technology, not only do they have to learn the problems that are being solved by that code, but they also have to learn how to code in that language and do it effectively. We can learn languages pretty easily. People can observe and they can read, and we’ve got auto complete and copilot. But ultimately, it comes down to, if you don’t know exactly how the code operates, if you don’t understand maybe how the JVM operates, garbage collection, how it works, you can write some really suboptimal stuff and if it gets out into the wild, it can really cause you some headaches.

David Joy:

There is a difference between somebody who can code, somebody who understands the code is writing, somebody copy pastes the code and understands the code, right? So you kind of segued into a space, which is a mammoth of a conversation, right? But we can touch on it, is you brought up copilot and the whole idea of AI. How do you feel about all of that impacting your teams and where all of this is going right now?

Matt Bowman:

Yeah, so as I mentioned, I probably get too excited about these things and my belief is them as being a silver bullet. It’s probably too strong. We started with kind of a little pilot program for copilot this spring and we rolled it out to anyone who wanted it. We said, “Hey, get on our license and use it and test it out, see what you do or see what you think.” And we did a couple surveys throughout the period to try and get a sense of what was going on. And while I think people did find it interesting and in some ways helpful, I just don’t think it’s where… It’s not at that point where it’s like, oh, my God, it’s amazing. I couldn’t code without it. I think people are probably at that point with autocomplete, but I don’t think copilot’s quite to that point. But it is exciting.

David Joy:

I agree with you. I think there’s some cool things that they’re shipping out in copilot, but again, I mean, before copilot there was this… If you use [inaudible 00:31:37], you had this package called Tabnine and that would kind of auto complete stuff for you. That was really good. Sometimes when I get boilerplate code, I’m so skeptical of the code, like where did it come from? So I have a distrust, so I will actually double check everything that I’m writing. And I mostly code… Whenever I get to code is right now I’m doing it in Node or React or JavaScript basically, or Python. Those are my two go-tos. And I’ve been trying to do some Go stuff, but this stuff around AI is changing the way we perceive problems right now. So I know your search product specifically has… You’ve added some AI capabilities to that. So does your ML and AI team… Are they exploring all the amazing APIs that OpenAI is releasing and testing them out? You don’t have to tell me if there’s a secret around it, but I would be surprised if you guys are not. So yeah.

Matt Bowman:

Yeah. I think there’s some secret sauce, but most of the secret sauce is probably in our people. I think when you look at the capabilities of what’s out there, it’s pretty incredible. But then you see what OpenAI just did with their stuff and they were like… They made obsolete a ton of people that were kind of just piggybacking on their technology. So yeah, what we do want to make sure that we’re doing though, is paying attention to the different offerings they do have. And we’ve spun up a machine learning engineering team, and really what their goal is to kind of act as the bridge between our engineering teams, our non ML engineering teams and our data science team. So our data scientists can focus on data science problems and kind of going out there and figuring out how do we make the best use out of our [inaudible 00:33:29] that we’ve got.

Because those are expensive as hell. And then our engineers can also say, “We know there’s a problem,” or our product managers can say, “We know there’s a problem. We know that there’s a solution out there that we’ve already developed,” and that they can actually interface with it. So we’ve got our kind of gateway for all of our… We call it our data science gateway that acts as kind of the broker between our internal systems and the data science machine learning models, the training. There’s tons of data that we’re just throwing off from these things that are then being used in retraining. So yeah, I would say that gateway also though, has a way to open up to openAI to models potentially being run by hugging face. So there’s definitely… We want to make sure that we’re capitalizing on all the things that can help our customers the most, but we are doing a ton of work internally to develop new models and train them on customer’s own data because they really like that when the AI sounds like someone who works there.

David Joy:

I think the challenge is that the speed at which this is developing is so difficult for us to keep up, especially when companies are working on this on the site or somebody like me who’s testing it out without the enterprise umbrella of security and making sure that it does what it does. Because at the end of the day, you’re trying to build a product that is AI driven and helps your customers and say, it says something, it can impact their revenue, right? That’s the piece that I’m shocked by that. The space [inaudible 00:35:03] is innovating is extremely wild.

Matt Bowman:

So there’s been this challenge, and I’m sure you’re aware of it. This big challenge where AI bots will kind of go off the reservation. You can think of binging and getting the Bingbot… I can’t remember the name of it, but just getting upset with the user, getting upset. It’s really not angry at them, but it sounds like it’s getting upset with them, and you don’t want your customers getting yelled at by your AI bot, right? That’s a really bad experience. So what we’ve done in order to prevent that, and I think it’s becoming more of an industry standard, was we figured out… Because our search product was so good at finding the information, using an inbound query, using NER, trying to figure out what is the intent of this search? Not just taking the words and doing a tiff diff and doing index, right?

We’re doing a lot more intelligent stuff. We’re examining the query, we’re hitting a bunch of different endpoints, we’re figuring it out. Anyway, that’s our search product. We’ll actually do that with their chat question, and then we’ll transform it into something that we think would be a good search. We then use that search in order to query our knowledge base or our customer’s knowledge base, and then we use that to formulate and feed that back into the AI to respond. So that kind of keeps it a little bit more narrow, a little bit more on the reservation.

David Joy:

Oh, yeah. Yeah. I mean, what you’re saying is… I mean, it’s not completely a rag system, but you have added your own knowledge and you’ve taken the prompt to get a better response out over just relying on the AI’s response. I don’t know if… This is a tidbit that I can share with you is about a week ago, there’s this company called Vectra who released an open source model. It’s on hugging face, it’s called Hallucination Evaluation Model. And what they’re doing, it’s like an open standard where they’ve taken a small model that has been around for a while, they fine-tuned it. And then across that model, what it does is it looks at questions and it summarizes responses, and then it looks at GPT-4, GPT-3.5, Llama, all of these models and rates their results on how the hallucination levels are.

And it’s interesting. So they’re going to… It’s open source, and you can kind of look at those in your context as well. But the interesting findings from that was that GPT-4 and 3.5 have the lowest hallucination rates, whereas… Or Google’s palm oils have the worst hallucination rates. So that’s why nobody’s using, there’s a direct correlation there. But what you’re trying to build, it sounds like, is the right approach to implementing AI for enterprises and to B2B customers, right? And I’m glad to hear that… I mean, that’s the consensus amongst seasoned engineers and teams and leaders, so that’s good.

Matt Bowman:

Yeah. Yeah. You still get sometimes the hallucinations, you just don’t get the AI yelling at the customer for prompt engineering and them. We’ve been able to really do a great job eliminating this, but early on we were seeing stuff where you would ask it a question and it would just make up an answer, and you were like, “You don’t actually know that that.” Nowhere in the search results did this information come up.

David Joy:

Yeah, yeah. Yeah. That’s like classic hallucination, right? So that’s where it is, but it’s cool to know. Let’s pivot a little bit, Matt. The most exciting thing when I was seeing you come on board was learning about your leadership philosophies, right? Obviously, you’ve been a software engineer, but leading a team now of people and guiding, and you have to make sure that principally they’re driving to outcomes and things like that. So what are your leadership philosophies that you’ve developed that you can share with everyone?

Matt Bowman:

Sure. Yeah. So there’s a really good book called The Ideal Team Player that we’ve modeled a lot of our core values off of. And really what that comes down to is we’re looking for people that are hungry, people who want to learn, who people who want to grow. People want to find the next thing that they can do to help make an impact. We’re looking for people who are humble. We don’t want people who think their stuff is too good and no one can approach them and ask them a question. We want people who share the credit for things that get done, and we want people who have kind of a higher emotional quotient. We’re looking for higher EQ folks. We call it smart. Well, ideal team player calls it smart, and our behavioral is entirely focused around that… Sorry, our behavioral interview, the one that we do for inbound folks, and it’s delivered only by our most senior people on our team, and it is also the way in which we evaluate people’s performance.

So really what we’re looking for and what we found based on this model is that you can have some really high productivity stuff. You’re not dealing with silly sort of interpersonal issues with people. Everyone can kind of focus in around what are the things that are going to move the business forward? And I think what we found is that generally it helps increase morale, it helps increase output, and we’re not even looking for output from purely from a Yext thing. Obviously, it’s great for us as a company to have high output engineers, but really what this means is that we’ve got developers who are learning, we’ve got developers who are growing, we’ve got developers who are becoming more marketable, and they’re becoming excellent at what they do. And for me, that’s really the rewarding part is seeing people who came in not asking questions because they’re used to school where you’re not supposed to work with others, you’re supposed to solve the problems yourself. And seeing them grow into either managers or into super high output ICs or into architects, it’s very satisfying to see that and see what they can produce.

David Joy:

Early on in my career, I mean, I was also so afraid to ask questions, and then I got introduced to open source and there [inaudible 00:41:18] on Discord channels or Slacks. I’m just seeing randomly people throwing their errors and saying, “I’m getting this error. What does this mean?” And then there would be somebody nice enough to… And it kind of opens up your thought to the fact that, well, not everybody knows everything and you can ask questions and you can grow. And I think my growing has happened through learning from these people with this level of humility. So I think it’s great to have that. Have you had experiences where you have an engineer who just struggles to… Who does great work, but struggles to communicate how good his work is? How do you handle that?

Matt Bowman:

I think that is the problem that we probably encounter the most, is that the folks that really are really have embodied the core values of the organization, they’re not going to toot their own horn. So we need to count on our managers to be doing that for them. Because the first thing you ask them is like, “Wow, that’s an amazing system. How did you build it?” And they’re like, “Well, actually, I only helped out in this area.” They’re going to push away the credit. So yeah, how do we deal with that? Our hope is that our managers are keeping an eye on the stuff. They understand what’s going on, and all of our managers are engineers, current or former engineers. So we’ve got people who understand what it takes to build systems, who understand what it takes to manage a system that’s gone from new to production to legacy. There’s so much that you can kind of learn over the course of managing the lifecycle of software, and I think it helps them understand. I’m working with this legacy system, no surprise that our regression tests tend to fail when people make big changes. That’s just how it works. Whereas maybe someone less seasoned would be like, “What the heck? Why are you making this change and breaking everything all the time?” It’s like, “Well, this system’s a little brittle.”

David Joy:

Yeah, I get it. In this scenario, right? You have a team of 170 people. How do you foster innovation within new product development and things like that? Do you have a design team, a software engineering team, an architect team working together with a team that has data from what the next requirements are? How do you operate in this [inaudible 00:43:40]?

Matt Bowman:

So for incredibly new stuff, our former CEO who I’m still friends with, and I started working with him in 2005, he had wild ideas and he would come up with them and he would figure out how to market them, and that was kind of the driver of innovation. That wasn’t necessarily innovation from use this neat technology. It was more like a product innovation. So there’s that type of innovation. Then there’s the other type of innovation, which is like how do we embrace new technologies and actually make sure that we’re staying current? So the way that we’re organized is we have our product areas, there’s a head sort of engineering lead for each of our product areas. There’s a product manager for each of our product areas, and they work together on the roadmap, building user stories. And then we’ve got, within each one of our product areas, we have what we call a group architect.

And so this is kind of our top level engineer for that organization, and they’re in charge of how does the system perform, what are the error rates, what’s the regression? How frequently are we having regressions? They’re kind of like they own the capabilities and the functionality of the system and how it’s built. And then we have our sort of teams that have tech leads, and those tech leads are in charge of various projects. So a project will come in, they’ll work with a group architect to kind of bounce ideas off each other as how it’s going to be designed, and then the team swarms it and works on it. So that’s kind of how we’re structured.

David Joy:

You also have the challenge of the technology. There’s so much going on. So what do you do to keep up with what’s happening in the space, stay up to date and kind of bring a distilled version of, “Hey, we need to start looking at it this way and start thinking about it.” How do you manage that?

Matt Bowman:

Well, listen to the Big Ideas in App Architecture podcast of course. I have gotten some good ideas. Yeah, I have gotten some good ideas from that. I was reminded of The Golden Path by one of your guests. I was like, “Oh, yeah.” I looked at it and I was like, “Oh, my God, we should be doing this.” So definitely that. But 10:00 PM, Twitter scrolling. I used to be much more active on Hacker News. I haven’t been on there in a while, but that’s a really good place to stay abreast of what’s happening. But I also lean pretty heavily… We’ve got some just very, very brilliant engineers that are dedicated to their craft and know it and love it, and they’re constantly investigating new stuff. And I remember when I first joined, one of our engineers was writing his own… It wasn’t even a test app, it was a big thing, and he was writing it all in TypeScript and he was like, “Yeah, I really like TypeScript. It’s pretty cool.” This is like nine years ago.

David Joy:

There are a bunch of people like that who love TypeScript. Yeah, I’ve met a few too.

Matt Bowman:

Really, when it comes down to it, I’m relying so heavily on our engineers and our architects to help drive those decisions and test things out. Getting them in the right roles and having them do those things is just so important to stay on top of the innovation and staying on top of the new technologies.

David Joy:

I always ask people who have been in the space for a while, because we have so many young folks who also listen to the podcast. What is your advice to teens or young engineers who are looking to advance and learn and grow in this space right now?

Matt Bowman:

Yeah. So this is really geared toward the more junior engineers, the folks that are just getting started on their career. I would look at the best people in your organization and try and do what they do and then try and do it better. Because if your company’s hiring properly, you probably can do better than that person. I know most of the engineers here can probably smoke me when it comes to anything like this. So I would say just work really hard, make sure that you’re working well with others, and yeah, just become excellent. Become excellent at what you do. That is the way you’ll grow your career and everyone will know it and people will know you by reputation, and that’s a really important aspect of becoming great as a software engineer.

David Joy:

Awesome. Spoke like a true veteran of the game, Matt.

Matt Bowman:

Those are all the people I want to hire.

David Joy:

When the episode releases and if you see a bunch of people hit you up on LinkedIn and ask you for a job, well, your goal is completed. All right, so if you want to look for Matt, look for him on LinkedIn. Matt Bowman, VP at Yext. It’s been such a pleasure having you on the podcast. I mean, I am super excited to follow what you do next. And I mean, I hope this is just the first conversation we are having and hopefully have you on again and talk about some more fun things.

Matt Bowman:

Awesome. Yeah, this is a ton of fun. I really appreciate it. You make the experience so easy, so I appreciate it.

Big Ideas in App Architecture

A podcast for architects and engineers who are building modern, data-intensive applications and systems. In each weekly episode, an innovator joins host David Joy to share useful insights from their experiences building reliable, scalable, maintainable systems.

David Joy

David Joy

Host, Big Ideas in App Architecture

Cockroach Labs

Latest episodes