Scaling for 2 billion events per day with Principal Software Engineer at Red Ventures

Majid Fatemian

Principal Software Engineer, Data Platform at Red Ventures

Never miss an episode

Spotify
itunes
google
youtube

With a growing library of media properties including CNET, Lonely Planet, and Bankrate, Red Ventures’ data platform has scaled from handling 16 million events per day to nearly 2 billion.

To dive into the evolution of their architecture and understand how data effortlessly travels through their systems, our host David Joy sat down with Principal Software Engineer Majid Fatemian.

Join as we discuss:

  • Why Red Ventures migrated from SQL Server to Amazon Redshift
  • How the data landscape has changed over the last two decades
  • The challenges every leader should expect when running a data platform
  • Majid’s time at Ubisoft and his involvement in one of David’s favorite games, Far Cry 2
David Joy: What is up, everyone? Welcome to this new episode of the Big Ideas in App Architecture Podcast where we are going to be speaking to Majid Fatemian who is the Principal Software Engineer at Red Ventures. In this episode, Majid and I get into some of the things that he and his team are working on at Red Ventures. He speaks about the data platform solution that their team has built that scales to billions of event streams. We also get into Majid’s experience working at Ubisoft and also, get into some of the cool things that he has done such as getting credited for work around Far Cry, and Assassin’s Creed, and a bunch of other games that he has worked on. So, without further ado, please listen in. Pump up that volume for a great conversation with Majid Fatemian. Hi, Majid. So welcome to the Big Ideas in App Architecture Podcast. It’s a pleasure to have you. How are you doing today? Majid Fatemian: Very good. I appreciate it, David. Thank you. Thanks for having me. David Joy: I wanted to say thank you for taking the time to come to the podcast. It’s an absolute pleasure to have you. For everybody listening, Majid is actually a principal software engineer at Red Ventures, and he’s part of the Data Platform Team. They are doing some interesting things. So, in today’s episode, we are going to dive into Majid’s career, some of the things he has worked on in the past. He has worked at the Ubisoft, the company that helps with gaming and a bunch of other things, and it’s an absolute pleasure to have you, Majid. What is your current role, and what is Red Ventures? Majid Fatemian: Absolutely. Thank you, David. It’s a pleasure to be here, and just to give a brief introduction about Red Ventures. Red Ventures is the parent company of many industry groups. That includes travel, financial services, media, and commerce education. You may know it from brands like Cena.com, Lonely Planet, The Points Guy, Bankrate.com, CreditCards.com. Red Ventures as a whole helps people discover and make informed decisions through a variety of different brands that it has to provide all those information to the people. My role as a principal software engineer. I’ve been with Red Ventures for a little bit over eight years now. I’ve been working remotely for most of this time with the team, and the headquarters are in Indian Land, South Carolina, a suburb of Charlotte, North Carolina. I go back and forth through to Charlotte very frequently. My role is I’ve been part of the data platform for the first seven years of my tenure at Red Ventures, and in the last year, I’ve joined one of the verticals to help them integrate into the central data platform that we develop with the new modification and changes that we have done to the data platform, which we’ll talk about the details. David Joy: Yeah. That’s brilliant. I mean, that’s one of the things I was really curious to learn about. Every business is different, and they do some interesting things. Even though we are developers and architects, there are certain things that we carry that are similar, but the way we put things together is so different. Right? Majid Fatemian: Exactly. David Joy: It’s like everybody is building a different engine, so I’m always curious to dig into that. So, when you started Red Ventures, the kind of team you were getting into, was it like you had to start building everything, this data platform from the very beginning? Majid Fatemian: Exactly. David Joy: Tell us a little bit about your beginnings at Red Ventures and what that landscape was coming into the company. Majid Fatemian: Absolutely. So when I joined Red Ventures, we were building proprietary data platform out of very basic, traditional data warehousing ecosystem. Very, very simple database. SQL Server at the time was a hot database for doing analytics, and it didn’t have that much data relatively at the time. We thought that we have big data, but in the millions of data pieces or events as we call it. But then, as we grew and Red Ventures, as the company started doing some acquisition, so they acquired CNET Media Group, they acquired Healthline, Lonely Planet, The Points Guy. There are plenty of different acquisitions that they have done along the way, but with that, the amount of data that was gathered through the ecosystem was expanding exponentially, and for being able to keep up with the volume of the data that’s being added, we had to revamp that data pipeline, data structure multiple times. So from very, very traditional ecosystem of data warehousing, we went to, now, data lakes. So data lakes was a hard thing in 2016, 2017, so we started building data lakes. Again, the scale was something that we were not aware of. So we started with building a data lake that’s capable of handling a couple of hundred million events per day, and then very quickly, we grew into a couple of billion events per day. David Joy: Wow. Majid Fatemian: So that scalability had to be pretty much pre-designed into the nature of the system, forced us to re-look at things and making things more efficient like, “What is available?” Technology, especially on the data side, is changing so fast. Every day, there’s a new tool, new technology, new best practice of doing things to the point that we have looked at things, multiple tools, multiple times, how we can efficiently leverage each of these and different parts. As you mentioned, SPARC. SPARC wasn’t there years ago, and then it came in, and the ease and ability that it provided with itself was much faster, easier data processing, easier distributed computation for all the data teams. On top of that, Databricks as a whole, as a vendor is doing a phenomenal job of improving, providing more, and more, and more functionalities to the point that we recently switched to Lakehouse, the new data warehouse offering. Yeah. David Joy: Yeah. So what database were you on? What warehouse were you on? Was it Snowflake or something before? Majid Fatemian: After migrating from SQL Server, we moved to Redshift. So we were heavy, very, very heavy on Redshift. We still have a lot of Redshift instances because of the ease of use and the familiarity of the company with Redshift as a technology. But now, we have switched, and some of our verticals, we have switched to Lakehouse and Databricks offering as well, but we still have very powerful presence of Redshift within our data warehousing. David Joy: No, I agree. I think Redshift was a big thing in 2013, 2014, and 2015. Majid Fatemian: Exactly. David Joy: They were one of those first people to build SaaS, ETL that did massive parallel processing so you could run… join query, and it would get everything fast, and you’re like, “Okay. This is great. I don’t have to put stuff on SPARC. I don’t have to manage my SPARC cluster. So this makes sense. I can use it.” But also, Snowflake came out, and they did something phenomenal, but I think Databricks is doing something interesting with the Lakehouse as well, and I keep following… I have a bunch of friends at Databricks, and every time they hit me up, and they’re like, “Hey, look at this. We have a new tool that we are coming out. This does this kind of machine learning. Now, we can do LLM.” Majid Fatemian: That’s true. They do a fantastic job, and they are pushing the boundaries. All of these companies, as you mentioned, they are doing very, very interesting stuff. Redshift itself, like AWS and Redshift, they’re coming up with a lot of tools and a lot of integrations with the Redshift. I got introduced to Redshift Streaming a couple of years ago, and the first time that I saw that piece of technology, I was like, “Where were you five years ago?” We had to build all this stuff from scratch for our use case that now is a single command statement. It takes care of everything. So this is a very healthy competition to see all these companies are trying to push the boundaries of what’s available, making it much, much easier for the end user. I think that the end user benefits from this, given we adapt ourselves to technology and the changes to the technology. David Joy: Yeah. I mean, I have thought about this previously is in this decade, I will see… and I believe that a lot more of this maturity or mature products are going to come from outside the cloud providers. One of the things that cloud providers do really well is, in my opinion, give infrastructure, and they give compute and storage, and they put it together in a way with a bunch of abstraction that you can connect with it and start using their products, but they don’t have that exact focus or an entire company focus on making Redshift the best Redshift out there. Right? Majid Fatemian: Exactly. David Joy: When it comes to Databricks, same thing happens where they’re one single company focused on providing a data experience, and that’s what the company is doing. Similarly, at Cockroach Labs, we are focused on building the best database, distributed SQL database. Of course, there are other cloud-based databases that are available, but then we as a company are focused, and that’s what allows us to kind of- Majid Fatemian: One thing. David Joy: Yeah, and I think as the decade goes, the same thing happened with streaming with Confluent doing with Kafka, Snowflake is doing their thing. So I feel like the next decade is going to be more exciting for adopters like us and especially you who are trying to put this into your data platform. A question I had, Majid, that I wanted to get into was tell us a little bit about this data platform as to who uses it. Is it like an internal use, or is it for your customers? What motivates the company to have a platform like that, and what are the business implications of this platform? Majid Fatemian: Absolutely. So this is mostly an internal data platform that we built for the verticals, the business verticals that Red Ventures operate as. So we have these different business entities, but each of these are using a central data platform. This is the central data that goes into… all the data it goes into, and that helps facilitating the data governance, the storage, the ease of distribution, and monitoring operational aspect of this. It started back in the day as a clickstream data pipeline, so only page views and who clicked on what. That’s the very, very basic analytics that was gathered. But then, there were requirements to build this data pipeline faster. So instead of having hours of latency, we got it to the point that we had data available for reporting and in the dashboards within seven minutes. So that pushed the boundaries a little bit further, and then we wanted to have reliability built into this pipeline, so we made it to the point that almost 100%… 100% is unachievable, but like here, at least six levels out of nine, 99.49 of reliability that we built into the system, including ledgers that tracks everything, make sure that data is received and persistent where it’s supposed to go to the point that we no longer only receive the clickstream data. Businesses started pushing operational data, insight data, revenue data, accounting data into the central pipeline because now it’s become the defacto place to put data into, and that pushed all the boundaries that was originally built as the intention of the data pipeline. The consumers are internal businesses, so each of these business verticals. They consume the data, and then they connect it to their own dashboarding, BI tools, different analytic tools that they leverage, and the technology itself gives… The fact that it’s built in-house, it gives the flexibility to a company to adapt to the way that we want it. David Joy: Got it. Majid Fatemian: So the governance, data privacy is very important, so we built everything as a DNA of all these things, and then the nice things that we get out of it, for example, like your customer data platform so we can build on top of it that goes into… Because we have a variety of brands, we can see the customer across all these brands. David Joy: That’s amazing. It’s always good to know how the solution you’re building is impacting the business, right? Majid Fatemian: Exactly. David Joy: So it seems like once you do the ETL, once you have all of this data put together from all these different sources, they can be then collected by different teams to come and make business decisions, and drive some business outcomes. So that’s really significant. I’m pretty sure this happens across multiple companies. There are different ways folks can get into that. So you were mentioning that you guys went from a million events to a billion events happening every day. Was it like a every day thing? Majid Fatemian: Every day. Yes. Yeah. It was maybe 60 million events per day when we started to closer to 2 billion events per day at the peak. David Joy: Wow. Majid Fatemian: Yeah. David Joy: Got it. So tell us a little bit about how the transition was for you guys to go out of SQL Server. How did you guys start thinking about making the decision to like, “Hey, we need to move to the cloud, and this makes sense?” What went into the philosophy, and inferring and making decisions around it? Yeah. Majid Fatemian: So cloud was basically a no-brainer. So Red Ventures traditionally had their own data centers and managed their own servers. But then, very quickly, towards the early… It was 2010, 2012, they started pushing stuff into the cloud and started realizing the functionality of the cloud and the ease of use. Slowly, slowly, we migrated all of the data centers to the cloud. If I’m not mistaken, it’s been many years that the whole data center is deprecated, and it was shut down, and everything is now in the cloud. We’re mostly an AWS shop through some of the acquisitions we got, some of the GCP, but majority of our stuff are on AWS. Cloud has provided the ability to… the freedom, the ease of use, the agility that we want to experiment with stuff. When we were seeing data growing, and SQL Server was not capable of keeping up with the data, and then a lot of other factors went into it with multi-tenancy. We wanted to have separation of data privacy and all the security measures. We very quickly realized, “Okay. Having a single SQL Server database, even it’s a cluster, it’s not sufficient. It’s not enough.” So data lake backed with S3 was a no-brainer for us that we have to go this route. David Joy: Right. Majid Fatemian: Then, there are tools with SPARC, Databricks, and that’s when we started using Databricks and building tools around Databricks. So we built some of our original data pipeline through Databricks as well and SPARC in particular. Then, again, it scaled so fast that we realized, “Okay. We are not efficient enough with some of these, and we have to revisit all of this.” The company’s culture is you can think everything, and we can change things fundamentally, and nothing prevents you to do that. So we really get the data team thinking about how we can make things faster, “Is the tool that we are using today, SPARC, was the best tool, and what are the other tools that are available to our usage?” We started looking into AWS native solutions, and then combined everything with a mix of SPARC plus some of the AWS native solutions, serverless solutions to make things faster, more efficient. Now, we are transitioning to the Lakehouse with Databricks and everything, but again, still a combination of a lot of AWS pieces, ECS containers, streaming services, and then for storage and data is the Lakehouse. David Joy: Yeah. No, that’s brilliant. I was curious. What did you feel was the biggest challenge for you to maintain and manage this platform? Especially in scenarios, I would say… I mean, maybe a secondary question to that is… We are very good, especially in the data platform teams I have met. We figured out a way to get the ETL then pull all of this data into a place, but then you also have to do education sometimes to teams where… I’ve had this issue in the past where we had BigQuery, and we put all of this data nicely set up, and we said to our external teams or teams, consuming teams like, “All of this data is available,” and they had no idea how to use this data. Okay? They were like, “What kind of decisions can I make? Okay. Can I make a pie chart?” or we as the data team would also become an enablement education team to guide these folks on how to go and start doing some data analysis on top of that because at least the team I work was not that mature. We didn’t have advanced data analysts or data scientists. So do you end up in situations like that? Of course, not at a high level, but you can get into some details of that. Majid Fatemian: So I’ve been in the technology long enough to know that actually, the technology is the easy part, and it’s the people aspect that it’s the most challenging part of everything. As you pointed out perfectly, David, building these new tools, building these ecosystems, pipelines to get the data from somewhere to the other is relatively easy. From technical perspective, the technology is moving so fast, and it’s becoming easier and easier every day. It’s the other end of the spectrum that now… from the business perspective, from the people who are the end users of this data. How can you get them on board with the new tools, with the new services? Especially when you’re dealing with a business that’s revenue-generating, the main question is, “Why should I get into this new tool or new system because the way that I had things was working? What’s in it for me? What’s the value add for the business?” In our effort to transition, so some of the transitions that we made was drop-in replacement. So, for example, one of the data lakes from the version one that we had was relatively slower, less scalable to the point that we had much, much improvement in processing and scalability. It was a drop-in replacement. David Joy: Right. Majid Fatemian: So we made it in a way that nothing changed. We basically pulled the carpet underneath the end user without them noticing, and that was one of the approaches. It took a very long time to achieve that because it was very difficult, but we were not ready to go through the route of pushing the business and changing the business. But with our transition to Lakehouse and new set of tools, we thought that this is a fantastic opportunity to do a fundamental change. Now, we are pushing them to, “This data is available to you much, much easier. You don’t need to have multiple UI, different applications to be able to interact with your data anymore. You can log into a single place, have everything within your reach, and explore the data, and that expands your ability to play with the data, come up with new queries, come up with your different tools and ways to get the data out.” That is the selling point to get them on board. In today’s world, I think data is becoming part of everyone’s role, so everyone should, to some extent, be familiar with how to interact with the data, how to get their data, and how to explore with the data. That’s what we are pushing, to work with basic SQL knowledge. Basic understanding of how data is organized and modeled is basically everyone’s responsibility. David Joy: Yeah. We joke about this at our company because we built a database completely from scratch, and the language we communicate with is SQL because every college in the world is still treating SQL as the language to communicate with data. So I think it’s very important. I think some basic fundamentals around running SQL queries is so critical, and I think pretty much everybody should know that, especially with the whole push for LLMs and all the other things that are happening as well. Majid Fatemian: Exactly. Yeah. David Joy: So one of the things I was curious about from what you were saying was that when you build this, the data platform, right, and when you are going through scaling this, all these different aspects, what are the source systems? So when you say stream, is data coming from APIs or from applications, or is it coming from databases, or do you have Kafka streams? Tell us a little bit about what the upstream looks like. Majid Fatemian: All of them both to answer because majority of the properties that we have are customer-facing websites like Cena.com or Bankrate.com, if you log into them. So we have a JavaScript snippet that sits on the website, and clickstream data. It starts sending those data into… like who is viewing what page to an API endpoint. So that is our ingestion endpoint which is highly reliable, scalable, and very, very available for getting that data. Then, from that point on, it’s off of the client side. Everything is on the backend. We interact with Kafka stream, so that data is put on a Kafka stream and then it gets routed into multiple destinations depending on… We’ve built a very, very strong self-service mechanism into a place that you can route every event, as we call it, every clickstream data into many locations. You want it in data lake, Lakehouse, you want it to participate in… go to register directly, or you want to send it over to external third party services if there are other external services, and these are all configurable. Then, that customer stream, having multiple consumers depending on who’s sending what type of data, then they know how to interact with that data. Again, along the same lines, the security privacy. So if they contain sensitive data, does it need to get redacted, or based on where the data goes, all the privacy rules are being enforced on that data. David Joy: Oh, that’s interesting. Yeah. So it seems like most of your data is not batched, but more real time in that sense. Majid Fatemian: More real time. Exactly. David Joy: All event-driven architecture, right? Majid Fatemian: Event-driven. Exactly. David Joy: Exactly. Well, that’s awesome. So, before we swing to… I really wanted to dig into some of the cool things you did at Ubisoft, but one of my questions was, what do you enjoy doing the most on a day-to-day basis? Is it setting up the infrastructure and getting data, pulling all of those pipelines, or it’s just analyzing the data? What are you passionate about? At least, what are you are you inclined to? Majid Fatemian: My role involves multiple things, so I still try to keep myself hands-on. I still try to do a bit of coding every day, but many of them, it involves system design, like how do we design the full flow of the data, and what really excites me on the whole data ecosystem is the beauty of a combination of software and data. So, in my opinion, there’s no separation. So data pipeline, to be efficient, it has to be scalable, reliable, available. So all this stuff that goes with the software engineering, and in my opinion, it’s a place that they go hand in hand. So you have to know a good amount of data and a good amount of software to be able to be efficient, but what really excites me as well in terms of seeing the end-to-end. So, in the past year, I was involved in the other spectrum of the product that I was part of producing for the past seven years and joined one of the verticals. Now, it really helped me to see the end-to-end. David Joy: Right. Majid Fatemian: “Wow, I wish I knew that your pain point two years ago, that I could have facilitated this,” and this is putting different pieces back together. So that really opened my eyes into, “Okay. Just focusing on developing the best product is not sufficient.” So sometimes going to the other spectrum and seeing how that product is being used is a very eyeopening experience. David Joy: Yeah. No, I agree. I’ve had similar experiences where we start doing something, and we are technologists, so we are like, “Oh my God. Look at this. I’m building something super amazing, and everybody is going to love it.” Then, we present it to people, and they’re like, “How do I use this, and do you even know what I want?” So I think those kinds of things happen all the time, but again, that’s the beauty of the world we are in right now. We get the opportunity to go back to these problems and improve it. Right? Majid Fatemian: Exactly. Yeah. David Joy: Thank you so much. I think one of the last response you gave was beautiful. I really enjoyed getting your philosophy and perspective on the way you approach things on your day-to-day basis. I was really curious when I saw somebody work at Ubisoft because I’ve never spoken to somebody who was part of a game that I used to play. I used to be so many games that are developed by Ubisoft, right? Were you part of that gaming division, or what was your role? Tell the people a little bit about some fun things you did there. Majid Fatemian: Absolutely. So my role at Ubisoft was very similar to what I have today. It’s mostly online backend services or what they call as live services. So a lot of system architecture, backend architecture, implementation, APIs which involves in leaderboards, matchmaking, multiplayer, basically, everything that goes over the wire for multiplayer or as a group setting. I have been fortunate enough to have my name as part of a couple of games like Assassin’s Creed, Rainbow Six, and Far Cry. I worked on it a little bit, and it was one of the Xbox Connect fitness game called Your Shape: Fitness that… I worked on that project too. David Joy: One of my favorite games, 2008, 2009, I don’t remember exactly, was Far Cry 2. I remember I had a GPU at the time. I had an Nvidia GPU. I played so much that it died on me, that I could go get a new one. I think it came out with the best refresh rate, and I had pumped everything up. Everything that I could push, I was pushing it, and so such a great game. I always played Assassin’s Creed. Were you a gamer while you were there? Did you like game also? I’m just curious. Majid Fatemian: I like games, and that’s basically one of the reasons that I joined the company because I like games. But then, when I joined, I realized, “Okay. I’m not a gamer.” The people that I saw there was so hardcore gamers that I considered myself, “Okay, I’m not a gamer at that extent.” Funny enough, one of my recurring annual feedback every time it was their review session was like, “You have to play more. You have to play more.” David Joy: Oh, wow. Majid Fatemian: I was like, “Okay.” David Joy: Hey, when your company encourages you to do less work and more play, that’s a good environment to be in. Yeah, but I do remember Far Cry 2, Assassin’s Creed, the first version, I really enjoyed it. After a while, it feels pretty redundant in the sense that it’s the same kind of approach, but Ubisoft did some really cool games, actually, and even still. Even now, they’re doing it. What was interesting for me when you said matchmaking and things like that, when you talk about matchmaking, and you talk about multiplayer experience, a lot has changed in the last decade. Majid Fatemian: Yes. David Joy: In gaming especially. For example, with a game like Call of Duty Multiplayer or Fortnite, these are like… You have to do matchmaking, and one of the interesting things to solve is… or even if you’re playing FIFA. I don’t know if you’ve gone into FIFA. You have to start playing the game, and I want to play this game multiplayer with somebody else. I want the game to connect me with somebody who is in US or in my region, not to somebody who’s in Europe or, say, Australia because then it’s going to add lag. When you’re programming these games and you’re making this matchmaking systems, you have to consider all those things. Right? Majid Fatemian: Exactly. David Joy: My question to you is, when you were part of the team working all those things, and this was way before where we are with e-gaming or multiplayer games, what kind of problems were you coming across? Majid Fatemian: In the gaming industry, there are a couple of challenges which are not experienced outside of the game, in my opinion. One of them is deadlines. So the deadlines are actually deadline. There is a huge marketing campaign going for a specific game, and it has to be ready by that time. Otherwise, it’s got to be a big hit for the company not to release a game on a date. With all the process, things may not have changed or may have changed since I was at the gaming industry, but the process that you go and submit a game to the first party services and the audition that it has to go, it takes time, so it has to be ready by a certain date to be able to release it for the date that the marketing campaign has pumped. So those are very tricky, and then your client is fixed. Your code is shipped within a disc or within… Now, it’s downloadable, but the nature of the code is fixed, so you cannot have the ability like a web application that you go and change something on the client relatively easily. That makes it restricting, but also interesting in terms of the challenges. So that’s one of the times. For example, again, I have not been part of the gaming industry since 2015 when I joined Red Ventures, but prior to that, it was the inability to change the client side or it’s so expensive that sometimes it isn’t worth it because you have to go through a DLC process that… It was at the time that MongoDB as a NoSQL document database was rising, and we leveraged that in a very beautiful way with the combination of traditional relational databases to be able to extend the functionality of the game. So one type of data is coming in from the old version of the game with these fixed schemas, and now we want the data to be in the same place for competitions, for matchmaking. Now, MongoDB as a NoSQL database at the time really helped us to get there with a hybrid solution of the databases. David Joy: It’s very interesting. How do you keep up with all the changes to technology? What’s your discipline around, “Hey, there’s something new going on. I need to go check it out, and learn, and keep up with this?” Because I got the opportunity to speak to some folks the other day, and I was telling them that when I was in college, I would learn something in my first year, and by the time I am finishing college, that thing is old. This is a four-year timeframe, and then somewhere in the last five, six years, you learn something, maybe in two years, it’s old. But now, everything has changed. Every six months, there is something getting obsolete. Majid Fatemian: Exactly. David Joy: So how do you go about all that, Majid? Majid Fatemian: So you have a perfect point. I think the speed that the technology changes, it’s so fast that it’s not even getting to the books. So, like previously, we got a book, and you read the book, and that’s how you keep up with the technology, but now, it’s changing so fast that the whole cycle of writing and publishing a book, and printing out, and distributing it is so slow that by the time it’s out, it might be deprecated or some parts of it might be deprecated. That’s one part of it, but then the other is now the tools, especially cloud, provides you the ability to play around with the new technology very quickly. So the hands-on experiment with the new service is significant. So if a new database has come out, if a new technology has come out, you can just spin up… AWS, for example, provides a lot of free credits for people, or Databricks does a lot of single-cluster, free that you can leverage, and those are the opportunities to play around. Playing is something. What we usually do is we try to do… First of all, keep ourselves up to date with technology, what’s out there, what’s available because in my opinion, software engineering is like you have a toolbox, and all of these technologies are different tools within your toolbox. You just need to know that you have this particular tool available. How it works, it doesn’t really matter, but what problem it solves is the most important thing. When the time comes for you to use that tool, you will go and learn the details and nitty-gritty of that particular tool, but the fact that it exists, right? As an example, graph databases. We know what is a graph database, what problem, what kind of problem it solves. That’s sufficient. When you see, for example, an identity graph, you realize, “Okay. This could be a good potential for a graph database because you’re shipping an identity graph.” So that type of approach, and then doing a small proof of concept, the POC within the service. Timebox? Yeah, I want to dedicate this two-weeks timeframe for this particular technology to see how it performs and behaves compared to what we already have, and that gives us a very good understanding of evaluation of the technology and basically, being open-minded that just because something is a shiny new tool, it doesn’t necessarily mean to be the best. David Joy: Exactly. Yeah. Majid Fatemian: So we need to evaluate it, and one thing… Although the technology is changing very, very fast, but the basics and the fundamentals are consistent. So the fundamentals of computer science, distributed computing, all those are… They stay the same. Yes, the high-level abstraction changes, but the underneath and at the core, they stay the same. David Joy: Yeah. It’s well-said, actually. For anybody who’s listening, this is Majid’s routine to keep up with what’s going on. So if you are trying to figure out how to do the same thing, this is sound advice. So spoke like a truly experienced person here, Majid. Majid Fatemian: Appreciate it. David Joy: I just wanted to say for people who are listening, if they want to follow you, do you put something together? Do you write, or are you active on LinkedIn? Where do people get to see all the awesomeness? Majid Fatemian: I appreciate it, David. Thank you. That’s very kind of you. Yes, I do have a personal blog, majidfn.com, and this is also my Twitter handle or X as we call it today, majidfn, and I’m also available on LinkedIn. David Joy: Very cool. I believe if anybody wants to talk to you real time, then that’s probably X or LinkedIn, and then blogs where you get to write stuff. Majid, I wanted to say this was such an absolutely fun conversation to get into with you. You’ve been an absolute awesome guest to be here on the Big Ideas in App Architecture Podcast. Thank you so much for speaking so candidly, and I hope you had a lot of fun doing this podcast. Majid Fatemian: I appreciate it. Thank you. I really appreciate having me here. It was a pleasure talking to you too. David Joy: All right. Thank you. Thank you everyone for listening, and we’ll catch you in the next one.

Big Ideas in App Architecture

A podcast for architects and engineers who are building modern, data-intensive applications and systems. In each weekly episode, an innovator joins host David Joy to share useful insights from their experiences building reliable, scalable, maintainable systems.

David Joy

David Joy

Host, Big Ideas in App Architecture

Cockroach Labs

Latest episodes