Pybites Podcast

#137 - Sentry, a Developer's Partner, Interview with Co-Founder David Cramer

Julian Sequeira & Bob Belderbos

This week PDM coaches Hugh + Ryan talk with David Cramer, Co-founder and CTO of Sentry!

They delve into the journey of Sentry and its rise as an essential tool for developers.

David highlights how Sentry's developer-first approach significantly contributed to its growth.

They touch upon community-centric decisions and the emphasis Sentry places on giving back.

The discussion also ventures into the synergy between Sentry and Python and the attributes they prioritize when hiring.

This episode is a treasure trove of insights for anyone in the tech industry.

Chapters
00:00 Introduction
03:00 Wins of week
06:32 What is Sentry?
09:37 Growth of Sentry + developer centered
14:20 Top down decisions + giving back
18:24 Industry events and branding
21:18 Sentry and Python synergies
24:50 Htmx developments and Python features
27:19 Attracting talent in Sentry
31:10 Valuable attributes of people you hire
34:43 Pairing app and infrastructure metrics
41:06 "Blade runner concept" to debugging production system
41:46 Key message / insight / final thought for audience
45:48 What do you do in your free time?
47:20 Books & videos tips: Cal Newport + YT construction content
49:48 Wrap up and resources
50:36 Outro music

Links and resources:
- Check out Sentry here
- David Cramer on LinkedIn
- Factorio game
- Blade runner movie

We're north of 50,000 customers. And so when you just think about the sheer scale of how many companies and developers we've been able to help, it's vastly different than most companies. And so that, to me, is almost like the most important decision we ever made at the company. Hello, and welcome to the Pibytes podcast, where we talk about Python career and mindset. We're your hosts. I'm Julian Sequeira. And I am Bob Valdebos. If you're looking to improve your python, your career, and learn the mindset for success, this is the podcast for you. Let's get started. David, welcome to the Pie Bytes podcast. We're very happy to have you here. And, yeah, you met one of our PDM coaches, Robin, in Berlin. Yeah, great to be here. I met him at. We were doing a fast API event with some folks, and he was there. And, you know, we both have a lot of hair right now, so people with hair flock together, I guess. Fantastic. Well, let me introduce myself. I'm Hugh. I'm one of the PDM coaches here. I actually am a PDM graduate in pibytes, and for those who aren't sure of the acronym PDM, it's the Pibytes developer mindset program, where we mentor people through an entire Python project from beginning to end. Ryan. Yeah, and everyone. Ryan Austin here. And also a PDM graduate and PDM coach, I guess. A little bit about me. I develop a SaaS app, and I work in Django and Python daily. Terrific. And, David, what do I do these days? I am a software developer by trade. I started a company called Century a long, long time ago. Done a lot of open source these days. I do. I don't know, strategy. We're not a small organization anymore, so I don't really get to write code for the company, but I'm still very hands on in my spare time trying to keep up with technology. Unfortunately, I'm mostly writing JavaScript these days. I read that somewhere. I think I heard you say it, and I was like, oh, crush Python. But we're happy that you started with Python and Django, because it seems like you made it a really, really integral part of sentry. Yeah, absolutely. Our entire backend is still, at least for the most part, it's still Django and Python. And honestly, I love Django. I think the challenge these days is just you want the rich UI experience, and it's a pain to set up on any of the traditional frameworks just because then all of a sudden, you're running multiple services, and you got to deal with compilers and all these other things. And so these days I would say I'm giving it a good hard try to use typescript and whatnot on backends, but it is not nearly as easy as just spinning up a Django project. Right, right. You know what, you know, what's traditional here for us at PDM and podcasts and all of our calls is asking about wins. So maybe, I don't know. Hugh, you want to go first and take it away with some wins? What are your wins of the week? Well, on the python side, I've been actually spending time learning about a lot of the python modules for data science and manipulating data, and I'm hoping to apply it to my current day job, which is doing Devopsy type work. And I'm trying to see how I could use Python and data science to analyze infrastructure metrics. I got quite a bit of that under my belt this week, so pretty happy about it. Sweet, man. I'll take it next and then ask you, David. But so this week I'm traveling. I attended a conference, d conference and made some really good connections there, met some really fantastic people. D three is it's fintech conference and crypto and everything happening there blockchain. So it's just interesting to see because, you know, here in SaaS web, I guess, web two world, we're currently doing a lot of useful things and that's evolving every day. And I was interested in seeing where web three was going because it seemed to have or crypto seems to be in a bit of a winter right now. I don't know if any of you are following anything crypto, but I guess the win of the week was just taking part in the hallway track at that conference and just meeting the people and just putting myself out there meeting the people that I met. I think that was fantastic and starting some really interesting conversations. What about you, David? I feel like I'm still working towards my win of the week. I've been messing around with a side project again, JavaScript, and I'm trying to spin up a cluster on Google Cloud right now just because I've had some challenges getting, frankly, traditional web services just running seamlessly on other providers. I don't know why this is so hard, to be quite honest with you. It's not like things have changed so much in the world, but I'm very close. I was up late last night, I'm so close. And it's mostly I'm just setting up like Kubernetes clusters and stuff to run what is the lowest traffic node service of all time. But hopefully by the end of the day I will have a nice win with that. So you know what, I have a bit of a crude joke. You know, you did say JavaScript and a lot of too hard, lots of mentions of too hard, too difficult, lots going on kind of there. I don't know if it has any correlation, but. Just kidding. It is a little bit. I think it's one of those things where like I try to fill my days with like a mix of stuff. And so I'm very much, you know, attractive computers because I can build things, not because I like managing people. And so sort of spending the time just like hacking away at something. And to some degree, brute forcing technology is, some days it feels like therapy, other days it's just very frustrating. But, you know, so it's a good way to balance things out, but it does create a lot more, I guess, stuff and context in my life. Yeah, yeah. I want to get back there because I have something about that that I want to ask you. But I guess for listeners, particularly in the python, that might not be familiar with sentry, because you mentioned it as a kind of sidebar, like a long time ago I started sentry, but sentry is, I kid you not, we work with PDM clients, clients, python clients all the time. And one of the first things that they want to do when they go to production, or even before they go to production is get sentry. And it's like it's become a de facto standard. So why don't you maybe explain to our listeners, like, what is sentry and what are the primary reasons that developers might benefit from using it? Yeah. So sentry is great for the python community because that's what inspired it. It's like the Django debug page brought to production. It's the best way I can describe it. And that's actually very much what the intent was for a while. And so traditionally you ship some code to production. You have logs somewhere. If you've ever dealt with looking through logs, it's not fun. Half the time they don't even have what you want in there. And then finding the thing you need is complicated. And so what we did is we, we built this little client that would run inside of your application and it would create this rich debug report, effectively like the Django debug page. So you'd have not just the error, but you'd have some source code information around the stack, trace you'd have stack locals in Python, which is a really big deal for debugging, and you'd have simple things like the HTTP request and some other context. And we do that in production. We capture those, what you might think of as a crash dump, but for web applications, then we bring it up to a dashboard where we aggregate it and make it a lot more actionable. The main thing there is we just deduplicate them. So you at a glance can very quickly see, oh, this is what's happening a lot. It's probably important. And that's what we did. Like, what year is it? 15 years ago with the open source project. And then for the past decade plus, it's really been focused on bringing that technology to more developers. So different programming languages, different types of runtime. I said web apps, but we've also spent time bringing that tech to, say, mobile apps and desktop applications and to some degree, embedded systems, and we do more than that these days. But fundamentally, sentry is this production monitoring tool that's really trying to give you an experience that helps you debug the problem versus just giving you a bunch of graphs and numbers and logs like you're typically used to on monitoring providers. And so we're pretty far along, but we're also still pretty early. And I think one of the interest, the most interesting things about sentry is at its core, it's an open source project. There's a big asterisk on that as we use a special license these days that we call eventually open source, but it was started as an open source project. A lot of what we do is still liberal, permissive open source. What it means in practice is actually the century service that you see that we charge for. You can actually just go spin up yourself. And the only caveat is you can't sell it as a software service. Done that for a long time, but pretty stable these days and phenomenally good in the python world. Not all languages are as great as Python, unfortunately. So we can't quite give the same amazing debug experience in some of the other runtimes. Talking a little bit more about sentry over time, Sentry has witnessed some pretty remarkable growth over the years. Can you highlight some of the pivotal decisions that contributed to the success you have now? Yeah, I think we've actually tried to codify some of this. Sentry is not a common business model. So most companies in our space are enterprise. Historically, they're just proprietary software. You charge lots of money, you find hundreds of customers, you make a good business century is backwards of all of those. Just because we wanted to be different, we wanted to do things a certain way. And so the open source thing from the get go was already new, especially at the time when we were starting the business. But when we sat down to like codify things, the first thing, and I think the most important thing that's made us successful is we really focus on this market share aspect. So we treat century less like an enterprise company, more like e commerce or consumer in the sense of like how do we acquire a larger audience versus how do we charge more money? And that's a very different thing than almost any business you'll find in the space. A good example of how that applies itself in practice is we had Python, we built the Django stuff, it was doing pretty well, and then we're like, okay, we want more customers, where are the most more customers we can find? Back in the day it was like, well, everybody uses jquery, so it's like let's build a JavaScript thing. We got to support JavaScript because these jQuery, that means everybody in the world can probably use sentry. A little bit has changed in the industry since jquery days, which was fortunate for us. But that single decision, as an example, 60% of our customers use JavaScript. They might use Python or something else in general, but that's huge for the business. And what I mean by that is we have something like, we're north of 50,000 customers. And so when you just think about the sheer scale of how many companies and developers we've been able to help, it's vastly different than most companies. And so that to me is almost like the most important decision we ever made at the company. And we think about that still daily, like, okay, where new audiences of developers that we can help, a good example today would be like, there's a lot of this ML stuff going on in the industry and we're like, we've kind of ignored it. There's obviously hype waves. I don't know that this is entirely hype driven anymore, but we had to step back and we're like, oh, but those developers building those systems actually have different problems than just production monitoring of a normal web application. And so we're like oh, we should probably build something there just as an example of like where are lots of developers with large needs? I think beyond that, the only other decision I think that's critical for century and I actually think more people should think about this. Generally speaking, as you build a business, it gets lost in executives that don't understand the audience. We say developers are a customer and nobody else. And that's super, super important from a focus point of view, which I strongly believe in. And what I mean by that is like, we built the air monitoring product. Obviously developers were a customer. Our next product was performance monitoring. It wasn't, I don't know, reports or analytics that might be a different Persona in the company or might be for the developer's boss or something like that. Right? And to me it just seems like common sense, like a no brainer, but sort of those two focal points of like, how do we help more developers? And the developer is our only customer. So if we invest the dollar, the dollar goes towards that developer. Those are the two focus statements that we've really held true that have helped us guide the product and effectively, century is a product led company. You know, David, I think you encompass so well. I believe some of the sentiments that the development community, if I'm speaking very broadly here or speaking at least for myself, or what I hear from my developer peers, what we typically feel about century as an entrepreneur, the typical paradox is you want to do this thing because you had this dream or you had this problem and you want to solve it for yourself, right? But then as the business develops, the business grows, you get investors or whatever have you, you know, people are trying to figure out how do you scale this thing? And, but somehow you've really told the line with sentry, with being so developer centric and developer focused. And I think we want to give you like, kudos for that. That's been amazing. And it really leads me into my next question, because I've observed that Sentry has been diversifying its involvement from supporting open source projects to hosting podcasts. And in fact, I saw a tweet yesterday that I could have written just based on how tightly I've been following Sentry and the tweet is. I'm going to just credit the tweet author. It's a guy named Aiden Bai, and he says sentry single handedly solving error reporting, perf monitoring and open source funding at the same time. What in the world is going on there, man? Whose decisions are that? That are those? And is it a company wide thing? And it kind of speaks to what you just said, but tell us about that. Yeah, I think, well, first and foremost, I think the way to think about companies is they are almost always top down. Like in the sense of culture, starts at the top. It's very rarely a bottom up. And I think Twitter might be an exception to this where it was a very big, like, internal culture that was pushing things, but top down allowed that in support. And I think that's really important. So things like the open source funding and it's always been near and dear to us, like that market share thing is effective because we're open source. Like early days, we had this motto that if we can't make money, nobody will. That was the way we approached the business, right, because we gave away the server for free. And so when we thought a lot about this, honestly, these things are not hard. I just don't think people have the courage to take these steps. The money we donate each year to projects. So we do a bunch of open source, and one of our guys at the company is going to write up more about this because we just to educate more people. And frankly, we want to shame people until they start doing it. But we do foundations. First off, we give away sentry. We give away the server for free. We actually host lots of open source projects on it. Then we give a lot of money to foundations that are important to us each year. Now we're giving increasingly large amounts of money to individual developers. I don't know if this is true, but it's very possible that sentry actually gives away funds to the largest number of developers out of any other company in the world. It would not surprise me if that's true, because it's novel what we're doing right now. We're working with new companies that are trying to make it easy. But that latter point, we basically just decided, let's commit. I don't want to say a percentage because it's not quite a percentage. And I also don't want anybody thinking they can extrapolate our revenue, but something equivalent to a percentage of revenue. We're like, let's just allocate that towards open source every single year forever. That's a top down decision. Now that could change. You know, maybe there's a day I'm not in charge of certain things, and technically I'm not in charge, but I'm very aligned with our CEO. And so I would say at the top, we believe in this at the top. We also think it's a no brainer and that ROI is dumb, simple, and it also costs us no money. Like somebody made a point that we had, we've raised like $200 million, which is true. We spend some of that money, to be fair, we employ a lot of people and people are unfortunately not that cheap. And so it's not like, we just have that money that we can just throw this problem, but we make a lot of money and there's no reason we can't do this on a continuous basis if we allocate that budget appropriately. Right. And that, to me, again, like I said, it's such a no brainer. It's really easy. And so I think there's that side that we're pushing. We're also pushing. We're going to talk about this probably in a month. We're pushing the licensing angle of like sort of sustainability from a protecting developers and allowing them to sort of fund their own projects. Because at the end of the day, most open source is driven by one person. Like, most projects are driven by one person. There are some, obviously, where that's not true. But if you look at so many things in the industry that we rely on, it's like a one person project and they just. They're very passionate about it, right. And we want to make sure we can help them. And so we're working on the licensing angle. I will say, Aidan, sweet. It's like, I don't think we've solved performance monitoring yet. I actually think we have a long ways to go on our product. Air monitoring, known quantity, we're great at it can always be better. Open source funding. We'll solve it when we get everybody else to jump on this bandwagon, I guess. But I do think we're making the right moves. So I don't know, it's fun, but it is driven a lot by like, what, what a lot of us at century care about, especially the early team and at the leadership level. So, yeah, I can just imagine the diversity of developers in the room at the first century event. Like major century event, industry event. Any of those you guys been thinking about like industry events? I don't know. We do a lot of events and stuff, but it's kind of complicated because, like I said, we're a large organization and so it's hard to understand and hard to manage the number of moving parts. And so, like, we obviously are a lot of like, technical conferences, for example, and they're great, but I would say 90% of the time we treat it like marketing versus other efforts. And so one thing we're trying to get back to is like, treating them as like, places that conversations, maybe not all of them, like reinvent, who cares, whatever. It's a giant money pit. But I have a lot of great memories of early days of things like Djangocon and Python and especially Europython. Where they're a little bit smaller. And it was just a good way to interact with people and like try to solve problems. It actually, a lot of centuries development came from being inspired at those events. And so we're doing a lot of that. And we have, like you mentioned, we're in a lot of businesses. We also have the syntax podcast. And so we're actually thinking about, like, can we do and ignore the podcast part, frankly, it's more about the brand. And we're like, what can we do around that? And we're starting to think about can we do live events around that to bring communities together and things like that. And so I think events and whatnot are really important. But the world's gotten quite strange since COVID as well. So it's. I think it is returning to normal. For what it's worth, I was driving to sf the other day, and let me tell you that bridge traffic is unlike anything I've ever seen before COVID so. But there's still a lot of conferences and events have become digital, which frankly, it's just less good. It's not the same thing. And so I don't know. We'll see. All right, so you know what? As a Django developer, right. I want to bring it home for some of our Django developers in our community. I personally find sentry indispensable. On a good day, no alerts mean everything's running smoothly. But when things go smooth, centuries trace back, and contextual details like, are crucial for diagnosis. In fact, coming from a non traditional background, self taught and just solving an issue, solving a problem that I had. So coming from that non traditional background, sentry was my go to. And then as I speak to my fellow friends and developers, they asked me, what are you using for logging? And my question is always like, sentry? And they're like, no, no, but paper logging and logging, I use sentry. Like, what else do I need? And on getting into it and getting into it with fellow developers, I'm trying to figure out why or what else do I need other than sentry? It's giving me the trace back. It's giving me everything I need associated with that, the full context. But I guess to ask a very specific question, can you delve into the synergy between Sentry and django? Yeah. And maybe even speak about how century, or if there are any plans to evolve the django synergy and python for Python. Sorry. For python developers in the future? Yeah, absolutely. So Python is super important to us, though. I actually think my job frankly, I feel like half the days is coming up with bold ideas that otherwise people would be like, that's crazy. One thing I've thought about lately is, can we fund efforts to bring python more mainstream again? Because honestly, over the years it's become more and more data science focused, and it was always data science focused, but it's kind of. And Ruby is similar for Westworld, minus the data science. They've kind of struggled to keep up with mainstream adoption on web. Some people still use it, of course, but the growth is kind of more plateaued versus JavaScript rocketship. And I don't think that's great, because I think Python in a lot of ways is phenomenal. And even things like Django, the amount of boilerplate you have to write in other languages compared to just using Django for backend is insane. Or even if you choose a smaller framework like flask or fast API, we've thought about, can we help fund that? But even from a point of view of how do we make it more functional? Python, especially because we use it, is always one of our most important languages. Um, and I think the nice thing is, uh, it doesn't really change. Like, Python's evolved a little bit and I think it's made our lives a little bit more difficult, like mostly with the async patterns. Um, and we'll see what this conversation about the gil does. But, um. But for the most part, like the stuff we built literally 15 years ago, it's not that much different these days. Like, a lot of that same code, it's probably been refactored, but it's the same code at the end of the day, and the, the heuristics around it are the same, and all these other things, which is actually really nice, and it's really telling about how stable Python is. Cannot say the same about JavaScript changes every day of the week. I think it's really good, but I do think there's this challenge that the Python community and the Django community has, that it has to get back in the mainstream light. It has to get back to a point where people want to use it with these modern UI frameworks, whatever those look like, because they're changing a lot. And again, I don't know what that is, because century setup is unbelievably complicated. It's not like we just have some drop in stuff that makes it easy to use Django with, say, react or something like that. It's like, no, we have the most complicated Django project you've probably ever seen as well as probably the most complicated react application you've ever seen. We are very invested in that. I can't say there's anything particular we're doing, but I am very keen to see if there are ways we can push the python ecosystem forward. And I just don't know, because some of it people also have to believe in that. People have to want that, I think some people do, for what it's worth. But that's very different than, oh, I'm just going to build an API service with a python application. It's like, no, you need to be able to build. Actually, I'm not a big fan of DHH, and I'm not a Ruby fan at all, but he said something in this talk the other day, which is like, rail should be a one person framework, and I actually like that idea of like, yeah, I should be able to build a business online still, because I always used to be able to, I should still be able to do that as a single person that just has enough experience to figure out all the moving pieces. Right? And you can do that with Django, but if you want that rich UI experience, it's still, it's difficult these days. And you know, part of it's a JavaScript community to blame. But so that's kind of how I think about things. But you know, we are very invested in all of these technologies. Fellow python developers and django developers, they get upset if we didn't ask then if you were following any of the development of HTMX. When you speak about react, et cetera, lots going on there. Yeah, I think HTMX is interesting. I have not used it, I will say I prefer explicit. React has a lot of flaws, and I think they've made it more complicated in various ways over the years. But it's got a clear thing. You can see what's happening. There's a fuzziness to that where it'll do render loops over and over and you don't quite understand why. But if you ignore that the concept of it is clear, you pass data into a component and it re renders the component when that data changes, it's very easy to reason about. I think when you get into things like HTMX, and certainly a bunch of these other JavaScript frameworks are way more of this, it becomes a little bit more difficult to reason about, like where do I find the code that's doing this thing and how does it quite work? And I'm not a big fan of those kinds of abstractions. I think they are they can be great. But I'm very much on the keep it simple. I want to open a code base and be able to know what's going on. This is why I love Python. It's why I don't necessarily love async in Python and some of the other things. But I love Python because I don't ever read manuals. I just pick up something and I assume I can figure it out. That's how things should be. There's no reason to overcomplicate things, or at least for the most part. I'll give you an analogy, rust. You cannot just open the code and figure it out because you will not succeed. You might get it to compile, but it will probably not be correct in a lot of ways. There's just some concepts that have to exist in that runtime to do things like memory management. Those are important trade offs. But we're not using rust to build web apps because it just wouldn't make sense. It would be too slow, and we just don't need that in most cases. But python, very easy to read, very easy to reason about, a lot easier than pretty much any other language out there. But we've not necessarily kept it that way. I do wonder what that's going to mean. I'm also not a big fan of typing in python. I just don't think it's needed, and I don't think it actually adds any value because we could already do type ahead and all these other things without explicit typing. Um, and it also makes it harder to reason about the code. It makes it harder for somebody to jump into a code base, somebody to jump into python, um, and you literally never used to have to learn it. It was basically like English, you know, and, and that's a such a valuable thing, and I hope we don't lose it. So I'd like to, uh, switch gears, um, a little bit. And, um, you talked earlier a little bit about budgeting and how people can be a bit expensive, uh, when it comes to expanding the team at century. What's your approach to attracting talent and recruiting developers? It's a good question. I don't know. I think just hope they show up. So one of the cool things about sentry, because it is so developer focused, everything compounds, which is really interesting. When we speak to developers, they are our customers, they are our advocates, because Dev Rel, for us is just the core customer base. It's not like, oh, the people may be doing the implementation, but we have a different customer. So it covers both of those they are the people we recruit. And engineering is the largest headcount, essentially the largest spend we have. It makes this nice compounding synergy effect where you can just target that audience and hopefully do the right things. And they will be interested in century. And I think some of those things that we do, like the open source thing, does help bring in people. And what I will tell you is my general belief, all of its marketing attracting new hires is marketing. Attracting new customers is marketing. And I'm a very big believer in brand marketing. Like I imagine if I say, do you know what liquid death is? You will say yes. But if anybody doesn't, it's canned water. That has the most ludicrous growth you've ever seen. And there's nothing special about this canned water. It's got a cool graphic on it. That's it. And it is unbelievably successful. Like they, they grew so fast and there is no reason for it other than brand marketing. It's like this coolness factor and all this other stuff. And I'm a big believer in that. You see this with Apple too. It's all brand marketing. Porsche brand marketing, beats by Dre. That was all brand, right? Like, and so I'm a big believer in that model. And so syntax to me is brand. We used to have some billboards that literally just said the most meaningless stuff. All brand. And that's how I get eyeballs. That's awareness for me. Right? And then I think beyond that, we look like a FAANG company to some degree. We hope we get the right talent. We hope we can evaluate them. Well, it's not an easy problem, I'll tell you. I don't think anybody's really nailed this. Some of the biggest companies in the world do a phenomenal job, much better than us. Facebook's attracted some of the greatest talent of our generation and they've empowered that talent to do great things. Even if you look at the open source angle, a lot of great stuff's come out of Facebook that actually enables us like react right or prettier or some of these other projects. I don't think we hold a candle to companies like Facebook, but otherwise, I think we're pretty status quo. And we try to bring in folks that are similar. And I think the great thing about the current economy and the size of the century is we actually don't hire that much these days. Mostly because more people translates to effectively more inefficiency. You hit these properties of scale and so if we hired 100 new people, we're not going to get the productivity or to be able to ship as much as 100 people could possibly ship independently. Right. And so we've slowed down a little bit recently just so we can kind of dial things in. And we've also spent a lot of time, and I think the most important thing from centuries hiring point of view is like, we spent a ton of time on new grads and recruiting out of universities and things like that. And part of that is just like, you need senior talent for sure, but you also need people that are just hungry and passionate and want to learn and frankly, are like, they're just not aware yet. And so they will mess things up. And sometimes messing things up is a valuable thing. It creates chaos. It tries new ideas that you might otherwise be risk averse to and stuff. And so we've done a lot of new grad recruiting. I think something like it might be north of 15% of headcount is new grads this century beyond what you see on a person's resume? What are some intangible attributes of new hires that elevate your team? Things like mindset or other things. Yeah. So that's always a hard thing, I will tell you. When you get beyond the technical skills, it's really hard to evaluate people, at least for me, I've never been great at it. I would say what we look for, what things that would stand out on somebody's profile for me is, like, I'm looking to see if they are interested, like, interested in technology. And I think a lot of people would say they are. But what I'm looking for is, like, are they interested to the point of view where, like, this is their hobby? And that's because that's, like, the greatest people I know in the industry. Like, they are in the industry not because it was a lucrative salary back in the day, because it wasn't. They are in there because they actually genuinely enjoy software and they're just interested in building software and things like that. So I'm always looking for that. And there's a lot of ways you find those signals. And to be clear, I'm not saying we only hire those people, but that's the outlier for me. And that's like the prime candidate. We find those signals via community interactions. So it might be, you know, if we can, if we can look into their mind and see what is on their mind, which sometimes you can do via Twitter, for example. And so if we get somebody that applies that we know them and we know them from community interactions, whether it's on Twitter, at conferences or something else, like a speaker. Okay, that's a known quantity thing. Another strong signal source we'll look at is things like GitHub. And now, not everybody can really contribute to GitHub for their day job, but often people that are really interested will find a way to be like, oh, can I open source this one library and work on that? I'm really interested in that. Or, you know, I will poke on this in my spare time once in a while, or I build this thing because it's, it's useful to me in my personal life. And those people are always, like, showcasing that sort of interest that we're looking for. And those are probably the two strongest sources. And what I will tell you, though, is, like, there's a difference in how we would look for talent from a new grad angle versus, like, post new grad, because there's just very different characteristics. And frankly, most new grads, they've not been able to do much other than have some internships, you know, but I think those are the most important. And then beyond that, it just what I look for in a technical evaluation, which I don't do them at sentry anymore, but if I did, I'm all about problem solving. And so I'm a big believer in whiteboard interviews, not because, or rather, I'm a big believer because I believe accuracy of the code or whatever you're doing is actually irrelevant. You can figure that out. It's syntax, right? Like, our editors fix it for us all the time, but it's like, let's throw you into an unknown problem and see how you reason about it. And this coincides with, and frankly, usually these are like, more senior engineering interviews, and it coincides with one of the most important skill sets that a senior engineer has. And that's debugging. Right. It's, there's an incident, there's a really big problem. How do you work through it? And that is an extremely important skill set. And every developer I know that is great has mastered that skillset. Nothing else actually matters. It's just their ability to dive into almost a foreign problem and be able to figure it out. And you can get that when you give them an interesting and hard, it's got to be hard technical problem, and you work through it with them. Right. So I'm just looking like, how do they handle that situation? Can they reason about it? Can they problem solve really well? And arguably, that's, like, the majority of the signal you need. You sometimes need signal? Like, have they ever written python before? Because I don't want to spend six months, you know, of them figuring it out. But that's much easier to get. I'd like to ask you a little bit about something that's a little closer to my heart. My own career has been in DevOps a lot and site reliability. And I know your main offering is application metrics. How might this offering pair with, let's say, infrastructure metrics to do event correlation? I was talking earlier about how my win for this week was trying to figure out how to do that on my own with Python. But how might sentry pair with an effort like that? We have this belief, this high level belief that people that use sentry don't run servers, even internally. We say SRE is not our customer. What I mean by that is, if you look at the state of the industry, what we all want. So we used to have data centers. Data centers are now the cloud. Data centers are almost exclusively gone. We outsource that entirely to other providers. Ten years ago, people might not have thought that would have been absolutely true, but it more or less is absolutely true, unless you're like bank of America. Even bank of America uses the cloud now. So that's the lowest layers is the way I think about data centers. It's now the cloud because we don't want to manage physical servers. Why would we care? It's like a hobby. If we do it more than it is a need for our business. The next layer is the platform. My belief is the platform gets fully outsourced over time too. What I mean by that is passes out like Vercel, Heroku, all that outsourced the platform. We buy all these SaaS services that outsource. It's great. There's a cost challenge sometimes, but that's not necessarily what I mean. Essentially, we outsource the platform for most teams where it's outsourced to our platform team. We're not physically managing those servers even in the cloud. We're not tweaking them. We're not adding memory, we're not adding cpu or any of these other things. We have a constrained unit that we work with. Take that now think about serverless, same thing. Think about mobile desktop browser, same thing. You don't control the device. We build sentry with the concept that you don't control the device. If you don't control the device, you have to understand, what can you change? What I mean by that is, I have no reason to monitor disk space because I can't add disk space. I can only change the application code. That's my constraint. I have no reason to monitor memory, but generically, because I can't add memory. Memory only matters when it's affecting my application code. And if it's affecting my application code, I need to know. But the change I can make is the application, not the server. That's how we think about the constraints of century. When I say that, take metrics as an example. We're actually building a metrics product. We'll probably release it in the next, before the end of the year. But it's not a generic metrics product. It's not like a Datadog or anything else. You can't just send us random counters because we just don't care about them. Because we don't want to paint a bunch of cpu charts for servers, because we don't have servers. Or at least that's the illusion we create. But what is interesting is, say you have an application that has problems when memory is high. You do need to know about memory usage. But what you need to know is, okay, memory usage is high, and I'm having problems. Find me basically traces of high memory usage. You need to be able to debug the problem, right? And so one thing we've done in our model is every metric is associated with a trace, so you can always back ref it. And so the idea is, it's almost like, what is that movie with Harrison Ford? Do you know the old movie with Harrison Ford where he's like, enhance, enhance with like a camera? I try to give this analogy, and I always mess it up. It doesn't matter. The idea is, I've got a chart, right? It's a 10,000 foot view. It says like, error rate. Error rate's a really good example. I see a spike in error rate. I want to zoom in on that spike and know what that error is. Right? I don't want to just know the error rate's high. I want to. You actually tell me what's wrong. And that is the simplistic workflow of debugging production. And so cpu is high. Why is cpu high? Show me traces that are using a lot of cpu, or maybe behaviors that have increased that have created the cpu. That's how we think about things and how we're thinking about metrics. Going back to Ryan's earlier statement on logs, it's the same thing there. We could do logs and metrics, but it's just not interesting. If somebody asked me, what do I use for logging? I don't, I don't need logs in any of my random projects because all I care about is when things break, and when things break, sentry has the errors. I only need logs at sentry because we have compliance and security controls and we need to audit a year old login history or something if there's any kind of signs of a breach. That's very different, though, and it's the same thing for metrics. I can have infinite metrics, but what am I going to do with them? They're not going to help me debug the problem sometimes, and I still have to go do something else, so why do I have them? And so that's really, it's a very strongly opinion way of thinking about things, but that's how I think about it. And so full circle when I think SRE and stuff, our goal is not really to enable the SRE or the platform developer. We think there's a lot of great companies that do that already today, whether you agree with their price tag or not is almost unrelated. But our goal is to lessen the need for the SRE to own the application concerns. Or if you are primarily building the application, why do you need those SRE toolings? And that's really what I want to get to. I want more independence for the software developer to be able to build and fix problems in their application that they themselves have almost certainly created. And I just think a lot of the historical tools are not set up to really focus on that problem. Yeah, I like that perspective looking, not necessarily looking from the outside in and infrastructure, but sort of from the application perspective of what resources is the application using and why. And then, and then I take a look at it from the history of the tracing and making sure that you have the right tracing in place so that you can make these determinations. I think that's great. And Ryan just looked up that movie. He believes it's Blade Runner. Blade Runner, that's the one. Yeah, I always think about that enhanced story, even though it's kind of janky. I used to say minority report because he's got some weird thing where he's moving around screens on the computer and it just, frankly, it just looks more cool. But yeah, the Blade Runner concept is how I think about debugging production systems. And that's the narrative I always tell people internally is like, it has to start from that high. You have to be able to go from the high level view and drill in until you actually see the exact problem. And drilling in is very different than trying to correlate anomalies and random stuff together, right? Yeah. So hot take. You heard it here first. This new tool by sentry is going to be called Sentry blade. That's great. Inspired by Monty Python. Oh man, David, this has been great. I want to be mindful of the time, just want to ask a final sort of wrap up question. I'm curious as to, as we wrap up here, what key message or insight do you wanna leave the audience with? Any kind of final thoughts you have? Here's a question for you, let me cater that. Who do you think is the primary listener? Is it people newer to the industry or newer to Python? No, I think it's, for the most part, it's people that listen and developing in Python all the time and looking for more Python podcasts. So what I would encourage everybody in the industry to do is broaden your horizon. Try new technology all the time. Sometimes it's hard at work because you can't get them to let you use a new thing. Who cares? Whatever. Sneak it in there or something. And I don't mean rewrite your application from one framework to another, that's not a good spend of time. But find reasons to try new technology and new ways of technology. Working async versus non async is just a good example of one of those. You're often building a new web service. You can try learning, say fast API and try Async code, because this is the issue I see in the industry is not enough. People are doing that these days, and there's an argument that we don't need to. There's an argument that technology is becoming so widespread you actually need people that just operate without the breath. But I'm not so much a believer of that. I think it's more important that people understand a lot of these principles and how systems work and what they can do, what their options are, and you don't understand options if you've not used them. And that to me is one of the most important things. It's also why I love startups, for what it's worth, because you get exposed to all these things, you try lots of stuff, half of it breaks, but it's okay. And JavaScript's the same thing. If you haven't even tried building a JavaScript front end with your python backend, try it, it's really interesting. Try it with probably like Vite, vite and react, or something very simplistic versus remix or next js there. There's a lot of really complicated stuff that you will quickly be underwater but there's a lot of really interesting ideas, and not all of them are great. Not all of them are going to survive, that's for sure. But that exposure to, like, new tech and new ideas is really good. And so maybe it's not on the front end, maybe that's not your thing. But another good example would be, like, you can actually link to rust in Python. So if you're doing something that you need to squeeze performance or memory out of, you could build a little rust library that interacts with Python. We actually do this at Sentry quite a lot. Now to talk to my performance. There's all these ways you can expose yourself to new things that will give you just a little bit more of whatever maybe fascinates you or interests you, and it will just continue to expose you to things. And I think just exposure to ideas and concepts is so important these days because so many people in the industry just don't have it, and you're not always going to get it in your day job if you just try to do, like, the literal task at hand, you know? And so super, super important, again, going back to, like, the greatest people I know in our, in our industry are those people who always want to tinker with new technology. That resonates so much with me, David. And I knew just on the surface, from what I've watched or read from you, that I really enjoyed this conversation because, you know, your style and confirms this, right? That you're laid back and casual and, and you're idiomatic and possibly even contrarian, you know, and I really love those things because, you know, you're not afraid of breaking things or trying new things, hiring grads, and say, oh, you broke that thing. But, wow, that was interesting. That's an interesting result. You know, I really. That resonates with me. Coming from a non traditional background, looking at some of the articles and things that you see, everyone says, you got to do it this way. And I say, well, but I had this work, this is working, and I have a really great business, and it's completely the opposite way. So doing things in different ways. So that really resonates with me. Here's a fun. I know you already gave a closing thought, so why don't we do something fun, like, what are you listening to or watching what's taking David's free time these days? A lot of things are taking my free time. It kind of honestly depends on the week. I'm one of those people that likes new hobbies. I'm always looking for that thing that will fascinate me. I have some form of add. I don't know what it is, but I always need something to engage me. And so, like, I'll play a lot of video games, for example, but I'll play the video games that are honestly, like, work. And so factorial is one of my favorites. And Factorio is just refactoring over and over and over and over. But I find it, like, therapeutic, you know? But I guess, like, you know, a lot of these days, I've been, it depends, you know, I've been doing stuff where we're up in a mountain area right now. So this summer I was trying to learn how to hydrofoil, which I'm not great at. And to continue that example, I learned to ski last season. So I do some stuff that's not just attached to screen once in a while, but otherwise, I don't know if I were to plug some stuff that I'm just a big fan of. There's another podcast, actually, if you've not heard it, called Darknet diaries in the cybersecurity space. And I love the security space, but it's also nostalgia for me because a lot of what they talk about goes back to the days where I grew up, where we had modchips and game consoles like PlayStation were just coming out and all those things. Phenomenal podcast. It's just really interesting to hear about. And then, I don't know, outside of that, I don't know. I'm just mostly hacking on side projects and things like that. And sentry is my twenty four seven, I would say. Awesome. What about you, Hugh? What are you reading or watching these days? Well, I'm not watching much by way of tv or movies. I've been reading a lot of books by Cal Newport. I'm a fan of his deep work, digital minimalism, world without email. So just finding ways to sort of streamline my work and my life and just make sure that I stay on track. But the things that I really want to do, make sure that I learn them well and deeply. So Cal Newport is. I kind of worship his mindset at this point. Yeah. How about you? I guess. You know what? I am watching something. I'm watching a ton of videos on building and construction. So I'm a fervent member of the University of YouTube these days. And I really love construction. I just love the idea of taking something from the ground and building it right, whether it's code. But I like construction, rising these buildings out of the ground. And my wife and I we bought some property and I immediately started developing it. But I have the guys out there, but I like to be able to understand completely what they're doing. Or I'll go there and say, hey, we need these rafters and these ledgers to hit in this particular way. And let's hit it 16ft on center and all these kinds of things. It and YouTube has been so good for that. And so I've been just consuming so much building and construction content and buying a ton of tools and just, like, books, I'll tell you, buying the tools is a completely different hobby than using them. But that's where I'm at these days. Fantastic. I have to remember that phrase, University of YouTube. I do spend a bit of time there myself as well. I've been down the same rabbit hole as you, Ryan, where like, I built a wood shop and everything. And I built the wood shop to build the wood shop. But it's fun. It's fun. It's like, think of a hobby, but it's building. It's fun. Yeah. So it's a ton of fun, man. And I even got into, like, sketchup, drawing and. Man, oh, just love it. Yeah, well, I live in a small New York apartment and they frown on me building shits, so. Well, David, it's been great having you on the podcast. It's been a lot of fun talking to you. And we're going to put links in the show notes for century as well as all the various books and games and things we've spoken about. Yeah. Should we include. Yeah, I'm looking at Factario. It looks pretty cool. And should we include a link to Blade Runner, you think? Of course we should. Yes, absolutely. I've looked at Factorio. I'm concerned about purchasing that game for myself because I may not get anything else done in my life, but we'll see. It's amazing. You will not regret it, but it will SAP your time. All right, well, if that happens, I'll put the blame squarely on you or the credit if I'm having a lot of fun either way. There you go. All right, thank you very much. Thanks, guys. We hope you enjoyed this episode. To hear more from us, go to Pibytes friends. That is Pibit es friends and receive a free gift just for being a friend of the show. And to join our thriving community of python programmers, go to Pibytes community. Thats pybit es forward slash community. We hope to see you there and catch you in the next episode.