
Pybites Podcast
The Pybites Podcast is a podcast about Python Development, Career and Mindset skills.
Hosted by the Co-Founders, Bob Belderbos and Julian Sequeira, this podcast is for anyone interested in Python and looking for tips, tricks and concepts related to Career + Mindset.
For more information on Pybites, visit us at https://pybit.es and connect with us on LinkedIn:
Julian: https://www.linkedin.com/in/juliansequeira/
Bob: https://www.linkedin.com/in/bbelderbos/
Pybites Podcast
#153 - Elevating Python Security: A Conversation with PyPI's Safety & Security Engineer Mike Fiedler
This week we interview Mike Fiedler, the PyPI Safety & Security Engineer at the Python Software Foundation (PSF).
We discuss the importance of security within the Python ecosystem and offering practical advice for Python developers to enhance their security awareness.
Highlights include a deep dive into PyPI security measures, the challenges of securing the Python ecosystem, tips for Python developers, and Mike's journey to his current and previous (DevOps) roles, emphasizing the blend of technical skills and mindset needed to tackle security and solving challenges problems effectively.
Enjoy this episode with Mike Fiedler!
Chapters:
00:00 Show intro
01:20 Intro to audience and stateless testing win
08:24 Transition into PyPI security and current role
17:43 Challenging securing the Python ecosystem
24:48 Tips for Python devs to be more security aware
27:18 PDM ad segment
27:44 Security tips cont'd
33:16 Solving a puzzling problem in production (mindset)
39:40 Psychological safe workplaces / blame vs accountability
44:18 Security trends, how to prepare, and resources
48:05 Books and wrap up
---
Pybites ad segment: apply for Python coaching here.
---
Show links:
- pytest-randomly
- pytest-socket
- Brian Okken's pytest content
- PyPI blog
- Trusted Publishers
- 2FA Required for PyPI
- git annotate
- Asimov Robot series
- Silo
- CPython 3.12.2 is SBOM-ified
- Trail of Bits blog
- Fastly blog
- Disaster recovery for Consul clusters (now this exists!)
- Bandit
- Pre-commit
Reach out to Mike here.
Learning how to navigate, like reading code and debugging code is so easy. Finding a memory leak is so easy compared to understanding another team member's motivation or why they're, you know, doing a certain thing or performing a certain way. The people problem is hard. Hello, and welcome to the Py Bytes podcast, where we talk about Python career and mindset. We're your hosts. I'm julian Sequeira. And I am Bob Baldeboz. If you're looking to improve your python, your career, and learn the mindset for success, this is the podcast for you. Let's get started. Welcome back, everybody, to PI Bytes podcast. With me here is Mike Fiedler. Mike, welcome to the show. How are you doing? I'm all right. Thanks so much for having me, Bob. It's really great to be here. Yeah, awesome. We're recording, right? Yeah, we're recording. Okay. I'm excited to have you on the stuff you're doing with Pypi in the Python space. DevOps background mindset. We can cover a lot of things, but, yeah, maybe. First of all, can you do a quick intro to our audience, and do you have a win of the week? Sure. Quick intros are always a challenging aspect, especially as you kind of mature in your career, because you have to pick and choose. What do you want to highlight? Want to talk about what you hope will resonate with folks? So. Hi, Mike Fiedler. I've been working in the technology space for about 30 years now, probably even more, and I've done such a wide variety of different roles, positions at companies both in the United States and Israel, where I grew up, and I've done, came out of it support operations and help desk, and kind of made my navigated through a variety of different experiences, all in the manner of learning how things work, how industries work. And that's what has gotten me here to where I am today, where right now, for the past six months, I guess, I've been full time employed at PyPI, run by the Python Software foundation. The PSF, the Python package index is that place that everyone pip installs or poetry installs or whatever tool you like, installs packages from. Some of us upload packages there. And now I focus predominantly on safety and security for that project, which people around the world use all day long. And it's a lot of fun to get into the depth of what it is and what it can be, as well as providing a foundational security for a common good. Yeah, that's a big thing. I'm definitely going to get into that. But do you have a win of the week you want to share? Yeah, I think the win of the week that I'm going to highlight is around software testing. Testing is always something that most people will think as an afterthought, but it is critical in any software that is beyond a toy project. And one of the python testing tools that I use is Pytest. It's very popular. Python test runner. One of the plugins is a very rich ecosystem of plugins. I have one myself. It's called Pytest socket, which helps block network activity during tests. That way you know if your test code is calling remote APIs, which is usually a bad thing unless you want it to, and then you enable it. But another plugin that I really like that I had not put into python API for a couple of years now is pytest randomly, where it will shuffle the order of the tests, every execution, into a random order in order to help you find hidden state or statefulness that is preserved between tests. Good testing methodology is you set up your conditions, you run your test, and then you tear it all down so that way it's clean. And then you can run every single test in isolation. And this helps preserve that guarantee by running them randomly every single time. Interesting. Is that also a plugin you wrote? No, I didn't write this one. I've used it on a number of projects and both open and closed. And I had used it on pypi a couple of years ago and found a few stateful tests. But this week is when I finally merged it into our main branch where, fingers crossed. Obviously, I can't run every possible random permutation ever, but I think we've gotten to the bottom of some of the statefulness and we can kind of prevent that coming into our testing methodology. Nice. I think Brian Ockham is always hammering on tests should not depend on each other. Right. It's just a good practice. And yeah, Brian is. If you haven't listened to Brian and anything, he's the pytest person I would go to for a lot of advice. Yeah. So one, don't leave the home without Pytest. Two, for pytest, go to Arkham. He has a lot of good. Well, his book, obviously, and now he has a video course as well. Right? So yeah, he's got a lot of content online, so check it out. But I think the importance of testing. I had this conversation with an engineer long ago where they said, I know my code works. I tested it when I wrote it, I tested the application behavior. I didn't write tests for it, but I tested it. I said, yeah, that's great, I trust you. I believe you tested it based on the specifications. But how do you know that what you did today is going to work tomorrow? You've effectively written some code that makes an implicit promise that this is the way it behaves and maybe you make a change, maybe a colleague makes a change, maybe a library makes a change. How do you know that that promise is going to be upheld in the future? And by writing tests and keeping yourself at a high testing discipline, you can write high quality software that if somebody else changes a condition, a test ought to fail. And then you can decide, does the test need to be changed in order to change the promise or, whoopsie, I've broken a promise and I need to fix that. Yeah, exactly. Because code braces always grow and there always changes. And, um, just need to have that regression suite too many times. Right. I've been called back now. You may messed up somewhere and if it wasn't for the test, then I wouldn't have known. Or, you know, you discover later on, then you have a disappointed customer or something. Yeah, maybe this is part of some of my like, operator background where, you know, when I was a sysadmin and taking application packages, you know, package jars and wars and whatever, and running them for, for developers on production hardware, there was, okay, well, has it been tested? Did it run through the test suite? Because not being the developer involved, I was like, well, what kind of guarantees or confidences do I have that this won't break in production? And if the change set doesn't have a combined set of tests, well, then it's more risk than not. So I think my testing approach probably grew out of there, but more as of more recently, it's been a pleasure to be able to work on well tested code bases because there's a safety net. You're flying through a trapeze with a safety net beneath you so you don't go splat. Yeah, cool. We clearly are both very passionate about testing. Welcome back to testing code. Oh, no, that's Arkansas. But no, thanks for that win. And that's a good reminder, right? Like, you cannot go wrong investing more time in testing, especially in the long run. It will come back. I want to know about your transition into the pipe AI safety and security engineer role. Yeah. Tell me a bit about how that came, how you moved into it and where you're currently working on. Yeah. So it's an interesting path. Throughout my career, I've worked both individual contributor. I went up the management ranks and went all the way to senior director of engineering at some startups and enterprises and have worked different aspects between right, as part of a manager's role, you often don't. Your job is not necessarily to do the thing, it's to work with people who are supposed to do the thing. I'm obviously over simplifying a lot here, and many managers do jump in and pitch in, as I used to until I was high up enough in the ladder to understand that it is more detrimental for me to use my time on individual contribution than it is to work on managing my folks who can improve the individual contribution for the company. But in order to keep my chops active, I worked on a lot of open source because that's my own free time. That is not company critical. That is not anything. It's just something I can poke at and help out here and there around the world. And years ago, when the chef community was big and there is still a very large chef community back when I was using it and there was a whole lot of you can learn how to use this tool. You can write code and then share it. That was like, oh, okay, so I can open source some of my code. Cool. You start sharing that, and then you start looking at other people's code and seeing I can contribute to theirs and fix it. You know, there's a bug here that I'd like to use this, except for this one edge case. So let me add some code, let me write a test, hopefully, and contribute back. And for me, that was such a gratifying feeling of I'm not only kind of taking from the open source world, but I'm also giving back, which is very critical in this overall sustainability of open source ecosystems is engagement, contribution. And it doesn't only have to be writing code, it can also be in GitHub issues, triage, reviewing, pull requests for accuracy, et cetera. There are so many things that you can do in the open source that aren't writing code. But for me, I got my really heavy start in open source, back in chef, contributing cookbooks, getting back into policies, procedures, client side tools, etcetera. And I enjoyed it so much that when I was now working as a manager, I was looking for other software or tools that my team were using that I could go poke at and see, can I improve those software tools and kind of indirectly help the team through a non critical lens. And as I left my last role at a startup about a year and a half ago, at this point, no, just over a year. I took a break. I was like, okay, I need a reset, take a work break, figure out what I want to do next. The landscape is open. And at that time I had already been contributing to PI PI. I had worked on a couple of different front end features and JavaScript, which is not my first language or language of choice. I figured out some of the CSS and HTML stuff there and it's like, oh, these are all, these are all fun things that are again, not critical to the operation of Papi, but they were kind of on the periphery that were a good way for me to get my toes in. And major kudos to the maintainers at the time who were very excited to have somebody contributing. Ill shout out to Dustin Ingram who recognized that hey, this person can do something, lets ask them to do something else once theyre done right. So Dustin kind of geared, guided me towards the next issue that he thought I could take on. So I did that for a while and then I got invited to become a volunteer maintainer. As a volunteer maintainer, you spend whatever time you want, work on the things that are interesting to you. And then I was doing this for a few months, working from home, having a good time. And the Python Software foundation got some funding from generous donors to create a safety and security role for specifically PI PI. I was very excited about that. I threw my hat in the ring. Another bunch of very good qualified candidates put their names out there as well. We went through an interview cycle and I got the gig and that was very exciting. I started in August. I blogged about it on blog pypi.org, which I helped build up and is nice and pretty now. And it was very exciting to be the first one, and then specifically the first full time employee for only PI PI. And then specifically to focus on safety and security. The big projects that I've been working on since, which one of which has been the completion of the mandatory two factor authentication project for PYPI, which just generally increases safety for the entire ecosystem as a whole, minimizing the possibility for account takeovers, which is, it's a thing, it happens, we're trying to prevent it, so play along. And then the other one has been this work that I've been focusing on to do the automation or provide some methodology for automation for reporting security vulnerabilities around packages. On PYPI, we get a fair amount of spam malware garbage uploaded to pyPi, because again, it's a free service. It takes a few minutes to create an account, log in, set up your two fa. But at that point, it's free. It's for the common good. Some people abuse that. That's unfortunate, but that's the line we have to navigate by providing a free and open ecosystem versus should we charge people to have a PI PI account? I don't think so. That'll never fly. But we have to come up with creative ways to combat the bad behaviors. The one that I've been working on is providing a methodology or API driven approach for security scanning. Companies and trusted partners who scan every single package that gets uploaded and report it to us. Today they report via email, but we're looking at how can we provide that via API and then minimize the amount of time it takes from reporters detecting until the malware is taken down. Interesting. That's a heaps. Let's distill this a little bit first, how you got here. That's interesting. You started with open source, contributed to PI, PI, gone on their radar. It's almost like if you would have asked you, like five years ago, you would be doing this. You wouldn't know because it organically happened from one thing to the other. And even the security aspect, I mean, it makes sense that that's in your background that you have with DevOps as well. Right. But that really came from another thing. So that's interesting. The work is interesting. The fact that there's now full time staff on it is awesome, because that's very important. Yeah. That would only come around from the generosity of donors. The Python Software foundation is a nonprofit organization. We're not a venture backed company or something like that. That is just piles of money. We rely on companies, organizations, individuals to. To help make the mission of making Python the best and fastest and most secure language. That's not the exact mission statement, but that's the one that I want to have in the back of my head. Safe, secure, fast. All the things you want out of a programming language ecosystem. That wouldn't happen without the very generous donors. The shout out that I would ask is anybody who works at an organization that uses Python in any fashion, and you probably do, almost every company does in some department to look at what is your company's stance on open source and open source donations and donate to the PSF. It makes these things happen. Yeah, that's a good point, because, again, this is really critical. Tell me a bit about the challenges then you face in ensuring the integrity and security of Python packages. Yeah, I mean, where does one even start, right? It's such a wide ecosystem. I think one of the important parts around the Python ecosystem, as opposed to many others, is that we have a pretty strong standards process for interoperability. This is also called the PEP or Python enhancement proposal process. This is just, again, Python is now about 30 years old, PyPI is about 20 years old. Conceptually, the actual code base that I work on day to day was re envisioned in around 2015 and launched in 2018, something around those circa timelines, but the concept and the packages have been around for 20 years. The challenge is how do we ensure that the changes that we want to make continue to work for all the people who use it? And in modern parlance you might call that contract testing or API contracts, which is effectively what we try to do through the Python enhancement proposal for things that are end user facing all of the different API interactions that exist on PYPI. For the installers, such as PIP or poetry or any of those, those move very slowly. They don't change very frequently because the cost of changing them would be very, very high. It would break a whole lot of people's workflows. And it's how do we minimize the impact of a given change? And then the other aspect of security is, okay, well, how do we continue to evolve our security practices without breaking those interfaces? One example is our two fa change that we did recently. By requiring two FA, we also bucketed in requiring of API tokens. Historically, again, we're dating back 20 years. Way back then, you didn't even need username password. There was no index. The index was just a list of where to download other packages. Then we got a server and people could just FTP files onto a server. That was great. Then we matured and got to smarter technologies. We did basic authentication, which again, you're putting a credential, a username and password in a encoded header, which sure, we rely heavily on SSL TL's in order to make sure that nobody can sniff that, but it's still a credential that you could deserialize. Instead, we moved on to API tokens. We did API tokens in maybe about 2019, and that's when they were new. But we supported continue. Again, not to break backwards compatibility, we supported using basic authentication by promoting API tokens and then making them even easier to use through some of the work that we've done through trusted publishing with security firm trail of bits, we got to the point where if you're using GitHub actions to publish your package to PyPi, you don't even need to provide a credential anymore, you used to have to provide a secret token, a PI PI token, into the GitHub actions, and that's a long lived token. If anything happens, if anything gets exposed there, you don't want that token to leak because then somebody else could upload packages your package, and that was undesirable. So with trusted publishing, we're using technologies like OpenID connect to say we know who GitHub is. We have a shared relationship with GitHub, and if the user has correctly set up their project on PyPI, and they use the correct GitHub action flow on GitHub actions, GitHub actions will identify itself to PyPI, will validate that. It will then request a short lived API token, give that back to your GitHub action. GitHub actions will then use that short lived token to publish packages, and then the token expires. So even if anything malicious happens in that small window or after that token is invalid, these are steps towards a safer way to do this. We've had work done to get a Google OIDC very similar to GitHub. It's not live yet, it's live on test pipe PI. We also have some initial work for the activestate platform. From them, they're contributing this. So again, this is all being done by volunteer contributors once we set the pattern. So if you're running a publishing platform and you want to enable OIDC connect, then hey, the code is there, send us a pull request. Yeah. So it seems that integrations and partnerships are critical to your work. Right. So they are in as much of. I need to make sure that I'm reviewing their contributions. Part of my, my role is not to currently reach out and kind of get them to do these right now, because it's like, if they don't want to do it, they don't have to do it. The partnerships that I am pursuing are the ones with the security research firms or malware scanner type companies, who are graciously and generously volunteering their time on their platforms to help us secure our platform. Yeah, because that was my other question. How do you keep up with the security stuff? There's a lot of unknowns, and isn't it an eternal battle to stay one step ahead of the hackers? Always. Sometimes you're one step behind and you got to catch up. It's a challenge. I think a lot of it comes down to the partnerships that we have, right today, most of PYPI runs on top of Amazon Web services who are pretty good with their security paradigms and our content delivery network infrastructure partner is fastly huge. Shout out to them. We could not survive without them. They carry 99% of all the traffic for packages because it's the same packages. But they are an excellent partner, always looking to work with them on newer ideas or implementations that could mutually benefit both of us. Yeah cool. So if I shift a little bit then to the Python developer because security starts with the developers as well. Any advice how we as Python developers can be more security aware, adopt safe coding practices? Maybe you have a couple of tips there. Yeah, I guess theres never a perfect answer to that question. Like you said, its always evolving. If you can afford a chief information security officer at your company, hire one. If they can afford a team to run security, hire them. If you can hire a team of analysts to sit there and pour through things, hire all of that. But most of us can't do that. I think the part that I try to rely on heavily is being aware of your dependency chain. Software dependencies are a thing. You could write all of the code in your applications and stack yourself. You could, you could also invent your own language, which some people do. You can do that and then you are on the hook for all of that. Which is fine if that's the trade off you want to make. Most businesses would rather execute by picking some open source software that largely accomplishes 80% of what they're trying to do and then sprinkle their 20% on top. So dependencies are a reality because it accelerates our goal to the, it accelerates our accomplishment to the end game. And as such we should be aware of those dependencies and when they fall out of date or if they have published security advisories. So I try to do a little due diligence on dependencies before inclusion. There's some resources out there like safety or piup or sneak or libraries IO which we link to from PYPI that will show you a little bit around kind of the package, the metadata. You can then go explore their repository, look at how many people are there. There is work from the OSSF, the open source Security foundation, to develop a scorecarding tool. They have a scorecarding tool, but to continue to evolve it, where you as an end user could run a scorecard or see a scorecard for a given project and then make a decision, make an informed decision in just twelve weeks. Pivots elevates you from Python coder to confident developer, build real world applications, enhance your portfolio, earn a professional certification, showcasing tangible skills and unlock career opportunities you might not even imagine right now. Apply now at Pibit Es PDM. So part of the dependency chain is also keeping them up to date. I use GitHub and they have a great dependabot offer. There's other solutions out there like renovate bot and other companies. Mendrenovate they renamed just to continue to be aware of. This package has had a security vulnerability, and here's its updated patch. Oh, okay. Take a look, review it, see if it applies, then build, test, deploy, get that out the door. That way you minimize the time you would be running a potentially vulnerable package. The other aspects are around static analysis. You can always review code with tools. There are a bunch out there. One that I like is called bandit, where you can run it against a codebase. There's even a pytest plugin to run it. There is a pre commit plugin that'll run it. There's a variety of ways to execute this static analysis to confirm what's going on and tell you. Okay, you're using a SHA one algorithm which is known to be broken. Are you certain you want to use a sha one? This might be for a cryptographic purpose. Don't do that. So Bandit will surface things like that, which is very helpful. Yeah, those are great tips. Thanks. Also. Yeah, Bandit just put it in your pre commit and now it will run on point commit and it will probably shout at a couple of things you might have not thought about. Yeah, I think it's very akin to like any other linter. Right? You want to keep little bits of annoyance out of your code base before they become a problem. So if you can run a tool early enough in the pipeline so it's not in production, that will warn you this could be a problem and then gives you, the developer, a decision point. Like, a lot of people complain about linters and, oh, the linter forces me to do something. That's a rule. You can disable the rule, you can change the rule if you and your team agree. But like, the whole point is that these are some practices that we think are good. Uh, you don't have to adopt them unless your company says you have to, in which case that's, that's a different policy problem. But ultimately, uh, you know, being aware of secure practices out there is important. If you're doing web application, there's the OWASp top ten list that they review and revise periodically, usually once a year, once every two years, where they'll call out some of the biggest important vulnerabilities in web applications such as SQL injection. If SQL injection is a very common pattern where if you take untrusted input, such as a search box, a lot of sites have a search box. If you take input there and then ran that just straight as without any standardization or validation in a search in your database, that attackers might do a drop table an update, do something malicious, and then you'd have a bad day looking for opportunities in your code where it's like, well, this will work, but this leaves a door open for some attacker. Maybe we should try and close that or make it a little harder to get to are always good options. Yeah, and I think that's a good reminder as well. Do not reinvent the VL red, especially in web development. If you use a framework like Django, there's a lot of security features already in there. This whole SQL injection problem is already resolved by how they handle forms. The forms have their CRSF token right there. Doesn't that just submit the form unless you have the token in your template. There's the security checklist. There are settings and settings tie that. So, you know, and if you would build all that from yourself, yeah. Then you're on the hook. And you're probably not that knowledgeable as 20 years of whatever the Django framework is around and the security practices and specialists that have bent over it. You know, I'd say the best, the best, best thing that one can do is periodically hire a security firm to audit, figure out what you want audited, and then have them audit, because they're going to be the experts. They're going to be those members on your team that aren't you, that aren't experts or aren't necessarily the looking at the same problem with the existing context by having new folks take a look at it. There's also a great opportunity for new hires on a team or new team members, is these folks don't have the context. They're looking at it with fresh eyes. The things that you might have glossed over or say, you know what, that's probably fine. They might call out and say, why do we do that that way? And that's a great opportunity for you to either sit down and say, yeah, I don't know why we do it that way, maybe we shouldn't. Or here's the reason, let's document the reason in line so that way the next person understands why we're doing it this way. So that way the conditions under which you've kind of made an exception are clear in the future. Yeah. Well, it's a fascinating world, the whole security thing. I do want to pivot a bit to mindset. You mentioned the puzzling problems in your bio, and you're asking for a story, an example of a particularly challenging problem you've solved in your career can be anything, right? How did you approach it and what lessons did you learn? I guess persistence and whatnot that our audience can benefit from because you have to tech. But, you know, complex problems also involve a lot of mindset. Do you? I mean, if you have an example. Otherwise, we just get to the next. Well, I'll say for any managers out there, or people aspiring to be managers, you know, if you love software development because of coding, managing is a whole different ballgame. It's a very different career choice. There is advantage to having a background there, but people are the most puzzling problem of all. And I say problem with tongue in cheek, people aren't the problem. But learning how to navigate, reading code, and debugging code is so easy. Finding a memory leak is so easy compared to understanding another team member's motivation or why they're doing a certain thing or performing a certain way. The people problem is hard, and it requires a certain degree of investment and empathy. And to learning people. We all spent a lot of time learning our editor shortcuts or our code semantics. People are more complicated than anything you can imagine. Just because you're a person doesn't mean you understand the person next to you. So for those on the management track, but for a more technical, puzzling problem, this was a lesson I learned the hard way many years ago when I did the thing that you should not do. It was an early console cluster that was running the configuration management backend for our fleet of instances, and I needed to update some configuration on all of our cluster, uh, primary instances like the, the leaders of the cluster. I forget the semantics, apologies. But the configuration was okay. You need to edit, you know, the line in the Etsy config file and restart the process. And this is before I was using configuration management on these particular files. So I was like, all right, well, I need to edit it on all five. Let me open up five parallel ssh windows, right? Same thing. Edit it in all of them and issue a restart. And then after I hit enter, I remembered, oh, you should not restart all members of a consensus system at the same time. So we had five instances, they all received the same restart, and they all restarted and said, hey, I'm not the primary of this system. Somebody else must be the primary. Let me go ask around to see who they are. And none of them would say I'm primary. There was no tiebreaker back then. This is I think, console version 0.5 or something. I think theyve fixed it long since. But it was like this moment of, none of the cluster will boot up, no systems will receive their configuration. Everything is down. So it was a puzzling problem of, okay, when all of these are forced down, how do you bring them back up? How do you force them back up? And it took some really hard thinking of, let's read the source code of why these binaries are operating a certain way. And back then I think it's still written in go. And that was not my strong suit. Still isn't. So I was like, ah, I'm walking through the darkness and trying to find out. I had a bunch of folks helping me out with this. It was a really good team. We eventually got it restarted by kind of faking the state on one of the instances to say, ignore the other machines. Assume everything is okay, but you'll be leader. And then once that came up it was like, okay, bring up everybody else and they'll join the cluster and then cycle the whole thing out. Because it was all messy at that point. But this was in production, right? Oh yeah, yeah, this was, yeah, don't do what I did. Right. Like this was truly the wrong choice to do today. It would be a much different approach of, okay, you deploy the configuration selectively to a minor volume of the cluster, restart those, make sure everything is back up, and then cycle through an eventual rotation process. But it was like, oh, the minute I hit enter I was like, oh, I messed up. This is bad. How do you stay cool in that situation? Because it sounds like you really had to just cool down and stay focused, read the source code, engage the right people. That's some, you know, yeah, I mean, right there, I'll lean on some of the, like the DevOps, the sysadmin, the operator methodologies back then of like, yeah, you need to call an incident commander. You have to have somebody run the chat room. You got to get people together and just say, here's where we are. We can argue about how badly I screwed up later, right? Like we can, we can, but right now we have a situation to deal with. So lets deal with that because we can run around screaming with our heads flying off the hook, but thats not going to solve the problem. So lets solve the problem and then we can get into a conference room and yell at me or drink beers. Nobody yelled at me. Its fine because its like, this happens, production incidents happen, and the learning for everybody is. Yeah, dont. Don't mess with a consensus based system like that. Historically, if that was a bunch of web servers, yeah, of course. You edit a bunch of the same file, you restart them all. It's a web server. They're independent. But because these systems were linked, it was, uh oh. But the remaining calm in the storm, I think that's just something that, again, you have to practice. You have to know that, well, hopefully, you know that you work at a place that has the psychological safety of, well, I'm not going to get fired over this. I might get a talking to of, like, let's figure out how this happened, why this happened, why I made those choices. But I'm not getting fired because these things happen. Right. I was trying to do the right thing. I wasn't trying to take down praed. So I think having a lot of working places that have good psychological safety is important to remaining calm under pressure. Yeah, those are great points. Yeah, exactly. There should be room for failure, because inevitably, failure will happen and how you're then going to deal with it. Yeah. Interesting. Cool. Uptime of 100% all the time, every day is very nice, but you haven't learned how to deal with failure. So when failure happens, you kind of might lose your cool. Whereas if you periodically have an error budget and you do fail a little bit here and there, or you introduce failure as a testing methodology, you learn how to interact, how to operate under pressure, and how to recover. Yeah. And if you see somebody very skilled and experienced in the field, I don't believe that's only from successes. Right. Like, there's a whole bunch of failures to back that up. Yeah. I mean, the phrase I've always liked is the way you get good at something is you start out by being bad. Because none of us were born with any of this. Right. Like, nobody was born writing python, unless it's Guido, which he might have been, I don't know. But we all had to learn. And the way we learn to walk is we crawl, and then we try and stand up, and then we fall a little bit, and then we're close enough to the ground at that point where falling doesn't hurt as much. So the risk is not that high, but the reward of, oh, okay, I can continue to learn. I can learn how to walk. I can learn how to run. I can learn how to drive. Right. All of these things are learning opportunities. And you make some mistakes and the way you recover from those mistakes is truly, I think, a good sense of who we are and what our character is. Yeah, that's awesome. Mistakes are going to happen, but how do you react them? Are you going to go in victim mode and complain and play the blame game or you just deal with the issue? And I shared that experience with you, that there is room for failure if you can act upon it professionally and fix it. And then actually you might come out as the hero, even come out more mature and experienced. You've learned. Yeah, I will say you mentioned don't play the blame game. So very often folks will misconstrue blame with accountability. Blame is often also associated with shame. If you and I work on a team and you make a mistake and I call you out in front of the team, that's not going to feel good to you. The team aren't going to say, what the hell is Mike doing? That's not cool and it's not good. Accountability can be handled, again from a manager to another team member. You know, lets take it aside, lets talk through why these choices happened. Heres the impact. How are we going to try and mitigate this in the future, this particular thing? Right. Lets talk about that. Thats accountability, right? Its, yeah, you did something. It happened. Thats a fact. Im not blaming. Im saying that, yeah, this happened. Right. This is the truth. This is the historical fact of the matter. But if we can kind of separate this notion of blame from accountability, I think we can re encourage people to both take and request accountability from their teammates. Yep. Maybe we should open a pr to change the git blame command. I think that's already git annotate. Is that no longer, I mean, get blame exists. I use it all the time. But it's git annotate. It has a slightly different semantic. But it's there. The reason I use get blame is because my muscle memory. Yeah, muscle memory. It's good to know there's an alias that has been interesting. We're coming up on time. Thanks for sharing. This is really coincide and inspiring. So two more questions. Looking towards the future, what trends or advancements do you foresee in the realm of python security and how can we prepare? And feel free to loop in AI. Yeah, AI is going to solve all the problems we went. How long, Bob? Without mentioning AI? I was going to not do it, but open question, it wouldn't be a podcast without mentioning AI and AI wouldn't be AI if it wasn't for Python. So there you have it. But in the realm of software security. Some of the things where I foresee the future, I foresee a greater adoption of trusted publishing as a practice. We've seen it picked up not only in Python, we've seen it picked up by rubygems as the same way of just use GitHub actions and they will transact with the platform and securely minimize the amount of tokens that are just floating around the universe, which is great. I'd say we're seeing a desire for more and more folks to have software bills of materials, or SBom SBOM, as a way to ask themselves what software is in my application stack, what versions, what licenses, what is in this? Because very often you can look at a container or an application running on a server, and it's not always apparent all the ingredients that went in to make this beautiful cake. You could, on demand, spend a lot of time to dig through and find out every single one, or you could automate a lot of that. So I think we're going to be seeing a lot more work automating the creation of software bills of materials. So that way any package you download will contain, or could contain this list of sure, I am written in Python and I'm a package, but I rely on these other three packages and here's what they have to say and what their extensions and licenses are. So that way companies and individuals can know what they're putting in to their systems. If you or others are more curious, I'd recommend you follow Seth Larsen's blog. He's the security developer in residence for the Python Software foundation, and he's been working a lot on software bill and materials stuff, and he's been an excellent author on a lot of the topics and I'm just doing them a little bit, little early taste of what that is. But he goes into a much bigger detail, deeper, bigger, much more detail as to what these are and how they apply and why they are important. Yeah, good to know. Cool. Is there another resource we can follow? Blog at pypi I guess. Yeah, blog dot Pyp is a fun read. It's very pretty. It's a lovely resource that we, the administrators use to publish some insights around what is happening specifically to API. As big changes happen, I'd say listen to some of the other Python podcasts, talk Python, real Python. They're all really good resources to just stay abreast of python related topics. Yeah, indeed, indeed. Cool. Lastly, what are you reading? Any book recommendations? So I was just gifted. I can't read the title, so I haven't read it yet, but what am I reading right now? Right now I just finished rereading the silo series by an author named Hugh Howey. Its a three book series, science fiction. They have taken it to Apple TV and made a television series out of it on Apple TV. Its quite good. Its different than the books, but after watching the series I sat down and said, a lot of this feels different. So I reread the books and it was a lot of fun. Im considering rereading foundation by Isaac Asimov now that that too has turned into a television series to just reread some of the early science fiction concepts and see how much has changed in the 30 or 40 years since. Yeah, that was very interesting. I've not read what's his name? Isomov. Yeah. Isaac Asimov. Yeah. He has many, many books, right? Yeah. If you're looking for an intro to Asimov, I'd look at the Elijah Bailey series. These are Elijah Bailey is a human detective, and he pairs up with a robotic Android or humanoid robot in order to investigate a murder of a human, which robots are supposed to not be able to do. So it's a fun kind of murder mystery but futuristic with robots. Awesome. And it was written long before a lot of our modern AI stuff existed. I think the audience will appreciate that. I got a link, of course, all your socials and stuff, but do you have any preferred one where people can reach out? Yeah, I'd say Mastodon is the place. I'm on the hackyderm web server for Mastodon, Hackyderm IO, and I'm iiketheman there. That's probably the easiest. I'm on GitHub. I'm on Discord. Yeah, I'm on all the different platforms. Are you on py bytes? What is pibytes? A circle community? Oh, no, I'm not on circle communities yet. In fact, I only learned about circle community from listening to your podcast yesterday. I didn't know about it. Well, if you decide to join in there, people can hit you up there as well. But mastodon it is, right? Mostly, yeah. Mostly, yeah. Awesome. Well, thanks for hopping on really fun chat and yeah, a lot of golden nuggets, so I appreciate it. Awesome. Thank you so much for having me. It's been so much fun. Yeah. All right. Have a good day. Thanks. Take care. We hope you enjoyed this episode. To hear more from us, go to Pibyte, that is Pibit es friends, and receive a free gift just for being a friend of the show and to join our thriving community of python programmers go to Pibytes community. That's Pibit es forward slash community. We hope to see you there and catch you in the next episode.