Pybites Podcast

#022 - The Karmabot Story and Contributing to Open Source

Julian Sequeira & Bob Belderbos

This week we talk with Patrick Groß, long time PyBites community member and lead developer of Karmabot, the bot that handles karma points and other tasks in the PyBites Slack Community.

We talk about how Patrick got into Python and how he really learned it, his switch from a university to a data analytics job and ... his overhaul of our early Karmabot prototype, turning it into the mature open source project it is today (thanks!)

You'll learn about the improvements Patrick made to Karmabot, integrating "Hypermodern Python" concepts, and some of the lessons he learned from contributing to open source overall (hint: both tech and soft skills were important!)

We hope this episode will inspire you to start (or continue) contributing to open source, it's a beautiful thing.

Karmabot project and Hypermodern Python series:
https://github.com/PyBites-Open-Source/karmabot
https://cjolowicz.github.io/posts/hypermodern-python-01-setup/

To reach out to Patrick:
https://twitter.com/patrick_grs
https://pybit.es/community 
(DM or #karmabot_dev channel ... actually while there let's give him some Karma, just type "@Patrick +++")

And I can remember that at some point, you, Bob, wrote in the chat, so, ah, sorry, karma bot crashed, and all the karma from the last week is gone. And there were some basic problems with the bot, so people lost their points because some problems with the bot. And then I thought, yeah, why is it? And then I looked into the code, because it was on GitHub, and you pickled it, and I think the pickling had some problem. Yeah. Hello, and welcome to the Pibytes podcast, where we talk about Python career and mindset. We're your hosts. I'm julian Sequeira. And I am Bob Baldeboz. If you're looking to improve your python, your career, and learn the mindset for success, this is the podcast for you. Let's get started. Welcome back, everybody, to the Pywytes podcast. It's another episode. I'm here with julian. What's up, man? How's it going? Good, good. And we have a special guest again today, and we have Patrick Gross. Hi, everyone. How are you doing? Yeah, everything fine. First time on a podcast. Oh, we're happy. Glad that you could be here with us. Where are you based, Patrick? So I'm based in Germany, and Braunschweig, which is in the middle, northern Germany. Oh, that's cool. I've only been in Germany once. Definitely didn't go there. So that's it. Got to add it to the list just to meet everyone. Now, for everyone listening, Patrick is part of our pie Bytes community. You can see a trend. We keep talking with people, part of our community. We love it. And Patrick has been around. How long do you think you've been in the community for now, Patrick? I would have to look it up, but I think about two years, two and a half years, maybe. Yeah, one of the old hats. So thank you, obviously, for sticking around with us and all of the commentary and support that you've given to everyone else in the community over the years. We do appreciate it, and that's actually why we want to bring you on the show, because you're doing some really interesting things with Python, with PI bytes as well. But before we dive into that, do you just want to introduce yourself, tell everyone a little bit about yourself, what you do working with Python, anything like that? Yeah. So maybe start of how I got into Python. That's quite a funny story. I think. So about one year before now, I worked at the university and in research for mathematical optimization, basically. And we had a student there from my colleague who wrote his thesis, and he was quite good at using Python and he used it to collect some data from a traffic system, actually I think in Melbourne it was. And yeah, he basically scraped data from an API for our research, so we needed input data for our optimization models. And then he left and the code was quite undocumented and nobody of us knew Python, we all were Java people. But I wanted to use this data because it was quite a nice source for using, because usually in research the biggest problem you have is real world input data to test your stuff. And then I had to learn Python to understand what he did and how to use it and started digging through and it was very confusing because it's quite different to Java with installing these packages with pip and importing things and indents matter and you get really confusing used. But I got through it and I learned about the awesome ecosystem of Python. So I saw all this stuff, I could just install and use it, which in Java usually have to build yourself or have a hard time looking for it. And that's where I got started on Python and then I started using it for different things. I used it for building a scraper to get data out of excel sheets and plan our exams with it, building a GUI using Django for a literature system. So I did quite a lot of stuff at work using Python, which was really easy by using these packages which you usually would build by your own. And then, yeah, I got used to it and nowadays I basically only use Python, so never look back. Yeah, now in my current job, so I changed from university to industry and I'm now working in basically data science, business intelligence, working a lot with data driven stuff. And there you really need python because all the data science stuff is in python and all the cool processing stuff for getting data from, wrangling data, it's all python, so very good fit for that. That's an awesome story. Yeah, it was all driven by necessity and then the desire to use it. I love it. So just out of curiosity, how did you get from that Java role where you were just dabbling in Python to this role that you're in now? Basically I started with this source code I got from the student and I tried to understand it. I didn't understand anything because he already used stuff like list comprehensions and dictionaries and all the stuff you usually don't really understand if you're coming from another language. So it's pretty unique python stuff used there. And I actually then just bought a udemy course on Python that was really good, rated from a german guy. So I also started in German to get it better get a better understanding what he's telling me. And was like a 20 hours course on everything. So basic python, everything list comprehensions, all this stuff, how to understand it, how does Python work? It's not a static type language, different from Java. Again, indentation all this stuff. And I worked just through this course because I wanted to know how it works and how I use it. So I had a goal to use this code and I just followed the course and then I had a basic understanding of it and started dabbling around with it. And finally I got it working. I could use the data, and then in the next steps I found use cases like this scraping or this planning of exams with a special tool I wrote. So I usually have small projects I wanted to finish with it, and that was basically the starting path. And the next step I found out about pibytes. I don't know, I don't remember how I got to you, but somehow I found it. And that was really a push for me to advance my skills because it was different to the usual platforms like this hacker something something platforms where you implement the 9000th iteration of some sorting algorithm and you don't learn anything. And with Python I learned. So you have this problem, it's described, now solve it with Python, and then you learn how to solve problems and not this algorithm training. And that was really helpful because these problems occur very often, and if you know all these problems and how to solve them, you can do really amazing things with it. Awesome. So you went from video course to building projects to supplement it with bytes, which are not too algorithmic yet. They were really real world problems, right? Yeah. So, because before my job at the university, I studied basically computer science. So I know all these algorithms. And the problem is, usually you don't have the problem how to implement this or that algorithm if you're not a very deep dive computer science engineer. So you don't reinvent these algorithms, you usually use them. And that's the cool thing about Python, because in Java you usually have to re implement all the data structures. You have to re implement all these little stuff like replacing something in the string or getting data collected together. You have to write your own classes, your own getters and setters and all this stuff. And in python you just have it included. So batteries included, you just import collections, you import mafia import all this stuff. And there's so many stuff pre built, so you can really do really fast iterations on your real problem and not reinvent the wheel every time you write a new program. Yeah, that's a great observation. Compared with Java, simple is better than complex. And in addition to all that, you also took ownership of one of our open source projects, Commabot. And yeah, we would love to talk a bit about that. Yeah, how that happens, how did you take it over and what did you do and what did you learn? I can't really, it's too long ago, I can't really remember how it went really, but I can remember I started using the PY bets platform to learn stuff, so I just used this platform and then I saw you have a slack channel. So I joined the Slack channel and daggled around a bit in the Slack channel, and then I saw this spots doing stuff with people, so giving karma, answering questions about what are cool channels to look at and all this stuff that was already there. And I can remember that at some point you, Bob, wrote in the chat, so sorry, karma bot crashed, and all the karma from the last week is gone. And there were some basic problems with the bot, so people lost their points because some problems with the board. And then I thought, why is it, how can this happen? Because from my work around and my studies, I usually use databases and persistent data storages. So I thought, how can that happen? This software engineering, software writing chat, how can they lose their data? And then I looked into the code, because it was on GitHub, I think it was just a dictionary with key was the idea, the name of the pickle in the chat and points, and you pickled it. And I think the pickling had some problem. I love that you bring it up. I can remember from some articles I read that pickling is not a good thing in Python, so this is one of the things you should avoid. And yeah, then I thought again, like with the projects before, there's a problem how to solve it. I looked at Kamabot and the basic problem wasn't too hard because what we just wanted to do is save. So we had a name and end points as an integer, and we want to save it somehow that this persisted and not going away when the bot crashes. So my obvious choice for such things is just put it in a database. So that's what databases are built for. And yeah, that's the point where I really started digging into real open source projects, I think, because before I just worked on my own, on my own machine, with the code I wrote, nobody took a look at this stuff. I usually didn't test anything with real tests. So like Pytest, I just tested it by running it and showing if it breaks. And, and then I started looking into through the code of Karmabot and how you did it, Bob. So how is the flow in the code? Where does, what does it branch? What does it do? And then I just tuck in where the points were saved and basically expanded it from using some orm. So SQl alchemy I used, because I knew that from projects before where I used it and started saving the stuff to a database and somehow learned how to get it into the repo. So forking, opening a pull request, all this stuff I had to get through to make it useful for everyone because I could just have forked it and had it on my GitHub, but nobody would use it there and you would have a hard time to use it in the slack channel. So I had to think about how to deliver the code to you and make it accessible for everyone. That's an awesome experience. And remember that there were quite a few challenges with all this. So first of all, thanks for that change, because persisting it with a database is so much better. And that pickle solution was obviously not scalable. I think we also had it running on a, it was really like an experimental project, so we had it running on a PI zero somewhere, and it went down because a friend hosted it. So it's just this experimental project, but people were getting all these karmas. It was actually becoming more important. So it was really great that you jumped on it because that project deserved some more cycles. But yeah, I can also remember the whole deployment was interesting. And I think you started using poetry. I think you introduced us to poetry actually, and then we want to have it on Heroku. But yeah, you needed to have a poetry build pack to get that done. And that was another challenge, for example, right? Yeah, yeah. So that was basically the next stage of karma bots. So the first stage was modernizing a bit in a way, how to save the data and how to handle the commands. I think we also fix some bugs in the code. Usually you have some bugs there, so some small fixing. And I think the next big step was hyper modern Kamma bot. I called it because we saw this article you talked already about, I think hyper modern Python with all the cool new stuff like using poetry for packaging and Pytest and Knox and all the stuff you need basically to have a real running open source project, because you need to manage your dependencies, you need to make sure the tests are running. When you have a pull request, you need to have linters running so you don't reformat your code on every pull request for the whole repo and all the stuff was very well described in this blog post, hyper modern Python, but it was quite complicated if you didn't do anything similar before. But again, I worked through it. So I took the hard path through this article and step by step implemented this stuff into Karmabot. And in the end we had a. It was really awesome because then we had an installable pip, installable comma bot package. We had an easy way to deploy it. We had running GitHub actions to run the tests. So everybody who contributed to the project, we could immediately check if it breaks or if it, if it's okay. We have pre commit hooks to ensure that the styling is hell two. And all the stuff, it's really helpful because usually in my new pro, new project now, I use all this stuff again. So it's not just for karma, but it's a habit. But I would say because it's just a good habit how to manage your project using this stuff. And I think that was the next big step. And funny thing, as you told, because used Heroku you have on Heroku, if you deploy, you have to build your package and install it in the. I think it's a vm or a container that runs on Heroku and you needed a special build pack because the standard build pack for Python doesn't work with poetry on Heroku. And there was some existing build pack from some guys, but it didn't work with our version because they had harder card coded the version. And I also contributed to their repo where the split pack was hosted and then improved it a bit. So you can choose the version via environment variables in the setup for Roku. So even going further, just doing something for Kamabotch, contributing to the next repo, wouldn't have done it if I didn't look at this project, man, that is amazing. I really love this story because we get a lot of people that we talk to who say, I don't even know how to get started with open source. It's just so daunting, it's so scary. And I mean, look at how much you learned and got to implement, you know, as almost like the project lead on this, just by jumping in there because you saw a gap and we didn't have the time or the cycles to jump into it, so you just did and took it over and made it this something we couldn't have even dreamed of having, let alone this professional. So one, thank you, but two, do you have any, I guess, message or thoughts that you want to share with people listening who might want to start diving into open source like you did, because there are huge takeaways that you've had from this. Yeah. So apart from this. So I learned a lot of technical stuff. You can learn also with tutorials or something. But I think the points that were important from a personal point of view is in the first place to just get going. Because if you look at this process, how to implement the database, how to implement this hypermoran, and usually you can't comprehend where it's going at the start. So you just have to jump in, work on it, keep going, and then at some point you finish and then you are very happy with the result. Even if you thought at the start, you can't do it, it's too complicated. Because usually this is very daunting topics and daunting tools you have to use. But if you just use it and apply it to something you know a bit about and you care about, it's easier. And what I also observed is after I started contributing to Karmabot, I think in the start it was just Bob and me, but after that people saw people are doing something on this project and then they jumped in too. So now when I look at GitHub, we have like ten contributors to the repo think in the start it was like two or three people, because people saw there's progress, there's something going on, they also jumped in and they saw it's not impossible to contribute. And I think that is something I learned, which I also now use at work a lot. You have to engage people, you have to start something. And most of the people, they will follow you and help you because they, they just don't want to start it. They are afraid of failing or something. But if someone is there to start it and to lead it a bit, then there's a lot of stuff is possible. And yeah, that's something I also learned from my personal experience now. So not only technical stuff, but also social stuff and working with people together. So when you now do something with karma, what we have a dedicated channel in slack where we talk to each other, talk about features, we want to implement how to do it. And you also learn a lot about working in a team with people you usually don't really know. So there are people from India, from USA, from Australia, and they all come in the channel. I want to do this, how do I do that? And you just start talking with the people. And that also helps your communication skills, I think, which usually in a very technical field, are a lacking skill, I think. Beautiful, beautiful example of open source and all the benefits and yeah, not only technical, also the communication skills. That's really important. So yeah, I think we're coming to the end of the episode. We will link modern Python, we will link karma. But of course there are issues. If you want to contribute, you can. We even have a contributors guide, I think. Right. With some rules and stuff. Yeah, that's also something I really learned, what you have to do. What I want. When I use a package in Python, for example, I usually want to look at the GitHub page or the pipe I page and I really want to know in the easy way how to use it. But usually the very good packages, they are technically very savvy, but they don't have any explanation or. Yeah, no, no. Readme know how to use and that is also something we did with kamma board, have a really good readme to show people how to configure it and especially how to contribute. So what are the steps when you want to contribute? Clone the repo, install this tool, use it with this commands, install this tool, use it with this commands and then you're ready to go. So you have to really be precise because people will get, will get frustrated really fast. If you just, if they're just trying it and it doesn't work, then yeah, this sucks. Just get away with it. I will do the next project. It's also a skill you learn in open source the hard way. If people say I don't understand how to use it. Your packages is not good. Yeah, no, you went above and beyond with the documentation. It's really sublime. Do you know if it's running somewhere else? Bye bytes. I at least know about two people because they posted issues on the repo asking for sync. One was how to run it on Centos and there was a bug with Centos, I think, and we fixed it. So I think people are using it too. It's still a bit hard because it's still on an old API of slack, and slack makes it hard for people to use it. But I hope with the soonish rewrite of the slack API we will do, it will become easier because we have to do it now. Some of the API endpoints already fail, like the top channels command. So if you want to contribute, please come to me. We want to rewrite basically the whole project to make it work with the new API. So we need some manpower. If you're listening and you want to contribute to a major open source project with some complexity APIs, pytest and all the good stuff and modern python reach out to us on our slack. Cool, cool. Well, thanks for sharing all this. There's definitely some golden nuggets here. And yeah, thanks for all the work you've done and you're doing with the project. It's really, yeah, it's one of our staples now in pibytes, so really cool. Yeah, thanks so much, Patrick. This has been great having you on the podcast. We really appreciate everything you've done. So if anyone wants to learn more about you, they want to talk to you about karma Byte, get involved, all of that. How can they find you and reach you? So I think basically the easiest way is to go via the Slack channel. Just search for Patrick, the guy with the hat. And I usually are in the karma writing, the only one or one of the people writing in the karma about dev channel. So just look for Patrick in the karma about dev channel with writing a lot and then you will find me and yeah, that's basically the best way to contact me. Nice. Well, thanks so much, man. Really looking forward to seeing the new features coming out on Karmabot one of these days and appreciate your time. Yeah, appreciate your work. So I also want to give a shout out to YouTube because you built an awesome platform and awesome community. And I think without pibytes I wouldn't have gotten to the, to the level. I'm now at Python and working with communities, working on GitHub. So you also enabled me in a technical and in a social way. I'm glad to hear that. Thank you, man. That made my day. I'm just going to end the week now. Yeah, take your time for this week. Thanks Patrick. And just to wrap this up, anyone out there who's listening, this is my call to action for you. If you want to get involved in open source, don't be intimidated. As you can see from Patrick's story, just jumping in. People are more than happy to walk you through it, talk you through it, and support you with your contributions. Because after all, the more help you can give, the better, right? So don't be afraid, jump in, contribute, and obviously come and check out Karmabot. Come and talk with Patrick and see what you can do. Indeed, you can start your contributions very small. You don't have to rewrite the whole code base and work with the new slack API. Even in documentation fixes could get you into open source, but the important thing is to get started. That's one of the main takeaways. So again, thank you, Patrick. Great meeting you. Here today. And yeah, go write some code. Go do some open source. We hope you enjoyed this episode. To hear more from us, go to Pibyte friends, that is Pibit es friends, and receive a free gift just for being a friend of the show and to join our thriving slack community of python programmers, go to Pibytes community. That's Pibit es forward slash community. We hope to see you there and catch you in the next episode.