#110 - Dane Hillard on Python packaging and effective developer tooling

Talks allows you to specify all of what happens when you take specific actions for your packaging. And you can assign names to those things and make it really easy to orchestrate the different parts of the project lifecycle. Hello, and welcome to the Py Bytes podcast, where we talk about Python career and mindset. We're your hosts. I'm Julian Sequeira. And I am Bob Baldeboz. If you're looking to improve your python, your career, and learn the mindset for success, this is the podcast for you. Let's get started. Welcome back to the Py Bytes podcast. This is Bob Elderboss. And in this episode 110, we have Dane Hillard on. And Dane is the author of two Python books. First, he wrote practice of the Python pro, and lately he published a book on packaging. And that's very exciting because, as we know, packaging can be confusing, and there's a lot of tooling and ecosystem around it. And in this episode, we really dive deep into why packaging can be confusing. Some highlights of the book, the writing process, even a bit of mindset. So without further ado, here's my interview with Dane Hillard. Hello, everybody, and welcome back to the Pyewise podcast. I'm here with Dane Hillert. Dane, welcome to the show. How are you doing? I'm doing well, thanks, Bob. I'm happy to be on. Yeah, we're very excited to have you here for multiple reasons, one being the awesome books you're putting out and the teaching. But before we get into that, maybe you can introduce yourself to our audience and tell a bit about yourself, your background, how you got into Python. Yeah. So, my name is Dane Hillard. I'm currently a technical architect at Ithaca who is most known for the jstor.org research platform. And my first foray into Python started sometime before that, maybe ten years ago now. I was working in the space of sort of big data analysis, genomic data for personalized medicine, and I had previously learned a bit of Ruby after doing a bunch of C and C, and I was like, hey, these other languages are great. And then started getting into python through that kind of data space a little bit, and the rest is history, as they say. Nice. Cool. And what do you do right now, day to day? So, as an architect, my job is often consulting, if you will, with other teams in the, to understand both their business needs and their technical portfolio, to try and make sure those things align. And a big part of what we've been working on specifically lately is micro frontend architecture, trying to help decouple teams so that they can deliver experiences independently, sometimes many teams on a single page to produce a unified experience. So that's been a focus for me maybe the last year or so. Interesting, interesting. And it kind of nicely ties into your first book, which I got a lot of architecture out there, so a bit of background. Right? And think in 2020, you launched your first book, Practice of the Python Pro. I read it back then. I was really impressed how much it covered and how much great advice there is in there. Definitely beyond the beginner level for writing quality code and thinking about architecture. But in this episode, I want to actually talk about your more recent book, which I think just came out about packaging. And if I can quote from the foreword by David Beasley, a quiet secret of most Python books is that nobody really wants to talk much about packaging, which I can totally see because it's kind of complex and sometimes confusing. So I wanted to start off by asking you what made you write a book about packaging for sure. I started maybe, maybe five or six years ago, getting a little more into the open source python packaging space or the open source Python space, and worked on a couple of packages during that time, and then built a package at ithaca that we ended up open sourcing. And then we had a couple of, you could call them packages. They were modules, really, within our Monorepo codebase, and we decided to turn those into separate, installable packages as well. And eventually we were maintaining maybe a dozen or more packages internally. Over time, we built a pattern of practice that seemed to work really well across our distributed team. Wanted to share some of what I learned during that. The other thing is that people out there have valuable work that they're doing, and some of it's like hiding in source code, in GitHub repos, not available as packages. And some of that could just be that they haven't had the time to do it. Some of it could be they don't have the knowledge. And I really want to be able to give people that ability, if they so choose, so that they can share that value. And I think in sort of the computational sciences and data science space, a lot of people aren't necessarily pythonistas. They're using Python to get their real job done, which is science. And so giving them some resource to kind of improve this secondary skill, if you will, I think is a big, ripe opportunity. Yeah, there's often a big leap between people, scientists using Python to get the job done, to be dangerous, even. But then to packages up and all that stuff, it seems like a whole different level. And there's definitely, for a lot of people like this moment when they ship their first package. It's a crucial step in their python journey. So if we say packaging is hard, which we might debate or not, but why do people even say that? Why do you see people struggle with this? Yeah, and I think I was one of those people, maybe even around the time I started thinking about writing this book. But I think a lot of the, I think a lot of it is perception that it is hard. And some of that perception comes out of kind of the history of the tooling and the diversity of the tooling. There's been several kind of major eras of packaging tooling. And one thing people feel when they start to research this is like there's several different things documented or blogged about out there. Where do I even begin? And a lot of the work that's been done lately is precisely to kind of streamline that picture, standardize some of those things, build a more extensible architecture, because a lot of times the problem was also that they built something that worked for a lot of people, but it also had its limitations, as any system does, and there wasn't necessarily a way to build around or through those limitations with that system. So people built new systems. And so the work to kind of make almost plug in based architecture with some of these newer packaging tools has really, I think, been a great direction. And I hope that really it will result in a true streamlining rather than there's sort of the famous XKCD comic that's like, we have twelve standards and it doesn't work. We need another one. Now we have 13 standards. I don't think it's that situation. Yeah, yeah. The book is well timed. There's a lot of work going on and it seems that it's becoming more zen of python, where there's one obvious way to do things where that was not the case before. Yeah. Okay, so I wanted to highlight a few things from the book. I have seven things here, but we can change that. You can add anything, but those things I found interesting and maybe relevant to the topic. Project structure, often confusing for people, maybe more beginner to Python, but basically I just use package directory, a test directory. But since last week or two weeks ago, and I did an episode on packaging as well, I use the source layout. So you have an extra directory source and then package and then your modules. Is source layout something we should embrace? Is there, is there an advantage of that extra source directory? And can you explain that? Yeah, great question. I think this is another area where there's so many ways to do it, and all of them are everyone likes one of them, so there's a lot of resources out there advocating one or the other. I personally advocate for the source layout, and the book moves in that direction. A couple of the reasons that I think that's nice are that it allows you to put your tests in a directory. That makes it pretty easy to exclude them from the package. If you have a package directory and a test directory within that, it offers the same thing. I also think you don't run into that a lot in practice. Network and storage are so commodified that whether the tests are included in the package or not doesn't necessarily bother me as a consumer, but it can help some people out. And then the other big thing is that Python adds the current directory you're in to your import path. When you're developing and trying things out with a flat layout, you can trick yourself into believing import behavior is going to work one way when then you package things up, ship it, and find that it behaves in a way you didn't expect. So the source layout kind of forces you to use your tooling and be a consumer of your own package as you're developing it, which I think is a really good practice. Excellent. Good to know. Thanks. Then. Secondly, we have the Pyproject toml file. Before we had a setup py, the TAML file seems a lot nicer and cleaner, but that could be subjective. What is the advantage of that TOML file standard? Whatever you want to call it. Yeah, there's also the setup CFG file that was in between those two things. And again, I think that file is meant to be able to streamline and move toward a unified way of specifying project dependencies, tooling commands, tooling dependencies, and really make that a more consistent one stop shop for where to find that information. And it also works well in packages, just as it does in applications, which is nice. Not that they all couldn't in some way, but I've appreciated kind of the standardization there. The book does still use setup CFG. In addition, there at the time of writing was definitely still some things that hadn't transitioned support yet. So some of these things are still going to be in flux depending where you sort of find them on their maturity journey with the packaging landscape. But I think hyproject Tamil is a nice direction. There's some debate about whether TOML is the best specification language and all that, but I think it's often debate that isn't germane to the getting things done. Okay, cool. And then I had a question about general flow of packaging and shipping projects. I think the book covers mostly the most basic tool set of build and twine. But there are now also many, many tools you can use. Hatchling. That's the newest, I think it might be in the book, I cannot remember. But then you have flit and you have poetry and PDM. PDM. Not to be confused with our coaching program. I think the book doesn't really go into all these tools, and I think for the better. But in your day to day use, do you have a preference for a certain manager or tool? Yeah, I don't really use hatchling or poetry or any of those at the moment. I've used poetry in the past. Hatchling is really interesting to me. I'd like to spend some time with it soon. I honestly haven't, though. The book kind of takes the direction of helping people understand the core moving parts of the flow and offers links to some of these tools. To say, once you. Once you understand these moving parts, here are some branches you could go down and follow and see what works for you. Because understanding the fundamentals allows you to sit on that as a foundation and then comfortably switch between. Maybe you want to try five different things and see which one works best for you. I think without the foundational knowledge, you could mistake some of the nuances you're seeing as fundamental differences, when really they're an aesthetic choice or a process choice. So that's kind of why I stayed away from some of the particulars of those projects. Yeah. And I think that's great. Yeah, poetry is amazing, but it's also doing a lot of things in the background and magic. This week I used Django, cookie cutter. That's even doing way more many things, and it confused me a few times. So these tools are nice, but, yeah, definitely when you're teaching it, it's good to stick to the core and the basic concepts and, yeah, I think you did that very well. And there's also a lot of tooling in the book, like linters and testing. And what struck me was that you hooked most of that up with talks. So do you want to talk a little bit about that? Yeah, I think that if you're doing packaging, tox is an absolutely invaluable tool. Just as an honorable mention, too, that there's a tool called Knox out there, which is a spiritual successor, and I haven't tried that one much myself. But if you have ever used make, for example, in a project. Maybe I'm old too. I don't know who uses make today or not, but if you enjoy declarative documentation and declarative task management definitions, TOCs allows you to specify all of what happens when you take specific actions for your packaging. And you can kind of assign names to those things and make it really easy to orchestrate the different parts of the project lifecycle. And so I think that without it you would be resigned to running commands, remembering which commands to run, hitting up arrow all the time, to rerun things that you've run previously and just, it takes away a lot of that tedium around remembering what to run and when to run it. So it helps you build a recipe for success in that way. Really nice. Yeah, because I always associated talks with testing different versions of Python, which might be important for an open source package, but how you showed it in the book was really like, wow, this takes a lot of thinking away. It automates it and it's really nice. And I will say that's how I got into talks originally was the testing matrix aspect of it, which it's great at, but you then start to see like, oh, it can do, it can manage any of these processes. So yeah, that's good. Cool. Three more on the, on the insides of the book. Dependency hell, I think most people have bought with that, but maybe if people are not familiar, what is it and what are. Yeah, some tips you offer about how to best manage that because, well, I'll let you talk first about it. Sure. So the concept of dependency hell is that you depend on two different packages at a specific version or a specific range of versions, and then they themselves depend on other things, and sometimes those ranges may not overlap and then you're stuck because you're saying, I need these two versions to resolve together well, and they need other versions that need to resolve together well. And when they don't, typically you have no way forward. And there's a couple really good blog posts. Maybe I'll try and make sure they are in the links for the podcast. I'm not going to remember where they live, but some of the knowledge, I guess coming out of those is to avoid specifying upper bounds on your package dependencies when possible, because you then allow for the sort of infinity of the future versions of packages to be to be installed and only add an upper bound temporarily if, say, a new major version breaks compatibility with something you're doing in your package. You have to think about your packages that you ship as collaborators in someone's project and they all need to play nicely together. So try and be as open to whatever possibility you might run into after being installed into someone's project. And don't unnecessarily limit what is allowed. Yeah, it's a nasty problem and also hard to predict often. But one takeaway from the book is I was very strict in pinning everything, but you actually taught me about ranges and stuff, and I'm loosening up that a little bit thanks to the book. So it's still something I'm working on too. I mean, there's a certain paranoia that comes with things, and I think a good sort of testing flow is one thing that definitely helps with that. If you can regularly test against new versions and make it easy instead of costing you an hour or two each time, you free yourself from a little bit of that paranoia. That's a great point. Hence the talks, right? Yeah, cool. Get up actions. There's also some cIcd in the book. I'm happy to see that. Anything you want to say about that? I mainly want to say that I think the spirit of CICD is also a big part of what drove the book and drives my day to day life. Because as a maintainer of software of any kind, there's so many aspects to it and there's so many things that can go wrong if you're doing them manually. That having an automation process in place that codifies what it means for that software to be successful is just such a time saver. And really, if you want to talk about, I don't like ten x engineer phrasing or anything like that, but if you want to talk about boosting your productivity to that degree, like CI CD is one of the best things you can do, even for small projects. Because if you have ten small projects, that's the same or maybe even worse than having one big project to manage because there's overhead to each of those projects. So GitHub actions, or Jenkins or GitLab, any process you can use to put Ci CD around your project I think would be a big benefit. Yeah, and it's easy to set up. It's free. GitHub comes with great templates you can just immediately implement, and that makes it pretty straightforward. And as you say is your bucket of fries is gross, right? You want to have as much of that automated because you will be working on our projects. Right? So the more that other stuff is in standby, the better. Yeah. Cool. Lastly, on parts of the book on highlights is very geared towards open source as well, and contributors and people working together. There's even nice guidelines in there, how to make a contribution file. That's kind of stuff you might not immediately think about. Do you have any tips for open source contributors and maybe can talk about the community aspect of packaging? Yeah, I in some sense feel unqualified because I am not a big open source maintainer. I tend to put most things I build into open source. The website for the book, the website for the previous book, my personal website, all that's open source. But there is a different. There's a different thing that arises when you're working on a lively project with tens, hundreds of contributors. And I don't necessarily know what it's like to be the owner of something at that scale. But for open source maintainers, I think the big takeaway or the big spirit of the book toward that is definitely what we were just talking about around CI CD and getting a lot of the logistics out of your way so that you can make sure the package is delivering the value that you want. And for open source contributors, my advice would be if someone has that flow in place, make sure you use it and leverage it to your advantage. It can reduce a lot of the kind of rework and discussions and your pr staying open for three months. And if you really want things to land and be able to see, I did that in that package. Try and meet people where they are, which goes both ways. But I think it's easy to make a change that you think will work and then you push it up and open a pr and the tests break. So if you can do a lot of that checking upfront, it only helps. Yeah, yeah. Cool, cool. Okay. Well, I hope you liked that selection of highlights. Was there any other unexpected thing you learned from writing this book? I think I was just newly amazed at the things that are out there. The history I for the sections on metadata specification in the book, I looked through all the peps that ever talked about metadata and it was really just cool. I think having a reason to go do that and to really be forced is the wrong word, but have a driver to go look at all those peps and kind of delve into the history of things in a sense. I like recommend that to anyone who might be in this space at all. And I would like to do it maybe for other aspects of python as well in the future. Yeah, nice. Well, thanks for doing that. That definitely made it easier for us because sometimes reading the paps cannot be easy. Definitely. When I read the book you just see that there's a lot of research that went in and on that topic of writing teaching. So we already established it's quite a massive topic, can be confusing. Change the standards. Also, kudos, because it's so fast paced that it's just writing a book on this is a challenge already. But how do you do research? And more broadly, how do you keep up with Python's trends in ecosystem? Yeah, I thought about this a little bit the other day because someone was asking me how I keep up with things in general in the tech space. And for me, a lot of it is just finding the right sources of the information, like the canonical sources of the information, and trying to build those into my sort of passive intake. And that means joining the Python mailing list, being where Python people are in tech, Twitter and Mastodon and things like that, and finding people's blogs who you respect and adding those to your feed reader, for example. So a lot of times I'm becoming aware of things just because something flew by my screen, if you will, saying Brett Cannon has a new blog post about unraveling some syntax, and you just sort of branch out from, from there. And as you're reading a post, you might say, oh, they're talking about this other thing. What is that? And you just sort of delve into things that way. The classic kind of Wikipedia problem, right? Yeah, yeah. So a lot of it is, is that passive intake, and then when it comes to active research, like I said, I went through the peps. For example, I use a tool called Obsidian, which a lot of people are buzzing about lately, which is just a bi directional linking, note taking tool. And I heavily used that for the book writing. I would take notes on individual peps or individual tools and link those together where there were relationships and just try to understand how it's all connected. Writing those things down in your own words than too, allows you to better digest and internalize that information. So that, I think is like a tip for anyone who wants to write is first write about the concepts individually and the relationships between them, and then when you want to write it for someone else's consumption, you have something to say. Yeah, yeah. I think other books kind of come from a whole series of blog articles. I think the flask mega tutorial by Greenberg started like that. So there's definitely like research, digest your thoughts, be it in notes or blog posts, and then go for the big goal. Right. Otherwise it'll be difficult. Yeah, yeah. What are some hard and fun parts of the writing process? Was there a lot of editing involved, or. Yeah, yeah, that's the reality of the writing, for sure. I suppose it depends who you work with if you're working with a publisher. I know that both my books were with Manning and their editorial process is fairly robust, and I think they do a great job of making sure that you continue to teach to the level of the reader that you mean for the book to be for so upfront. Even when you propose a book, they kind of ask you, who is this for? And throughout the book process, if you start adding too much context and your book is for experts, they'll say, you know, hopefully the experts already know this aspect of what you're teaching. Maybe you don't need it there. Or if you're not adding enough context, your book is for someone newer to the space. We'll say, you need to bring this concept in first before teaching these advanced concepts, or you need to do these in this order to make sure that they're introduced in a way that someone can follow it linearly. And I appreciated that kind of aspect of it. I think it's both a hard and a fun part because it's really challenging to find the right way of teaching something effectively. But it's very rewarding to find that, you know, whether it's the phrasing or the ordering of that information or visual aids, it's great to see someone say, wow, I never thought about it that way before, or I really learned something here. So it's hard and fun. I enjoy the diagramming part especially. I find those are amazing. Making a visual aid. Oh, thank you. I'm a little sad the book has to be printed in black and white because the full color diagrams are really, are really great. But being able is for that. What's that? I got him an ebook. Yeah. Yeah. The picture is worth a thousand words type of value. Right. Finding the right way to say I can communicate this in very high bandwidth just by showing you rather than having to tell you about it is great. Now, that's awesome. Also, I want to read it again, but 400 pages or whatever it is, takes some time. And actually I can just now go back to these diagrams and they're great summaries. And I think there are like 1015, 20 of those. There are quite a few in there, right? So pretty detailed. So yeah, the book is only 250 pages, I think, so I'm glad you felt like it was 400 pages worth of material. Not sure if I insulted you now, but. Yeah, yeah, that's great. No, but there was a lot of content in there, so if it's. Yeah, being at 250, then I can definitely attest that there's a lot in there. Wow. I thought it was more pages. Cool. So, yeah, I think we're about to wrap it up. Maybe. I have two more questions, one mindset, and then we always end up with books and stuff, what you're reading. So I can imagine writing a book on one of the most confusing, complex topics in python right now, and there's a lot of at stake. Right. So, yeah, I can imagine that that can trigger a bit of imposter syndrome and stuff. So was there a mindset piece to this whole book writing and how, if so, how did you cope with that? Yeah, I would say yes, definitely. There. There was. And even still is, I mean, the book has a perfect five star rating on Amazon right now, hoping to keep the streak alive, but, uh, it. You're sort of always waiting for that stream of comments to come in. Like, what are you talking about? Who even are you? Go away. Uh, and, you know, they could be totally right about that, too. It's. It's just, uh, such a. Such a weird thing to try and kind of have something to say, but also be, let's say. I'm just saying what I know. I I'm not trying to say I'm right. Uh, just putting knowledge out there and. And so I just like to share what I've learned and what's helped me, hoping it can help others and it won't help everybody and you kind of have to be okay with that. And just, there's a little bit of. Just be. Just have positive intent and make sure to try and not lead anyone astray if you can. And that's. That's sort of the best you can do. So. Yeah. Yeah. Cool. Great mindset. Yeah. Because inevitably there will always be somebody that's going to hate it, and that's just fine. We always also like to say, like, well, if at least you want to have some polarization. Right. Because if it was all, like, people nodding along. Yeah, sure, sure. Then maybe it's not that meaningful. Right? Yeah, yeah. Maybe you're not really saying anything at all. Yeah. So a good opinion here and there is appreciated. Um, cool. So lastly, what you're reading. Yeah, I am reading a couple things right now. So the one other thing we're working on is kind of upping our, our practice around experimentation on our platform, like ab experiments, hypothesis testing. So I'm reading a book called Trustworthy online controlled experiments, which is from a couple years ago, but really good text about what it means to run a proper experiment statistically, and what kinds of pitfalls there are in that analysis, and what is an organization you need to do to understand your data? And even before running experiments, make sure you know what your metrics are. And I've still got probably a half or more of that book to go, but it's a very valuable text if you're interested in experimentation. And the other thing that I read recently is not a book, but Stephen Wolfram's explanation of chat GPT I think was a really informative work. It talks about essentially how large language models function. And if you want to demystify chat GPT for yourself a bit, I think it's a good one. And then sort of on the totally non tech side, I'm reading this book that I actually have had for many years and trying to trying to finish that's about cheese making called reinventing the wheel. And it talks about kind of the history of european cheese making and how it differs from american cheese making and the economics and the politics behind it. And it's a really cool book. So cool nicely. I want to take you up on that chat GPT text, so if that's online, we'll link that below because I was actually asking myself the same question today. Like, how does this even work internally? Because yesterday I was using it and this is mind blowing, but it's already capable of doing. So thanks for that. Yeah, now really coming to the end. So I still have two more questions. So where can people find you? Reach out and any parting words, call to action, or even maybe something you were not in the opportunity to say and you still want to share? Yeah, up to you. Sure. So my Twitter handle is easy as python, and my GitHub and fostadon handle is Dane. Ah. And those are probably the best places to find me. You can find the book@pypackages.com that's pypackages.com. And you can, you can get the book through the links there. You can also redeem the book. You can redeem a code for the book or for any other manning product. 35% off with the code. Pod pibytes 23. That's pod pibytes 23. Nice. Please check it out. Yeah, we'll link all that in the show notes. And yeah, thanks so much for coming on today. Thanks for sharing, for writing the book, as I already said, but also for sharing all your insights on this show. People will appreciate it. And yeah, thanks so much. Thanks again for the opportunity. Have a great day. Bye. We hope you enjoyed this episode to hear more from us. Go to Pibyte friends, that is Pybit es friends, and receive a free gift just for being a friend of the show. And to join our thriving slack community of Python programmers, go to Pibytes community. That's pibit es community. We hope to see you there and catch you in the next episode.