Pybites Podcast

#170 - Advancing Python Packaging: A Deep Dive with the Pixi Team

Julian Sequeira & Bob Belderbos

In this episode, we reconnect with the Pixi team after our last conversation in November 2023 to dive back into the world of Python packaging, with a special focus on the latest advancements in Pixi.

Joining us are Tim and Ruben, both of whom bring their unique expertise in robotics and game development. Ruben shares his experience contributing to conda-forge, pixi and Python package management in general.

We explore recent enhancements to Pixi, including its integration with VS Code, the evolution of pyproject.toml support, and the addition of PyPI dependency resolution along with Conda.

We also discuss the current state of Pixi for enterprises, touching on its readiness for production and enterprise-specific features like package registries.

The conversation also delves into how pixi resolves issues reported by the community, such as platform-specific Pixi.lock file regeneration - and how developers can contribute to the open source ecosystem.

Finally, we look ahead to the future, discussing the team's vision for Python packaging over the next one to five years.

Tune in for a deep dive into the present and future of Python packaging, packed with personal stories and expert insights.

--

Links:
- Prefix-dev
- Add pixi as an workflow/package manager PR
- Our PDM coaching program

Package, a perfect case or SDL or any graphics library or something like that. Being able to get, like, an isolated environment for that means that you are able to run it on any machine. You were right that there are some. Some of these manual steps. Right. That you always have. I have to install open SSl and this way and this and this one and that. And with pixie, you try to alleviate this. And I even have had to get used to it, that I would see an issue, and I probably can't reproduce it because I'm, like, used to not being able to reproduce these things. But then there was a pixie file and thought, no, I can just run it and start, which is like, kind of like, great. And once you get used to it, you don't want to go back. And I think that is something we would like to change. Like, so many computer science programs where people have, like, one whole day of setting up their PCs and someone is using a Mac while the other ones are using windows, and we just want to get rid of. Rid of that frick shit. Hello, and welcome to the Pibytes podcast, where we talk about Python career and mindset. We're your hosts. I'm Julian Sequeira. And I am Bob Valdebos. If you're looking to improve your python, your career, and learn the mindset for success, this is the podcast for you. Let's get started. All right, so, hey, Ruben and Tim. Nice to have you here on our podcast. Yes, thank you for having us. So, yeah, it has been a while since we last spoke with Wolf. It was in November 23, actually, about Pixie, and back then, I was already, in terms of the idea, convinced that Pixie has a bright future. Now, we have been a couple of months, and actually, it's like August 24 right now, so it's not so long time, but there's been a lot of changes happening, and that's why we wanted to have you on the podcast again. And we're really happy when you accepted us the invite. And now, yeah, we would be looking forward to talk a little bit more about what are the new features and also who is behind Pixie. So what is the team of Pixie looking like? Therefore, I'm happy to have you two on the podcast, and maybe you can just quickly introduce yourselves with a couple of words, and then we move on to the topics and questions we have prepared. Yes, thank you. I can go first. My name is Rubin. I work at prefix since a little more over a year now, and previously, I've always been in robotics and that also spawned the requirement for having good installations of your computer because that was pretty hard. And that spawned me into the package management area, which later became my new job at Prefix, where I mostly maintain pixie right now. So that's also why I'm here. Nice. Hi. So, yeah, I'm Tim. I studied at computer science at the university and I was first really into gaming, so I worked at a gaming startup, but my graduation was similar to Rubens in robotics, so we ended up in the, in the same company. Like Ruben said, like installation robotics is notoriously difficult. And yeah, I was like, when I was working in games, I really liked building tools, like building tools for other game developers or artists, and that's what I did a lot. And we worked on our own game engine at the time. Yeah, I got really interested into packaging, which sounds weird maybe, but it scratches some of the same age that the tool development had. And so I was glad to move to prefix, which was then founded, and I'm having a blast since and it's been really good and also contributing to open source more. So that's been great. Nice, nice. Do you have some latest wins to share with the audience? I can maybe get going in the beginning. So now working a couple of months in the new startup and we recently had like a first live interaction, let's say, with the system, and that was really exciting. So, yeah, basically it's always, I think it's always nice when the software is not just software, but also having some real world application or effect, basically. Yeah. I think for me, I hope for Ruben personally is like, just the other day we were GitHub trending with pixie for rust projects at least. So we're like in the fourth place, which like, I couldn't have really imagined anything I made ever being there. So that was pretty cool. I had to say. I immediately send it to like, people that like, that we work with like marketing people in the US and stuff and then, and they were pretty excited. So it was a win, a definite win. Nice. I would fully agree with that one. I have one more to add to it that's that I have been to Sci-Fi in July and I've met a lot of people there, which was awesome. And then I obviously told all of them about Pixie. And now I'm still getting messages from people that tried it out and also agreed with me that it was pretty cool to use and they're going forward with it. So that's a huge win for myself to see those people get on the pixie, awesome. Yeah. And maybe we can talk later a little bit about your journey in open source and basically creating something that then people actually use, because that's something that our podcast listeners also care about quite a bit, like how to get started on the one side with programming in general, and then maybe when they create some packages, how to maybe set it up so that people actually use it. Right. And how to talk to people and what to build and all these kind of questions in general. Yeah, basically, maybe a couple of things. First, to just get started, why it's interesting to already have another podcast after only eight months now or nine months, there have been quite some improvements in pikce. I noted a couple of them. Maybe you have some more to add. But for me, something that was really exciting was on the one side, the visual studio code integration, until now you had to manually set the path and so on. Just recently, a couple of weeks ago, it's basically automatically detected and works really well. I would say so, yeah. I think that's also just another step towards showing that it's really mature if versus code integrates it and you can use it seamlessly. So maybe do you have something to add there, like background story? Also, how easy was it to integrate? I just followed it loosely and saw that there was quite some changes on the versus code side where you would have thought that maybe it's very simple change to just add that. But it was quite a big pr, so maybe if you have something to add on that from your side would be interesting. Yeah, I could definitely extend it a little more. To be fair, the big pr was written mostly by our colleague pals, and he did an awesome job on the typescript that is needed to make an extension or edit an extension. This particular extension that we added to was the Python extension from Microsoft. And to get into that you need to just make a pr on their open source version of that project. What was definitely good for us is that it was already done for other package managers, so it was already done for of course Pip poetry and kumba itself. So it was basically kind of redoing it again for the Pixi package manager. So you need to tell the extension how to find Pixi and if it found it, how to interact with the API of Piksi, and then from that get some information that's then automatically set up in the extension itself. So in theory it's not super hard. It's just a lot of places where this touches the extension. So that all had to be modified to also accept pixie information. Nice. And another one that I noted was also the PI project tamil support. So in November it was basically like there's the Pixie Toml config and that was the only place where you could set it up. So then the question was coming like how quickly can you integrate it with PI project TaMl file? And I think it was really fast. I don't know when it happened, but I don't remember anymore. But it happened soonish afterwards I would say. Yeah, so that's actually a great one to talk about because it also gets into the open source bits a bit more because mostly that work was done by Oliver Lacroix, who works for a company and is based himself in Australia. And so he was willing to pick that up and he made like this nice abstraction where you kind of choose between the two manifest versions, but you can reuse kind of like the same data types. And there had to be some decisions made. Like we have to interpret this a bit differently, or does that a bit differently, or we have to add this by default. But it was so great that he was like super quick and he's such a friendly guy and like awesome to work with. Like if I send him something to test, he tests it. Like, he does that work? Like he, uh, yeah, he works and I think he runs in Australia and then he does the work in the off hours at night. Right. So sometimes he just like does this big push of a lot of work and sometimes he's traveling and then he gets to it a bit less. But it's just like so awesome to be able to have these people and contribute to it. So I think he did most of the work, so a lot of credit should go to him and a lot of people have benefited since. Awesome. Yeah, I think that's definitely deserving a shout out. And as you say, Australia, that may be also something for Julian to host a podcast session with him, maybe in person or so in Australia. I wouldn't mind do it either, but it's just a bit more traveling, let's say. Yeah, so that could be interesting. And that's been great. Like we've had more people contribute and like have some regular people coming back. And it's just like, what I try to do now is, as a general thing is like when I see like some contributions, I contact these people personally, I try to contact them and see if they want to have a talk or even a call so they can also explain their use case and why they're interested in pixie and the positive experience. For me has been that most of them are like, totally, I want to call in an hour or something. Right? Super cool. So I felt like maybe I'm imposing a bit, but I haven't found that. Nice. Nice. So how does it generally work for you in pixie? I mean, you now mentioned that many times open source developers do their normal job, whatever it is, in science and industry or so, and then on the sidelines they also work on open source. How has that been for you? Like the journey towards open source, let's say, how does it look like right now? How much do you work, let's say, on pixie topics? And how much do you work on open source topics that are maybe related or completely unrelated, that are interesting for you? If you want to talk a bit about that in either order, Tim and Ruben, you go first. Ruben, thanks. So for me, I started with it when we started to work on package management at the robotics company I mentioned before where Tim also worked. There we did a project where we moved everything from super specific opt pip and some other tools meshed together into the whole Konda ecosystem. And there we required to update some of the packages. So I think one of my first real contributions was getting packages onto Konda Forge, which feels small, but is actually very helpful if you start with that, because you can very easily maintain it yourself while also helping a huge community at the same time. That moved me a little more into helping with these smaller projects around Konda and Kona Forge. And actually really quickly after that, I already started a prefix, so I haven't done a lot of open source in the company I was before, but it kick started there and I completely moved to the open source work after that. And currently at prefix, I think 99% of my work is in the open. So it's now completely my full time job. We of course, have some of the contributors that don't have full time job in open source, but then sometimes get time from their company, like, oh, we really need this feature and this tool. And then they contact us like, hey, I want this feature. I could build it myself. Is it allowed to built in? And with a little call we often find an agreement and then they develop that part of the feature in their company's time, which is also always a great experience because you don't have to do it in your free time. You can actually open source in your paid hours, which is always the best. And that's of course something to maybe not so easy to ask, right? If you're an employer of your company and you employee basically of the company and you maybe have that idea, but it's also not so easy to communicate to the managers. That's maybe a whole topic for itself, how to best do that. But yeah, it's nice that it is possible. And maybe when you, as a podcast listener hear this, maybe you also see some ways of how you could do that at yourself. Tim, how about you? How was your journey? So I had some small open source projects that I did, but just for fun, more or less. And then sometimes I made a contribution here or there, but not really significant. And then, yeah, actually prefix is just also my first time doing big open source work. Like at a previous company, we had some things that I thought we could potentially open source, but as you were saying, for an employer, it's sometimes difficult to see like how that could benefit them. And I think that's like something we struggled with. So yeah, I think that it's like if you get the opportunity to do something with a company that does open source, it's like the easiest way to get into it, like as the fast path, so to speak. But on the other hand, there's also, of course, a lot of people that do it in their free time, like as I said before, or where it's directly beneficial to the company. But now that we have worked in open source a bit more, we do see quite a lot of companies, even big ones like Airbnb and meta and things like this, that do, yeah, get dedicated time to do open source, and I think it benefits them. So I think as a company you could take note of that. And if maybe as a listener, you could look at one of those programs and see how they work and maybe try to implement that. And especially, I think, focus on things that are not in the critical path, like that are really needed, but not really what you want to sell. And I think that's just good things to open source. But yeah, and since then, most of my time has been in open source as well. It's just like almost all of our products are open source currently. So there's that. Do you have some other things that were added to the Pixi editor or prefix in general that you would underline as new features since November? I mean, maybe it's not so easy for you to think, which were the diffs, let's say, since then, but if there's something would be nice to add. I did a quick search on what that could be, but I saw that when you talked with Wolf, we already had a pretty good idea of what we wanted. So a lot of the work was just improving on top of what we already believed in. Some things we would like to mention are also coming from open source contributors. Doing a complete proposal is the multi environment and solve group tooling, which means that you can define different environments in a project. You could see this as the optional dependencies for the Byproject Automl, for instance, where you say oh, I also have a set of dependencies for the test. I also have a set of dependencies for my CI setup and pixi then can solve them as one and then create different environments where you meet the specific dependencies and it all gets locked into your log file. This also helps a lot with making sure that what you test is exactly the same as what you run on deployment or NCI or wherever you run your project, because we are able to solve the dependencies for the different environments at the same time making sure that the solve works with the complete set. But you don't have to install everything all the time. A good example of this is when you lint, you would like to use the same linter as in your development environment, but the linting environment is just the linter and not all the dependencies of your project, which is always cool. We actually made a list for extra topics, but maybe this is also the right time to mention the pipe I integration, because I believe that wasn't done totally. You're right, yeah, with the last podcast. But I would like to ask Tim to explain more about that. Yeah, so as I don't know what your listeners know, but you have like the pipe ecosystem, which has like a collection of wheels. Pipe has gone a long way, right? It used to be like this simple source things, source python distributions and then set up PI, and then it went through a lot of iteration and people often make a bit of fun of the python packaging part, that part. But they were also kind of like the first ones together with Burrow and like these other package managers that had to figure all this stuff out so you didn't have the benefit of hindsight. I think they've done a good job. So that's one ecosystem, so that's mainly wheels. And then you have Konda, which are Konda packages. And people associate that with Python a lot, but it's actually general packages. So it's more like, in that sense, more like deviants than wheels, in the sense that it can contain any kind of binary library or Weatherford. And the question since the beginning of Kolna is like, okay, this is not available as a Kolna package, but it is available as a pipi wheel. Let's say I want to install it into the environment, which seems easy enough because if you use a conda environment like the right python paths already set up, you will have a side packages folder. You have something similar to a vm, so it should be able to so you can install into it directly, but because of mismatched independencies, it can mess up your environment pretty quickly. So we always had the idea to like okay, let's do this a little bit better. So initially we had our own solver and reader from PYPi that could like take a set of locked kotlin packages and say I want to install this PI PI package. Does it satisfy the dependencies in some way? Or if it's not, then yeah, I can install, which is a bit better than just trying to install it and see what happens, because there can be a lot of incompatibilities. For example, wheels often package their shared libraries next to their c libraries next to the Python libraries, and then use a comma thing. It might not expect that at that point, or might expect something different, and that you would get a break at runtime, which is pretty annoying. So we made progress there by, like I said, looking at the comma packages first using that as like the locked versions, then at least knowing then the solver knows what was already installed, and then installing the pipe on top, like resolving and installing them on top. So we initially used our own project called Rip for it, and then uv came around the bend. Maybe you can talk about it a bit more later. And then we actually integrated with that because it was also a rust project, and we have our own way of noting down these dependencies. And this also helped with the PI project tomorrow because we could use the dependencies from the piproject, interpret them directly as pipe bi dependencies. So you could essentially use Pixie as a wheel only installer, but you lose some of the benefits. But that made that integration a bit better. Yeah, and I think that was a huge step, because like you said, sometimes packages are only available on Conda, or if you would want to install them somehow with Pip, you need to install some other dependencies on top. Other times you want to have the latest version which is only available on Pipe AI for Python. So then you have to go with Pip. And so yeah, that was great. That was the addition. And I also like in the previous podcast episode I already said that I like the combination of Conda and Pip, because normally I would always use Conda so that you can manage Python version, you can install Pip, you can install conda packages, everything together, whereas, for example, with poetry and so on, you only can install with Pip. At least to my current understanding. I think they didn't edit this support yet. And so I like that pixie also brings in everything from the two worlds, but also the usability of poetry. And I guess there were probably quite some, like, making Pipe AI available, or was it just smooth like you just said? Okay, using the PI project tumblr. And then a couple of things needed to be added to make PI PI packages installable on Pikce. Well, the funny thing is, some of the general concepts are similar between Koma and Pipi, but some of them are very different, like the way you get the index data and the way it's stored. And these are quite different, but when you get to the solve step, they are very similar. So when we were still using Rip, our previous project, that we could use the same software as we did for Komba, which was general enough to modify, which kind of gave us the idea, like maybe every packaging ecosystem has this kind of like, minimal thing that you could reduce it to, and then you could essentially solve every package together somehow. That would be kind of cool, but that at least worked for Kona and Pipi. But on the other hand, there were like, loads of different concepts that Kona doesn't have. For example, environment markers and wheel tags and things like that that you have to take into account and you have to, like, derive from your Kona environment. And there's like, yeah, so the integration of the thing is pretty difficult, but if you go live to the low level conceptual solving part, it's. Yeah, other than that, getting the metadata is different. That is quite similar. And I think most of these packaging ecosystems can be reduced to this, looking at this in some capacity with different requirements, of course. So I think that was very interesting to learn also, and very interesting to find out what the differences were. And also my team was super combat based. Right. And then they were saying, yeah, well, actually I've now learned today I've learned this and I've learned that. And actually, I quite like that, how they made these decisions, like, maybe it's something we could also learn from again and bring back to Konda. So that was also good. I think this cross pollination is good. Yeah. And I like how, for example, and fast API, Sebastian Armira said, hey, how these are all the other frameworks that usually do something like backend development. And I got inspired by this framework because they did this nice by the other framework, because they did that nice. And I think that's also something. I think you already started that on the pixie documentation, but maybe it could also be interesting to add all the different things that pixie basically combines from the different package managers and makes it therefore even nicer because it can build on top of giants, let's say, on the one side, and learn from new methodologies, let's say that weren't available, like you said, for the pipe AI. People who set up pipe AI, they were the first ones to do it, and they didn't have the benefit of hindsight to learn from others. Yeah, for sure. And that, I think you're making an interesting point, because we are, on the one hand, and you were saying this very correctly, so I just repeated failing on the shoulders of giants, which are Konda and everything, Konda fortune. But we also, when we started the company, we really liked the workflow of poetry, we really liked the workflow of cargo. And we thought, like, why does Kondlan not have something like this? And then that's how we kind of, like, looked at other things and thought like, hey, how can we use this? And even like, some of our command line flags are directly inspired by cargo, for better or for worse, because they're also sometimes seem to be a bit unclear if you don't know anything about it. So there was a pretty good. Sorry, Andy. No worries. So, yeah, basically talking about how you brought up a pixie prefix or how it was founded, maybe can you tell the story how you came to get and. Yeah, then I have a couple of company or enterprise related questions as well afterwards, but maybe just starting with that. Yeah, sure, I can answer those. Of course, the startup prefix was together with Wolf boss and Tim, Wolf being our CEO. Boss and Tim worked at the company. I was also working at previously at the robotics company because we were working on package management there. We got in contact with Wolf, who was working on package management, but also specifically for robotics. So it was kind of a good group. Then Wolf said he was going to start a company in this space and asks, asked boss to join and then also asked us to join. I didn't immediately start with the rest of the group, so they worked on Rattler and other projects the first half year. And after half year I also thought, ah, maybe that is the better option. Super fun to work in the open source. So I also joined the company and that that was basically the first year. And since then we've slowly grown trying out some freelancers and now actually also hired two new colleagues within the company. So we're currently with six people that work on our products. Nice. So that's also, I guess if people are interested in packaging in general and so on, then they can just start with the open source contributions. And how is the situation? You're looking forward to grow more and do do more and I guess there's enough problems to solve or how does that look like? So one funny thing I think is good to mention if you are interested in going into the open source world. We hired everyone. Everyone that was hired later was hired by first seeing them having a pr on our project and actually see what they did. And in that way we got in contact and became more interesting and the communications were nice. So that kickstarts your conversation about the job with an open source company. And I would definitely say going forward that's always the best way to approach us or any other open source company. I did forget your question. No, that answered it perfectly. Yeah. So basically if the listeners are interested in Pixi and maybe also even going to that job or any other company that is related to an open source project as well, that really can pay off or it can really make a difference to contribute open source, because then you may have doors that open up that previously would be hard to find even let's say similar story. We hear it also for fast API with the contributors that contributed there. It's good to hear that. Another related question is for the enterprise part, you're also now an enterprise, let's say, I remember a meetup we had here in Berlin where someone was asking, hey, is pixie already enterprise grade ready? Let's say, because it's somewhere in the documentation it says alpha or maybe beta or so it uses with caution. How is the status there? Do you already have enterprises, let's say that you closely work with or how is the pipeline there? How is your vision in terms of enterprise features and so on? What could be added and why would it be interesting for enterprises to work with you? You want to go over that one thing? Yeah, so thank you. Because yeah, in my mind it's definitely ready to be used in production because it is being used in production. It's also being used by enterprises. There are even like some meta projects you can find in the wild. And we've had heard from other companies like Airbnb that have used pixie. And more importantly, what I found out is that enterprises, they can have huge developer operations or tool builder teams. We have this open source library called Rattler which underpins pixie and other things that we built. We built bosses work mostly and I helped a bit at the beginning before we started prefix just for fun. And then we ramped it up when we started prefix more. And Wolf did a lot of stuff there too, and still doing a lot of stuff, so. And enterprise actually using that to build their own tools again. So because they have maybe ten times development that we have. So that's an interesting thing. So that library is very becoming, very battle hardened, I have to say. But also Pixie, there are companies that have switched to it, like most of their projects have switched to pikce. We've that, that alpha mutation, like very actually so timely for you to say, so. Yeah, for sure. And there are some things that would be interesting, might not be interesting for the open, like people working just on small projects or open source. And they're also not something we would like ever. We just want the package manager to be open source. So it would also always be like kind of related to it more or less. But you hear things like, like scanning for vulnerabilities for automatic updating base for large package repositories on the repositories, or very specific builds for architectures for speed. There are also things like package signing six store and that's also a difficult one because you want the part of it just to be open source and available. And there might be a part that is like creating an s bomb or something like that. That would be like something that you could potentially market. So these are things that are, yeah, we're looking into. We still have to look into it, but those are features that could be interesting for enterprises that might not be interesting for just regular users. Nice. And maybe that's a good point of time to just showcase a little bit how PKCE would look like on a package that you already have. Right. Let's say you are an enterprise or just a developer doing in just twelve weeks. Pivots elevates you from Python coder to confident developer, build real world applications, enhance your portfolio, earn a professional certification showcasing tangible skills and unlock career opportunities you might not even imagine right now. Apply now at Pibit DS PDM for programming and you already have requirements file or maybe Conda file or so. And how could it look like that you switched to the new Pixie package management? How much effort would it be? I think you guys prepared something for that, right? Yeah. So I didn't prepare anything to show directly, but I can take you through creating a Pixie project from one of your actual projects from pybytes so pybytes has this repository that's called search and I can show you how I would go about making this a pixie project. I can copy paste this. So I'm going to git clone the search repository from your GitHub repository and I'm going to go into the repository. So cd search typing, not the best. So we're now in your project and if I do small ls, you can see there's also a project tunnel. So if we're going into that you will see that it has just the basic setup of any projected tunnel by project, including hatchling as a build system, some dependencies like requests and typer and I think I can go a little lower. You have some optional dependencies for the testing and some optional dependencies for the tools you use to lint and format and the rest is just a general normal project. What I can do is I can call pixieinit without any specification. This will generate generate me the question like a I find you have already by project. Do you want me to extend this with the pixie configuration? If I just say yes, it adds some tables to your pipe project optimal so we can look at it again. And if I go down you will see Pixie has added the tool Pixi tables. We have a project where we define where Pikce can find the packages it needs. I'm currently on Linux, so it also specifies that this project will work on Linux which results in also creating a web file for the Linux platform. And then you can already see the environments that it added and in those environments it will install the required Python that you defined in your pipe project toml. So it will get you the python that you need and it will also install that project itself as an editable dependency, as an editable pipe guide dependency, which is kind of the basic workflow. If you would do a pip install e of your project it will get the dependencies, except pip cannot install Python. Pikce will also do that for you. So if I then do pikce install, I already start creating a log file. Pixie works by creating a log file first, then installing the information in that log file into an environment and that's currently done. So if I now do a pixie run and then I believe it was search go to help command, then it already gives me the command from the search repository that you built. And something interesting about Pixie here is that if we go into the pixie list, you can see it might be a little too small. Let me grow the list or, sorry, a little too big. Let me slim the list a bit down. So you can see we have installed Pybuyte search as an editable installation and we've installed some Python tooling from comma packages. We've also installed some stuff from Pypi. And this is all in one list that you can verify your project on. What you can see is that it only installed Python request typer, so the main dependencies of the project. But what we can also do is pixelist and then environment test, which is the other environment that we also created for you, which is including the pytest and including docs and including some other tools I presume that you need for testing. And if you run any test commands you could run in that environment and then you will have those environments, sorry, those dependencies available. One small thing I could add to this is we also define our own tasks and we have a task system which allows you to create a pixy task. We can add one for the search command that we just saw. So if I would do pixie task add and then the name search and I would, or search byte, I believe that you guys have, I can do search byte as the command. And do you know any examples that I could show? Oh no, you got me. Maybe just like a number I think, or you could just ask. Yeah, maybe let's try fast API because there we should have something. Let me add that as a task. And if I then do pixie run search, it autocompletes immediately and it will run that search for me, which is a more complex task than the single line that you need to write. With pixie run, this can be much more complicated. So I could add more dependencies to a simple, simple task. So if you have a stage where you need to download some data, you need to build a model and then you need to run that model. All these tasks could depend on each other and could actually have caching behind it to make this super powerful. This is a simple example where you can just run a single command in your environment. And as this is now a pixie project, I can check my git div status project and you see it will have modified my project Automl and will have created Pixie dock. If you commit these two files to your GitHub repository, everyone can recreate it on their own machine with very logged in log file, including the Python version, which is very powerful. If you want to reproduce the managed system, there's no extra necessities that you need to install yourself. And that's actually like there were so many things in this short introduction. So thanks already on that point and quite some questions now probably, but something I would also like to highlight is once you have installed Pixie, you don't need to install Python anymore because Pixie will always take care of that. That's something I have been asked quite a bit before by different clients or also in the companies by more junior developers. Let's say, how should I install Python this way, that way, or the other way? Or the confusion between do I use the right python and you're on Mac, because Mac already installs Python by itself. So all these kind of things, they all can be avoided, let's say, and make it much even easier to start with Python. And also just to add a couple of sentences. Search basically allows you to search any PI bytes content for specific keywords, like in this case fast API. And I think it's a nice example that if we just add the search byte to it, or search maybe fast API, maybe the better explanation. And for that, that search you can easily make it runable, because you just need to run pixie, run search byte, and you don't need to look into it. How the command is written out. Exactly. We also like it for fast API if you want to start a server for the backend and you don't remember how was it called again? Something with uv corn, something with columns and dots. Also you can just write that once and it just works. So I really like that there's even one extra thing that's so special about this, which I sometimes forget, but I am testing on multiple machines. I also have a windows machine at my house that I test on and these commands are cross platform. So we embed our own shell into pixie, which the commands that you write in the shell will run on their own interpreter. So you don't need to make sure that the variables are different on windows than their own Mac or whatever, that their tasks will just work on any machine, if you have tasks that work only one machine. So for instance, sometimes on windows you have some paths that you need to manipulate, then you can actually still make them platform specific, but in general you wouldn't need to do that. Nice, nice. And I think in general maybe sometimes I wonder these days if I should show the clients first how to do things with pip than with Condor, so that they run into some problems, so that they then appreciate Pixi. Because sometimes I'm asked, okay, why would, why, what do you need this for? I have never needed Conda. Let's say, but I like last year, for example, I installed like this transcription pipeline where you have to install something like Ffmpeg and so on. And if you don't use Conda, then you need to do that somehow manually install it in the docker or so. But if you just use Conda in that case, and you can do that via Conda or via pixie, then it's just installing it for you already completely, and then you can avoid some of the pain, let's say. Yeah, so I think that's really nice. Maybe like follow up question. So now let's say we are an enterprise and we have seen the value for using it for single packages. How is your vision? Or how does it look like if you have now a microservice architecture or so how would you set that up? Can we do already Pixie install, let's say one package and it has a dependency from another package. That's something projects. I think you have asked the exact question that we also wanted to talk about with you, and I think Tim can explain it very well. I would like for the audience to explain it a little more, what it means upfront. So this was a project, and with Pypi and poetry, you can also have projects. This is also in every language specific package manager. The notion of a project is very hard because you can have it as dependencies, you can install it locally, and then the deployment is something the package manager can also do, meaning you can build a package from your project. And this is something we're working on currently. And I would like then to explain more on our ids there. Yeah, so the fun thing is you can kind of do this now with current pixie in that you have to use pipe and you can use like dependencies to other project tunnels. But that is only because we use, that's part of our uv integration to kind of like expose those features. But the moment we saw that coming in, like we already knew it. But we, like, we get get this question like more and more like, how can we depend on our own source dependencies and. Yeah, so that for Konda is actually a bit more difficult than pipe because normally there's a split between the build. Like for example, Konda Forge has a recipe that pulls in the source and then builds it and pushes it somewhere instead of like the build being coupled with the source. This is because Kona had to recreate a lot of libraries, even lower level libraries like GCC or OpenSSL even to rebuild this. So you could essentially rebuild the world, because as you were saying you might need some system levels, dependencies like FFmpeg or something like that. That made it a bit more difficult. But what we're working on now, we call it Pixie build first, which allows you to build a Kona package from a Pixitomo and its associated source files. And that will create a conda package locally, and you should be able to include that. So our first version will probably only use like use like to try it out. We're going to use it like python packages, that you can build a Python package into a wheel from a Pixitomo file and then include it somewhere else. And then cargo and yarn and all these other package managers. I don't know about poetry, this part of poetry too much. But for example, we use rust a lot. And if you have something a, that depends on b, you can specify like, hey, use the same serialization library for both these things. So specify them once, use them in both places. So that's something you would want. That would be something we would need to build after. And the cool thing actually, and this is going to get really cool. So listen closely. So we can interpret it by PI dependencies. We can interpret conduit dependencies. So we could, for example, create a feature which is ready, like being kind of like set up, like someone is going to work on it to interpret the project dependencies array as condo dependencies, and they're normally interpreted as pipe dependencies. So at that point, you could build both a wheel and a condo package from your pixie tunnel, which is like, I think pretty cool, because then you could push through both sources. So that will make it a lot easier, because a lot of people know how to build Python. People know how to build a wheel, but don't know how to build a complex per se. They might want to, but it's like now it's a bit of a hassle, and you have to learn a lot of specific terminology, and it's like, it's very different from normal packaging ecosystem. So they get confused really quickly. And so I think this can be like a really cool feature at some point. So then you would have workspaces, and that could include features. And because Kolna is so generic, we could also create a c builder, for example, that interprets like user Cmake directly. At that point, you could create a project with a Python source dependency and the C source dependency. And I think that's kind of like unique in the sense that the only system that I think can do this really is either the robotics operating system ROS that we used to work with or nics. I think that will be an exciting feature even for Python developers, because you could maybe make a native extension somewhere, build it directly and install it into your workspace next to your editable source dependency. So it's going to be exciting times indeed. Awesome. And I understood, I think, 60% of what you said, not because I think it was well explained, but just because I have already used Conda and pixie and pip and poetry and so on, and so I could follow quite a bit. The other part of C and so on is maybe a little bit further away, and I guess for many listeners too. But I think the TLDR or TLDA, however you would call it when listening here, is that if you choose pixie, you can install it both. You can make packages both for Conda and for Pip, whereas if you would now use Pip dependencies, then it would be hard to transform it in conda, or if you use Conda then the other way around. So it's just additional level of freedom, let's say degree of freedom. And on the other side there is something growing up, let's say, that makes it much easier to manage all the package requirements with pixie. Also for microservice setup or for other kinds of setup where you have different packages and you want to import dependencies from each other. Yeah. So in the near future you would hopefully have one pixie toml file that can reference others, and it would all build them, build, build, and solve them together. That's the idea. And this would be kind of like nested pixie tumults, right? Or PI project tumults, because you have one PI project tumblr that references another one and both. In both of them there would need to be a pixie config. Or would it be sufficient if there's just a pip one like, or the usual ones? So it's sufficient, sufficient to have a pipe. I mean that, that, as I said, it already works because we use uv to do the heavy lifting there. But if you that doesn't know about any conda dependencies. So if you would like to use conda dependencies, then yeah, so there would need to be some, some pixie config somewhere like at the top level or something. But it will be easy to pull and just get cloned something. And ideally we would make the friction as low as possible, and that you can just include a PI project tomorrow and then use it in your workspace. Nice, nice. So for now that means basically you can already add the pixie configuration on the different pipe project tumults currently you would install these packages. Maybe if you have nested dependencies with a pixie like normal command that uses the pip dependencies, and then in the near future it would also be able to use the pixie configuration, meaning the conda dependencies as well. Yeah, that's exactly right. Yeah, that's super cool. Yeah. So just a quick one. If you have used PiP before, then it would be the equivalent of pip freeze, right, where you get all the requirements, let's say like the exact requirements. And if you have used conda before, then there is this conda lock package or addition, let's say, which you need to install additionally and run additionally so that you would have the exact conda dependencies locked out basically. And Pixie does this on the fly whenever you run commands, right? Is there something else to know about the lock file, let's say, and what are some best practices like, should it be in the git directory or should it be rather managed by Git lFS also, or are there some general recommendations that you have how to manage that? Yeah, I think I can give you some tips on how to use it. So the log file is a file that we always generate. There's no way yet to turn it off. It's up to you if you want to include it into git or not, but I would always recommend to do so because it's always easier to just remove it if you really don't like it on your own system than it is to make sure that it reproduces in CLI on your colleague system. And a lot of things we hear is that people want to be cutting edge and they want to use their latest greatest always. And a log file would limit them to use the latest greatest because they don't put it in automatically all the time they install. But you could easily run an update command. We also have dependable or renovate equivalent kind of tool that is built by pavel, one of our contributors, that can update your log file in CI. So you could do it like weekly or monthly or however you like it. And that would make sure that your war file is up to date with the latest and greatest that's available. And you can also specify to update only certain packages. So you could say, I don't need Python to be the latest greatest, I'm fine with the version I'm on, but fast API needs to be updated every month or every week. And commands like that you could add to your CI to make automatic. And this will create situation where you can decide when things start to break. So some of these updates of packages might break your own code. And we would in previous work have situations where our night X UI would suddenly fail and the whole sprint would be turned over because the issue that it created was so big that we couldn't just simply solve it in a few hours. And with this log file, there's no way that we pull in any packages that are suddenly updated by someone. You will just be safe. You can just continue your work and then your update PR or your update situation will let you know when there is a problem and when you need to fix or make sure that you solve for the right versions. Change your code, however. Nice. Yeah, and I think that's something that, again, when you haven't experienced that problem yet, it's maybe not so clear what is the benefit. But it's hard, right? It's hard to say something where it's important to experience first the pain, to then appreciate it. But I think it's also good to just maybe at that point trust that, hey, they say that there may be a huge problem, maybe have a minimal example of, of how that can go terribly wrong so that you can directly see it. But basically having that log file and knowing that the dependencies will be installed exactly like this on different platforms or as exactly as possible on different platforms is quite a big win, I would say. And in other tools it's really often like you need to set up something manually yourself, like the Conda lock flow. You need to think about it. And that's something that is really nice that in pixie it comes right away, I must say. We also had one current or maybe a couple of weeks ago. I'm not sure if that fixed yet. But one situation where when you have editable dependency and different platforms, then it would change the log file again and again, even though there weren't real changes. So is that something that is already solved and I just have to upgrade, or are you aware of that? Or could you explain what is behind that and maybe how possible fixes could be added soon to the PKCE environment. So with an editable install, the way it works is it builds a wheel behind the scenes and installs it into your environment. And the moment the metadata changes, like you touch the pipe project file, we need to rebuild because you might have changed something to the metadata. We've learned this from uv, how they did this. So we looked at that and they were using a hash of that tomo file. And the problem with this is that sometimes it just changed or something and it rebuilds for nothing. So we're thinking of taking maybe a subset of these fields to actually see if. So yeah, to see if that changed, then we don't rebuild, for example, because it's hard to know what build backend is doing what. So we have to be very specific there. But given that, so that's how it works. We've also had just a lot of problems where we didn't take something into account. And there's this check that's called certifiability, which checks if the log file is still valid to what the specification is, and then it says no it isn't. So I'm going to resolve and reinstall. So there could also just as well be a bug there. So definitely there's this flag that you can use minus v and you can do avv to get extra verbose mode. And if anyone is experiencing this and you think it's rebuilding too often make an issue with that verbose log, paste it in, and then we can definitely have a look, because this is something we already fixed a couple of bugs on. So it's well be that there's something else there. Yeah. And I think that's also a great example that I can just say like adding an issue, then the conversation about it and actually having a new feature added to Pixi is really something like a matter of sometimes just weeks or so. Because like we said in the beginning, we had the last podcast in November 23. Now it's August 24 and there have been so many things added, and sometimes you can really be directly edit integrated into the open source discussion. So yeah, whenever you experience something like this, I can also just encourage to reach out to pixie and, and just mention it. Yeah, I would also recommend users to join our discord if they feel comfortable with that because we are very active there. And we also always like to get some feedback on what the tool is doing and really care a lot about not breaking things. So fixing the real bugs as quick as possible is very important to us. Yeah, and we're also a small team, so we can tackle everything. So there could be sometimes, yeah. But we do try to keep track of what is important. But if you as a listener want to contribute to pixie, and I know it can be very daunting, especially when it's a large code base, but we're also very willing, if you want to do a contribution, to add either like a small call or like an explanation somehow that will be easier for you to get into the code. I think not a lot of people are aware that people do this, but yeah, I've generally found that more open source projects do this because they use a lot of help. So if we can get you up to speed a bit more quickly, we're happy to give some of our time for that. Awesome. We already touched a little bit about the last questions I have thought of regarding the vision and also what's next. Some, something that you are really excited about. Tim, you just mentioned like the nested, or let's say in general package management with Conda and Pip. How do you imagine Python packaging and maybe general packaging, but for our audience, more python packaging to look like in one and in five years. And what are some things that you're excited about in the more near term future? Let's say. I know you want to go first. Yeah. Whoever wants to go first. Yeah. So I'm really excited about Pixie build. Like, I'm really excited about it because it just gives me like something I couldn't have thought about. And like you were saying, the ability, I think a lot of people realize, don't realize, especially coming from Python, that you need a lot of system level packages. Like you had a case, Ffmpeg is a perfect case, or SDL or any graphics library or something like that. Being able to get like an isolated environment for that means I. That you are able to run it on any machine. You're right that there are some of these manual steps. Right. That you always have. I have to install Openssl and this way and this and this one and that. And with pixie you try to alleviate this. And I even have to get used to it, that I would see an issue and I like, ah, I probably can't reproduce it because I'm like used to not being able to reproduce these things. But then there was a pixie file and thought, no, I can just run it and start, which is like, kind of like great. And once you get used to, you don't want to go back. And I think that is something we would like to change like that. It's just, yeah, I mean, there are so many computer science programs where people have like one whole day of setting up their PCs and someone is using a Mac while the other ones are using windows. And we just want to get rid of, rid of that friction. And I think, yeah, build is one thing to do that in workspaces because you can do your own source things. But I think the bigger picture is just ideally not having to worry about these things as much as you're probably not aware that you're doing. But then if you think about it, there's always a lot of this custom setup per project. So yeah, that's the big idea. I think that we want to change. Ruben, anything to add to that? Because I think that was already pretty comprehensive. Yeah, I think I completely agree with them. I personally really, really care about the robotics community because that was like my whole study and my whole first few years of working. And I know the pain that these people are in. They might not always know that they are in a lot of pain, but I. So much easier if you're just developing with a tool where there's one solution for package management, workflow management, deployment and all those problems in one tool, they don't have it yet. And that's what I want Pixie to become for that industry. That's just something I'm working towards together with all team, of course. But that's my, my way of staying focused. Yeah, yeah. And I can say like in November last year, I was already definitely inspired. Now I have used it for a couple of months and now I say always, that's the go to, let's say, recommending it to the clients and so on. And I also really think that vision is great. Right. Having really pixie run and then it works and you can easily distribute it or share with colleagues or whomever. And something that I was thinking could be a nice addition. Not sure how, how hard that would be, but I saw Pulumi, for example, so infrastructure as code tool. They added their own Pulumi AI and they basically now have added some steps that if you get an error in the terminal, it will automatically try to fix that. So I'm not sure how far that would be a way, but for Pixie to automatically add that dependency that is currently missing or something like that, not sure how dangerous that is in an dependency world. Could also create some open doors for viruses or whatever, just installs anything. But yeah, maybe that that would be also amazing that you, at some point in 2030 or so, you wouldn't have these pains anymore that you, that I have certainly also gone through with Conda and all the different python package managers. Yeah. So one of the great idea, as we, as you, as you said, we're open to. No, it's. I think it's a great idea to have these and see how we can use that this coming role, so to speak, more nice. The nice thing about package management is that it's a perfect math problem. So the solve of the packages is like a complete solve, so there are some routes we can still take to improve it without having to call some kind of server that's running an LLM or whatever to figure it out for ourselves. But it would be cool to get help like you have these packages. Are you sure you don't also would like this one as well? There are steps in between we can do. Nice. Yeah. Thanks so much for explaining a little bit more of the world of Pixie and also introducing yourselves as part of the team. So I can only say I appreciate a lot that you solve all these pains that hopefully future developers don't even know anymore because it's just all solved by Pixie. But the developers that have seen the world before probably also appreciate it quite a lot, and I think you deserve a huge thank you for that. Thank you for saying so. Thank you means a lot. Means a lot to us. I feel like we're coming to an end on the podcast, so I'd also like to do a thank you to the rest of our community. We have some big players in it who are mainly working at companies like Quantco, who does a lot of the testing and has completely moved over to a lot of Pixi projects already. We also have a good contact with Rerun, who's also working in robotics, but also completely changed over to Pixie, and we learn a lot about their use cases. We definitely want to thank Astral for making uv and for sharing their great work in such a way that we can also help our community with the same benefits. And of course, unlike Kolba, for the initial creation of Kolba and open sourcing the whole Konda ecosystem for other developers to benefit from it. And without all these companies, we wouldn't be able to create such a cool product in such a fast time together with just a small group of people. Yeah, and let's not forget all the contributors to Kona Forge, which is the packaging ecosystem we rely on. And it's just like tireless work, maintaining, updating these packages, making sure companies people can depend on it, and things can get, as you said, packaging can be hard, so things can get really hairy there as well. And most of them are doing that in their free time. So thank you for all your effort. Amazing. I think those are perfect words for the end of the podcast. So looking forward to have you guys back on the podcast. Let's see when there are again, like some great additions to the tool so we can share some good news again. Thank you. Yeah, so see you guys. See ya. See you. We hope you enjoyed this episode. To hear more from us, go to Pibyte friends. That is Pibit es friends, and receive a free gift just for being a friend of the show. And to join our thriving community of python programmers, go to Pibytes community. That's pibit es forward slash community. We hope to see you there and catch you in the next episode.