Pybites Podcast

#060 - The QuietLinks search database and the importance of an MVP

Julian Sequeira & Bob Belderbos

This week we have Tim Gallati on our podcast to talk about his Python journey and a very interesting search database he developed: QuietLinks.com

We discuss the problem his solution solves, the stack it uses, and some of the challenges he faced when developing this app.

He talks about the importance having an MVP (Minimal Viable Product), which is in line with our philosophy of shipping soon and often.

Check out QuietLinks here and connect with Tim on LinkedIn or Slack.

Mentioned infinite scroll YouTube video.

I got an email from Cloudflare saying your app has used several. I forget how many megs of requests, and I was like, did I get hacked? What just happened? How could that be possible? I logged in, and I'm seeing 600 plus users in the last 30 days, and I'm like, oh, this is a flu. Hello, and welcome to the PY Bytes podcast, where we talk about Python career and mindset. We're your hosts. I'm Julian Sequeira. And I am Bob Valdebos. If you're looking to improve your python, your career, and learn the mindset for success, this is the podcast for you. Let's get started. Welcome back, everybody, to the Pyrites podcast. And this week it's Bob. Julian is asleep. He's having FOMO right now, but I'm here with a very special guest, Tim Galadi. Tim, welcome to the show. Thanks, Bob. Great to be here. Yeah, I'm glad we finally got you here on episode 60. It was a long time coming, and, yeah, we got some really interesting stuff to talk about today. But first of all, why don't you introduce yourself to our audience, and what do you do day to day? Great. Yeah. Well, thanks, Bob. I am. My day job is, I am a Cybrarian at Oracle. So. Cybrarian, it's like part robot, part librarian. The future of libraries is upon us. And I also do some nonprofit work, which I think we'll talk a little bit about today, which relates to my app, which is about noise pollution and the benefits of quiet places. And I'm based in the US. I know this is a global audience, so we're on the west coast in the US here. Sure is. Yeah. Lightning time zones, always. Yeah. Piwa is worldwide, so, yeah. Siberian. How does a Siberian get into Python? What is your story? How did you start using Python, and how did you learn it? Right. Well, geez. I want to say that it's a story of being lazy, or at least finding a better way to do things. There's a lot of work that goes into being someone who curates information, where there's a lot of work with metadata and also data. And I was doing a lot of that work pretty much exclusively in Excel, and I just came to a point in my career where I knew there were better ways. I had heard about these things called APIs. I would always ask, like, can we do this particular project, or can we get this data in a certain way other than having to massage it in Excel? Always, it would be like, yeah, that's available in the API you can always get that in the API. Finally, after years of hearing this, I just decided I have to, despite not having any background in development or anything like that, I said, I have to take a look at this because Excel is great and it's got a lot of features, but my wrists are getting tired, my brain is getting tired. There's got to be a better way looking at Python and trying to consider how I could be a much more efficient and just better at my job and to have more possibilities in terms of a career path, too. Nice. Yeah, you saw that opportunity in the job, dealing with a lot of data, and then Python automation becomes very powerful. That's similar where I got started. Cool. You have been using Python for a couple of years now, right? Yeah, I think my Python anniversary was in January, 2 years. So two years and change at this point. Awesome. And now flashing forward to your most recent app and maybe startup one day, quietly. So, yeah, you built this entire solution, which is a major achievement. Really, really proud of what you did. Do you want to tell us what it is and the problem it addresses and what it uses? Sure, yeah. Quiet links is something that came about as part of my nonprofit work, as I mentioned, on the effects of noise pollution and the benefits of quiet. And this was one of those API projects that we've, you know, Bob, of course, without you, this kind of stuff wasn't possible. As a mentor, you really helped, not just with the technical side, but with the persistent side, to really have this vision that I'm gonna describe and how to just break it down into chunks and to use it as a product producing learning opportunity. And what it is, essentially, is it's a collection. It's a database, a freely available public database that's searchable of all research articles that are available on the topic of noise pollution and the benefits of quiet. And the reason this came about was a lot of the work that I do with nonprofits. I get interviews from journalists and researchers and students, and invariably the first question they would ask me is, what does the science say? What are the effects? And what's the rub here? Why does this matter? And kind of like my experiences in Excel, there came to a point where there was just so much research out there, and more importantly, whatever angle someone was coming from. Cause there are many angles you can come at to this topic. It could be urban environments and what the noise is in those environments, children's education, the effects on concentration, marine life in water, how does that affect marine life? And on and on and on. Right. Somebody actually asked me recently about restaurants and the effects of noise on restaurants. And has there been research on that? And can we see differences potentially in income or revenue or where people go or that kind of thing? So it just runs the gamut. And rather than folks coming to me with these questions and having very broad some metrics, like, for example, the World Health Organization estimated that in one year in Europe, a million years of healthy human life are lost due to noise pollution. And that's just in one year in one specific area. And those stats are very powerful, but they're also very contextual. What this quietly was looking to solve was based on what your interests are in this topic. Go in there and search on that and see what work has been done, because at this point, the database contains over 25,000 research articles and there's many, many more out there. This is just from sources in the US. This is a big problem. It's number two in terms of pollution, right behind air pollution, because basically, basically any place that's polluting the air is probably pumping a lot of sound into the environment, too. So it's got a very niche following. And I was just trying to solve the problem myself of having to answer these questions over and over again to different audiences and do the research on my own for them. But what I found pretty quickly was not only was I solving a problem for myself, but there was tremendous interest, I want to say surprisingly, but just in a really happy happenstance, that folks really gravitated towards this, to the point where we noticed there were between six and 700 unique users in a month. And that kept happening. And it was just really surprising. And when I began to tell people about it, they really resonated with it and they started to have their own, their own ideas of what they wanted to see in it. So I started to get a lot of response and a lot of feedback on that, which. Happy to share more on that, too. But just the app itself, it was a great personal project that is turning into something much bigger than I had anticipated. That's amazing. What a story. First of all, the problem, it addresses the importance, right? But then also the fact that you saw that opportunity and just solving a problem for yourself first scratching your own itch, as we always say, and that led to so much more that you couldn't even have envisioned when you started. Those traffic numbers are crazy. So what is the technical stack? What is it running on? It's a web app, right? It is a web app. We're using a flask with a bunch of other pieces here and there, I think just thinking through some of the challenges we're using Flash Flask HTMX is in there for some infinite scroll. Because we started to have these performance issues pinging this database, sometimes there would be a lot of results. So we used that HTMX infinite scroll. Also the API that we were using for this was quite, quite old and it only returned XML. So we use the library XML to dict, which really helped solve that problem, get the results in a manageable fashion. The app is on Heroku. We're using Cloudflare for the security piece, and the app updates nightly. There's an API call that's essentially going every night and trying to find new records, see if there's new research involved that's been published, and immediately adding that to the database. So it has a lot of timeliness and some moving pieces, but overall it's been great. And I will say too that Tailwind CSS is what I ended up using for the front end, which for someone who's never done that before, it was pretty straightforward and out of the box. To be able to move pixels around on the screen for that front end for the webpage, which was a outside of python, was a bit daunting. I was a little nervous about that because I was thinking all about the backend mostly, but it turned out nice. There was plenty of free templates out there for me to play with and that included search functionality and how the results turned were returned. So a pretty, pretty extensive stack. Looking back, when I first started the project, I think, Bob, we were just saying, oh, I just want to pull results from an API, put them in a database and let people search them. And it just seemed like three steps. But boy did it turn out to be war, right? That's where we started, literally like a little import script, right? It's calling an API, storing it in sqlchemy table. And I think it was one table and very simple, but then it grew out. Now it looks super slick. The tailwind CSS, it looks really nice. Everybody pause this episode and go to quietlynx.com. And it really looks nice. Yeah. So on the challenges, I remember that parsing of those nested dictionaries, the XML two dict library, really saved a lot of code, made it simpler. Do you have some other challenges you had to overcome in this process? Gosh, I think probably the biggest one, and I mentioned it briefly, but it was just the performance, the sheer number of results that would be returned, and thinking about how to, through SQL alchemy, how to create that query, that search query, and how to be able to basically let it draw on a page. And we went through some ideas around pagination, just sharing say ten results per page. There were some others that we had looked at, but I think ultimately that infinite scroll was like a touch of magic. And thanks to you Bob for even thinking of that, because initially I hadn't thought of that. I was thinking through a much more complex solution where we were going to have to grab all the results and find ways to sort them or do something so that the performance of the app wouldn't just stop if somebody searched for, say, the word, just the word noise that returns, I think something like 1015 thousand responses article. So being able to play with the HTMX and Infinite scroll, it was along the lines of the tailwind CSS where it was much more manageable than trying to build out this really robust and difficult solution. And it was a crash course in how challenging search can be too, for sure. So I think that was probably the biggest hurdle towards the end. Yeah, I remember that. And of course shout out to HDMX, right, that makes this relatively well doable in the first place. I mean we could have gone with pagination, but then you end up having to click a lot of links at the bottom right to go to the next ten results. This is way more elegant because as you scroll down the results it's making additional queries to get the next ten results in the next ten results. And to the user it's really a nice experience. Kudos to HCMX, that library that makes it's a great interface that saves you from writing a lot of JavaScript manually. So that was a great match, but nevertheless it was quite challenging to implement it. I actually remember going through documentation and it didn't really click. I mean philosophically I understood it, but really to get it working I had to abstract it into a sample repo. And I think I did a tutorial on doing this on a large set of YouTube videos, which we'll link below. Maybe one more challenge you had in this process. Oh sure. So I mentioned at the top just the age of this API. I don't know for sure, but boy did it feel old, especially because it was only returning this XML and the documentation was so soft. The solution was Google. Some very very kind soul had shared their code of how the heck they authenticated got results from this API and it's way more code than you would think. It was a really complex piece to interact with because the metadata is so long and deep I mean, think about it. I think it's something like 30 million. Over 30 million records are in this articles or in this database. So there's an extreme amount of just all this metadata. But also, some people are filling out some things and not others. And how do you make a successful query to this? It wasn't straightforward. It wasn't like, say, a sports API that doesn't require any authentication. You could just grab results right away. This was. This was more complex than that. And it was. It was. That was probably the big challenge. And then the documentation of figuring out what was done where I think was also, it took some time, but it was way faster. I don't know that. I just don't know how long it would have taken, Bob, if we had just tried to code that on our own, because it's. It's something that was made in the dark ages. It just felt like, thank God it was there. But, boy, yeah, you know, thanks. Kudos to the programmer that gave us those results because I think they anticipated the amount of work that they did and the kind of time that they were going to save other people. And it was an open source sort of thing. Right. It's research. It's scientists who are interested in sharing knowledge, and they took that to. And applied that to this code and said, hey, I want to help other researchers, if this is helpful to you. Great, great. Please leverage this code. That's amazing. So generous. Right? And the value of open source. And one other thing I remembered is that we also hit rate limits. Right. API rate limits that you had to kind of put some sleeps in there or be tactical, like how to. How to get the data spaced in time. Yeah. The dreaded rate limit. Right. And pagination, too. It was. It's funny, I think all of us are, at least for me, being new to this was just like, oh, I can't just kind of get everything. Like, what is this error I'm seeing? And just also going back to the documentation, it didn't really specify, say, like, hey, here's the number of calls you can make in a day, or here's the recommended sleep time to give the database a rest. It was just sort of like trial and error. And that was, that was, that was my introduction to sleep. And that was also a challenge because we were pulling basically articles from, I'd say the mid 1940s all the way up to, you know, present day. So that's a big, big, big date span. And you essentially, it's not one call, it's. It's a daily call too. So you're having to go through all the days, and then within that day, depending on the pagination requirements, you know, get just so many for that day. Right. And sometimes there were none for that day. So, um, you can't buy, you can't do it by the year or whatnot. So it was, it was a lot of, uh, just toiling with trying to figure that out. Like, what's, what's too much, what's enough? And, um, it was just a one time import for that too. So it was like, okay, if this script runs for a couple hours, that's okay. You know, that's, that's fine. But, yeah, it was a bit shocking to me, the world of how do we get this data without either getting errors or just really annoying the person who's the API owner. We want to be on their good side, Bob. We want to be able to get that data. Exactly. Yeah. And I find that a beautiful example of how the real world, when you really start to develop those solutions, is different from the tutorials and the happy paths. Right. You learn so much by solving all these problems, and once you solve the problem, a new problem arised. Right. And so which brings me to the question. You clearly got out of this with a minimal viable product or an MVP. So when we learn development and we go through this process, what is the value of having an MVP or minimal viable product? Wow. In this case, it was everything. Without an MVP, I don't think. I think this would have been a learning project that was, as you and I did it, Bob, for five weeks to learn Python, because this was a part of a ten week program where we did two projects. And the other project I love too, I was really grateful to do that and learned a heck of a lot on energy, data and a whole different set of APIs. Without the MVP on this, I never would have put it on a website. So it would not have gone over to quietly because I just wanted to put it there again to just save me that time. I just started sending researchers there. Just was like, here you go. And I also linked it to a nonprofit called Quiet Parks International that I've done work with and was one of the founding members of. And so it was just sort of like out there. And I felt like, okay, good, I don't have to do that work anymore. But then with that MVP out there, people using it, I just, one day, I can't remember what made me think of to do this, but I went to Cloudflare and I checked the traffic. And, oh, what it was. And I think I mentioned this to you, Bob. We had a laugh over this was I got an email from Cloudflare saying your app has used like several, I forget how many megs of requests. And I was like, did I get hacked? Like, what just, what just happened? Like, how could that be possible? And I logged in and I'm seeing like 600 plus users in the last 30 days. And I'm like, oh, this is a fluke. Like, you know, again, we do, we do press, something happens and a lot of people come to the site, they click through. And I thought it was a fluke and I had to wait a couple, I waited like two or three months. I was like, I need this. This has to be proven true. I gotta see that this is real. And sure enough, it was consistent and I was. And then just that MVP gave me that opportunity to know, holy cow, this is really something that some people find important and now I gotta put some more weight behind this. So the first haul was getting that UI looking much better, trying to tighten up a lot of the things that we had done in the query and that's when infinite scroll came and all these things. So it was the MVP that actually gave me that drive ultimately to see the response, to give that, to hear that response, to then really make this an actual working app, not just an opportunity to learn, which was really great, but it's been a whole new experience trying to manage a real live app. And it's fun and it's anxiety provoking too. I hope what I do now doesn't crash the app and all these minor things that I think before when it was just a learning experience, oh, great. If it crashes, it crashes. But now if something goes down, I actually got my first email saying, hey, your app is down. What gives? What gives, man? So MVP is everything, right, Bob? I mean, otherwise you just don't know what's out there and how this actually, as a developer, how this app actually lives and how it needs to be fed and maintained. Are you stuck in endless commutes? Have you ever dreamed about being able to work anywhere, control your schedule, give back to society, become an open source contributor, or become a successful developer, doubling your salary? Well, it's time to look at the PDM program and it's time to actually build something that's going to help you get the future that you're looking for. The people that we've worked with in the PDM program have achieved some incredible things, including starting their own SaaS business with their own application. Imagine that. That could be you building your own application, selling it, making your own income. We've had people more than double their salary. I'm not making that up. I'll say it again, double their salary. After completing our program and applying for developer jobs. These are the sorts of things that you can actually achieve through ten weeks of dedicated life coaching in the PDM program. So here's the challenge. If you are actually serious about taking your future into your own hands and not letting someone else control that for you, click the link below and get on a call with myself or Bob. That's right. We want to talk with you about your goals and how you can use Python to leverage your career. So book a call below and we cannot wait to talk with you soon. Yeah, there are a couple of lessons here, right? Like if you don't ship soon, you might not shipping at all. Right. And then you might miss out on a huge opportunity. Right. So just for clarity, like the tailwind CSS and infinite scroll were not there. You had a pretty bare bones app, but you put it out there because you wanted to get feedback. Right, right. And, yeah, and the other thing is to also just quickly iterate and you got that validation and then you knew it was worth it to invest more resources into it. But, yeah, but for developers in our audience, I think the most important takeaway is like once you launch it, then you get all kinds of feedback and yeah, part of the challenge starts then, right. To iterate over it. All of a sudden you have something running in production, you have real users, you have skin in the game. Probably you need to set up a UAT environment, add more testing. It's a different game, right? Totally, totally. Yeah, yeah. It's just been so interesting and I think, you know, that part of the journey of someone, for folks out there like myself who really picked this up, development just sort of maybe later in your career, you know, if you have any kind of previous experience just managing projects, a lot of that kind of kicks in, right? Because you've probably worked with developers and you've probably had to ask for things or try to anticipate needs and all of that, and the MVP at least will get you there. But you find that these skills that you're picking up and bringing into development are some things that you might have already had there and you're more well rounded as a developer than you might have imagined. You think it's all technical, but there's just so much more than that. And the MVP gives you that opportunity to remind yourself of like, oh, okay, this is a project. I've done projects before. I might not have written the code, but at least I know from a user's perspective what they might be thinking or whatever it is that you need to make business decisions or make coding decisions on. So it really opens up a lot. And that's why I love the mindset pieces that bob that you talk about all the time. And Julian, who's sleeping as well, shares with us, he's dreaming about it right now from Julian's sleep dream. Let me actually ask you, because he would have asked it for sure that I will cover. What's one mindset hack you need to succeed as a developer? Oh, for sure. It's persistence. I think for me, what made my development as a developer possible is knowing that I was investing not just those five weeks or ten weeks, but I just knew that this was several years at least, investment of time and effort, and that I wasn't going to have all the answers right away. I wasn't going to know how to make this app or save time at work that I was spending in Excel files. I wasn't going to have that right away, and it was going to be a little bit of a mental effort and a time suck up front. But those scales of impact and how much it would change everything for me in my workflow were going to come slowly, but then eventually it was just going to just exponentially shoot forward, and I found that to be the case. I mean, two years into this, I am so glad I made those investments in Python and Bob, our time, and with Julian and everything, its just completely transformed everything. And if I had been really nervous right up front saying like, gosh, I'm not getting this. Like, I can't make this work or whatever it is, trying to set up an environment and not getting that to work and all those things that have nothing to do with code that are roadblocks. I just knew that if I didn't solve it that day, there would be somewhere down the line this would be a lesson, a learned lesson, and somewhere down the line, I would be much more fluent, much more confident, much more capable. And so without that persistence, I think it would have been really easy to give up if I had short term thinking, but it was just that long term vision that made all the difference. Nice. Really happy to hear. Yeah. And that takes a lot of persistence. Remember all those late nights and all these times you get stuck? And that's very easy to think, like, well, this is just not for me. Leave that to the magicians, right, it takes a lot of struggle, but then all of a sudden there's a tipping point, I think. And I mean, you still get stuck, you still struggle, but there's, you know, enough. And then things, there's a certain comfort, right, you must feel now, a certain fluency, right. You almost feel comfortable now, right, with even the debugger and the tooling and. Yeah, it's, it's amazing. Which brings me to the question, what's next for your solution and or python for you? Yeah, it's funny. The next solution for the app for quiet links is article submissions. Folks have been contacting me where they say, I don't see my articles in there. How do I get them into quiet links? And what's amazing about that is as a developer, from my developer brain, that's not a solution I thought to make right away. I was thinking much more about maybe some of the UI, how it looks, how the search function works itself. I was thinking about all kinds of other technical solutions, but the audience is saying we want to be in there. So my next solution is about how do I get as many people in there as I can who are querying so have solutions, whether it's a form they fill out online with, say, if they have one article, if they want to do bulk submissions, or if they're in another database, because I've pinged one, but there are others in the world. And if they're in there finding out which databases they're in and start bringing those APIs into the system too and really bolstering the database. So it's, again, it's sort of like the way I describe it is going from a learning experience of learning Python to a listening experience and listening to the users. And what the users have said is we want in there. And so that's the next step. And again, I would not have thought of that. That was not something in my mind. That's amazing. Yeah. I remember us implementing features at our whim on the platform, on the coding platform, and then didn't really end up getting used because your users, people that use your software, know very well what they want. Right. And we need to ask, and if you have that feedback, that's amazing because then you know what you're building will make an impact, right? Yeah, absolutely. Absolutely. Awesome. Is there a final piece of advice you want to share with our audience today? Yeah, I think overall the thing, the takeaway for me is solve a problem that you personally need to solve. And that to me was a big motivator I think if I tried to think about something somebody else wanted, or even, let's say, a manager or somebody, I tried to envision their needs. I think what really helped me was to solve something that was a pain point for myself as a motivator and just to stick with it and to know that, in a way, you might get a script that's out there that does solve that problem, but there's always a better way to do it. There's always another thing that you can do to make it better. And so just play with that and start there. And if you're looking to have something potentially that might garner larger interests, talk with the communities that would be interested and impacted by it. That's been such a source of inspiration to me. It's like drinking ted Red Bulls. Right, Bob? It's like when people start reacting to what you do, it's like, okay, great, I gotta. You know, suddenly you're super coder, right? You sit down and you just. You go to work, and. And then you start working on the things that matter the most, not just what's in your head. Right? Yeah, yeah. So scratch your own. It take a problem you care about, and then while you're developing and you get users, talk to those users and the communities and drive the development by the input you're getting. Right? Yeah, yeah, yeah. And have good friends like Bob around to give you all that support and wisdom. A community is everything. Thank you. Yeah, no, it's great fun working with you, and I'm really proud about the result you got, because I remember when you came into the python space, it was all new. It was pretty scary. And now you're just learning those technologies one by one. You just take them head on, like the tailwind CSS, you just take them on and learn them. And I think that's kind of a mindset we need to get into when. When we want to do this work full time, that we cannot stay stuck in tutorial paralysis and or be, you know, be motivated by imposter syndrome. We are constantly learning, and that's kind of the only constant is that you constantly have to pick up new things, and through that process, you gain the confidence. Yeah, yeah, absolutely. Yeah. It's like growing up, right, Bob? It's like you start as a young. A young person, and you, at some point, you go through your teenage years a little awkward, and then at some point, you feel like, okay, I'm not going to break stuff. I can be an adult in this world. Yeah, exactly. Well, last question. Where can people find you or reach out to you? Yeah, well, if you do go to quietlynx.com, you can go ahead and send an email through there also LinkedIn. Go ahead and message me, friend me on LinkedIn. Friend is probably not the right word, but yes, please connect via LinkedIn. Love to hear from you and thanks, Bob. This is as always, it's such a pleasure to chat with you and just to spend some time with you and especially the pie bytes community. Awesome, awesome group. Cheers. Thanks, Tim. So happy that you could make it on our podcast, sharing all this great experience and insights. I think our audience will really enjoy it and yeah, we'll catch up soon. Thanks. Awesome. Cheers, Bob. We hope you enjoyed this episode. To hear more from us, go to Pibyte, France, that is Pybit es friends and receive a free gift just for being a friend of the show and to join our thriving slack community of Python programmers, go to Pibytes community. That's pibit es forward slash community. We hope to see you there and catch you in the next episode.