An educational side project
What does a great side project look like, which helps learn new technologies, but also helps stand out when looking for a new job? Analysis of an Uber simulation app, built from scratch.
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in today’s subscriber-only The Scoop issue. If you’re not yet a full subscriber, you missed this week’s deep-dive on Agoda’s private cloud setup. Subscribe to get the full issues, twice a week 👇
I’d like to share a story about an educational side project which could prove fruitful for a software engineer who’s seeking a new job.
Juraj Majerik is an Amsterdam-based software engineer who decided to improve his applied knowledge of containerization, multiprocessing and observability. So he built a “clone” of a ridesharing app like Uber or Bolt: the app simulates riders requesting trips and drivers picking them up, then repeats this all over again. See it in action, here:
What grabbed my attention was how this app is much more than just a simulation. Juraj included system monitoring parts which monitor the server’s capacity he runs the app on:
And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used:
The app uses:
Node.js for the simulation engine
Go on the backend
PostgreSQL for the data layer
React and TypeScript on the frontend
Prometheus and Grafana for monitoring and observability
And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. You can read this here.
This “Uber clone” offers a nice blueprint on how to tackle a complicated project. Thanks to the detailed documentation of Juraj’s progress, we can reconstruct how he built the project. This is educational, as it shows one possible way to tackle a complicated project, and one for which the creator could only devote time to on the side, and at weekends. Here’s how Juraj approached the undertaking:
Phase 0: make a plan (Oct). Instead of starting with coding, Juraj kicked off by sketching. He sketched out what he wanted the final product to look like:
And he sketched how he envisioned the observability part to work:
Phase 1: Infrastructure (October-November).
Before diving into coding, Juraj set up the infrastructure. I assume he did this to familiarize himself with infrastructure which he hasn’t necessarily used in production before. This side project offered a good opportunity to try it out.
Set up a server on DigitalOcean (a virtual machine with 1GB memory and 25GB of disk space)
Set up the domain, and configure the DNS
Set up users on the server
Set up SSH keys for more secure authentication
Install Go on the server, which will power much of the backend
Set up a HTTP server using Go
Deploy for the first time, doing so manually, for now
Enable HTTPS by registering a certificate
Utilize environment variables to be used for configuration, instead of hardcoding configuration or secrets into the source code. Hardcoding secrets in production is poor practice. We covered how Stack Overflow learned this the hard way, a few months back.
Serving a web page. This is the point where the app reached the “Hello world!” stage
Improve deploys. Change deploys to be a one-command process, instead of multiple steps
See blog posts #1-11 for details on all the steps.
Phase 2: some business logic, and more infra (December-January)
Draw a map using JavaScript to map onto an SVG format
Build a graph and traverse it. Here’s a video of Juraj demonstrating this traversal.
Set up Docker to package the application into containers
Use Docker in production, and modify the deploy script to deploy using Docker
Set up PostgreSQL to persist data on drivers, riders and trips
Connect the Go backend to the PostgreSQL database
Connect the backend and the database containers
Docker Compose in production
See blog posts #12-20 for details on all the steps.
Phase 3: a basic UX (February)
Migrate to React
Design a car, from a vector image
Move a car with animations. Including adding unit tests.
Turn a car using rotations
Tidying up the project: refactoring the files now, that the project’s structure is more clear
Server-generated data
Animation fixes
See blog posts #21-27 for details on all the steps.
Phase 4: “hardcore” coding (March)
This was the first phase where Juraj did no more infrastructure work, and focused only on “pure” business logic.
Simulation engine: this component simulates the behavior of drivers and customers
Multiprocessing for Node.js, to avoid blocking the event loop
Generating destinations, and making sure the start point is not too close to the destination
Matching drivers with customers: doing this similarly to how a service like Uber does
Route planner: tell the driver which route to choose to collect a customer
See blog posts #28-32 for details on all the steps.
Phase 5: finalizing and monitoring (April)
Finalizing the simulation
Setting up monitoring & logging
See blog posts #33 & 34 for details of all the steps.
Standoud things about this side project
There are several things that I’ll highlight about this side project.
1. Persistence. Juraj built this project on the side, between October 2022 and April 2023, which is 7 months.
2. Documenting the steps. Every time Juraj made progress, he documented what he did, and how. This likely helped him to learn, and it also helps others wanting to understand, too.
3. Incremental progress. The project looks like a tough one to build from scratch on the side. But looking at the incremental steps, it is not nearly so daunting. Here are a few of the steps, taken directly from the blog:
Monday: installing Go on the server (17 Oct)
Wednesday: setting up the HTTP server using Node’s native http module (19 Oct)
Friday: the first deploy (21 Oct)
Saturday: enabling HTTPs (22 Oct)
Sunday: setting up environment variables (23 Oct)
(Step away from the side project for a week)
…then do another burst of tasks, making more progress
4. You won’t get the layout of a project right, the first time. So refactor! Juraj was about 70% done with the project, when he went back to refactor the structure of the project. This wasn’t for the lack of planning: but because as you set up new infrastructure, things turn out a bit different than you expected.
This is just the way of software engineering, and there’s nothing to be embarrassed about it. As you learn more about the project, don’t be afraid to go back and change the project structure – or do other refactorings – to help your work, going forward. See the project structure Juraj set up at the end of Phase 3.
5. Infrastructure is important, and setting it up right can be a consuming task! In the first 3 phases of the project, Juraj spent a lot of time on infrastructure setup. It was only in Phase 4 that he was able to focus “purely” on the coding part.
I really like how this project showcases just how much time can go into infrastructure setup. At companies with dedicated platform teams, those teams take exactly this kind of load off other teams building greenfield projects.
Both as an engineer, and especially as an engineering manager, don’t forget there’s a real cost to setting up and then maintaining infrastructure. Much infrastructure work is invisible as it does not involve commits, and most engineers won’t document the time they spend on these tasks, like Juraj has. But this is work that still needs to be done!
Here are the learnings Juraj had with this project. I reached out to Juraj to ask how this project helped him. His answer:
“I've touched on many topics I haven't been exposed to in my day-to-day job, such as server configuration, setting up a deployment pipeline, animations in JavaScript, or using Docker.
I decided to get better at algorithms and data structures a few months ago, and this project complemented it nicely. For example, I implemented the map and its associated methods (e.g. path-finding) from scratch.
I also wanted to get better at setting up a full-stack project from scratch and understand why certain technologies are used. Docker is a good example - I didn't use it because I ‘wanted to’, but because the necessity for it arose. I was surprised how much time all of the infra work took me before I was able to start with the actual simulation logic!”
And how did Juraj find the time to work this much on the project?
“While I was at my previous job, I dedicated 1-2 hours a day, usually after work. After leaving in March due to a layoff, I decided not to start my job search immediately, but spent some more time finishing and polishing this project. That's when I really ramped up my effort on it and started seeing a lot of progress.”
If you are thinking of doing a side project with the goal of learning new technologies – and, to also be able to share those learnings, and show off the side project – I can recommend taking inspiration from this methodological approach Juraj took. Of course: don’t copy the exact approach, as-is. But planning first, documenting steps, and building a “production simulation” are all ideas that you could use in your own side projects as well. If you do: your side project will surely stand out from many of the other ones.
Your own projects often seem less impressive to you than they are to others. I was impressed by the implementation and attention to detail for this project – for example, going the extra mile for adding in monitoring for the server components: typical for a production service, but rarely seen in a side project. When I shared that I’m impressed with the execution with Juraj, his response was surprising, as he told me:
“Working on this side project every day, it no longer feels that impressive to me (as I think is common with software projects). But having read your fresh perspective, it certainly gives a lot of encouragement!”
When it’s day-to-day work, most engineers don’t think they’re doing anything complex, or special. And, in all fairness: no single step in Juraj’s project was complex. The complexity comes from the combination of simple steps. This is the beauty of software engineering: that everything that seems complex can be broken down to simple to understand – and simple to implement – steps.
Thanks a lot to Juraj for sharing this project with me. View the complete source code here. If you are hiring for full-stack engineers, Juraj is on the market – at least for now! Connect with him via his website, on LinkedIn or on Twitter.
This was one out of the four topics covered in this week’s The Scoop. A lot of what I share in The Scoop is exclusive to this publication, meaning it’s not been covered in any other media outlet before and you’re the first to read about it.
The full The Scoop edition additionally covers:
Why are many Snap employees selling their stock as soon as it vests, and not a day later? Snap is a very strange publicly traded company, where shareholders have precisely zero votes. I’ve talked with engineers and discovered a surprising level of distrust within the company. Is this tied to the governance model, or something else? Exclusive.
Analyzing the split of cuts at Lyft. How much were software engineers impacted by ride-hailing service Lyft’s large-scale cuts? I went through data based on Lyft’s talent directory; it looks like tech functions were hit harder than other areas. Exclusive.
Robinhood is no longer a remote-first company. Few companies were as vocal about becoming a remote-first company than Robinhood. But less than 18 months later, the company has made a u-turn. What can founders and leaders take from this avoidable policy reversal? Analysis.