In this episode series, we are going to be building a carpooling webapp from scratch. I thought it might be useful to share my thought process and development workflow on going from rough idea to a working minimum viable product. It can be intimidating to create something from nothing, especially if you do not have the technical skills needed to build what you envision, so we will look at ways of breaking things into manageable chunks.
So, what is this series about? Well, one of my good friends is currently taking Computer Science in school, and I was chatting with him about how it can be an invaluable learning experience to work on his own personal side projects outside of school, or even work for that matter. In that, you will be forced to work your way from idea to working prototype on your own. Not only finding the right answers along the way, but almost more importantly, what does not work, and then investigating why. So, I guess my goal for the series, is to build one of my personal side projects and document the process, which will hopefully teach you to my pattern for breaking larger problems into manageable chunks, and exposed you to a bunch of different technologies as we work though this together.
What is the idea?
This is probably a good segue into explaining what the idea is, what specific problem I am trying to solve, and how it is different from what others are doing. The distilled version is that, I want to create a free on-line service, which looks at how you are your colleagues commute to the office, and then suggests carpooling options based around that data.
So, what does this actually mean? Well, lets quickly pull up a few diagrams for a minute and walk through it step by step. This is a map of Victoria, BC, Canada, where I live, but all of these locations I am about to show you are made up.
Next, lets put a dot on the map, and say this is where you work, and your office is located. Actually, let me change the contrast so it is a little easier to see. Next, lets put a dot on the map over here, and say this is where you live, and your house is located. The red line here is your route to the office.
Now, lets put a few of your co-workers dots on the map. We will call these points A, B, and C. We will put their routes to the office too.
All right, lets walk through it. For the sake of this example, say you typically work Monday through Friday, your hours are 9am to 5pm. You arrive and depart the office around the same time as everyone else. You are more than happy to commute with your co-workers, you trust them because you work side by side everyday, you just do not know them well enough to know that you pass a few folks on your way to the office.
So, the webapp we are about to build, will allow people to create private office carpool groups, then take in all this data about where people in your office live and their routes to work, and finally we will compare everyone in the private group to find possible carpooling options.
What are some methods for technically finding these carpooling options? One method would be to search along your route to the office within a certain proximity, say a few kilometers or miles. Something like this.
Using this method, you can see that, on H’s way to the office he passes by A and C, with B’s point being narrowly out of the search proximity. So, we could make a suggestion to H that he matched with A and C. Same goes for A, and C, they would be notified that someone passes them. So, each person in the group will get personalized matches based on who passes them and whom they pass on their way into the office. We could probably do this using some type of geo-special database search that would look for matches within this area, or maybe we can create something that will walk the route and look for matches like this. We will probably build both options and see what works best.
Hopefully you kind of get the idea at this point. Lets expand it a little by adding more co-workers points to the map, those being D, E, F, G, and I. Then lets add their routes to the office too.
So, if we compute the carpool matches based off route proximity like we did earlier you can start to compile carpool suggestions based off this data. For example, A’s suggestions would be categorized like this, having who passes A, who lives close to A, and who A pass on their way into the office. Lets take this further with, B’s, and C’s. C’s are kind of interesting because the majority of people pass this person on their way to the office. But, lets keep going, D, E, F, G, H, and finally I. What is pretty cool about H and I’s matches, are that they have three categories of carpooling options, people that pass them, people that live close, and finally people they pass on their way into the office. So, they kind of have their choice of people to ask about if they want to be picked up or maybe you want to drive.
So, the idea would be that you signup and login to this webapp and would be presented with personalized carpool matches based on your private office group. The carpool matches are grouped into three different categories based on what you might be looking for.
The data we can pull out of this does not stop there though. We can also compile statistics about the group as a whole, for example, total commuters in this group, carpools found, distance and duration times based on all routes to the office. It is kind of amazing that with only a few addresses we can pull all of this data together with a little processing. Obviously, this diagram depicts a rather small dataset, but imagine this is an office where 650 people work, there could be many thousands of carpooling suggestions.
So, who is the target market for something like this? Well, anyone who commutes to the same place on a regular basis, think all those office folk, what about call centers, schools, universities, manufacturing plants, or what about those large office campuses at Microsoft, Google, Amazon, Facebook, or Apple. Your not commuting with strangers, you work with these people, so there is a certain level of trust, and it could most likely be totally free too.
How is it different than what other people are doing? Well, it is not really like a Taxi service, or Uber, or common mass commuting tools where anyone can join, in that this is limited to people you actually work with. The overhead is also really low in that everyone is typically on the same schedule and going to the same place anyways.
Okay, now that you hopefully understand the idea, all we need to do it build it. So, lets jump back to the beginning and see how this series is going to go.
Bands of Skills
Going through this exercise, of having and idea and then building it into a product people can actually use, will quickly build your technical skills. You can honestly learn most of this as you go by simply doing it. So, lets look at the band of skills we will be covering while building this.
So, you might be asking yourself, okay, I know what the idea is, and that this series is about going from rough idea to working product. But, how is this logistically going to work? Well, I should probably quickly explain that for a minute.
Over the next month and a half, I am going to release about 30 episodes, or so, that will tackle specific problems along the way to actually building something like this. I’ll talk more about how we are going to identify possible episode ideas in a minute. Also, the finished product will be available for all to use for free at https://groupcommute.com. There is just a simple landing page there at the moment, but you can enter your email address if you would like to be notified when it launches. I’m not going to spam you.
Okay, lets jump back to the band of skills diagrams, as I wanted to quickly chat about why doing these types of projects, on your own time, is really important to building out your skillset and ultimately your career.
Full Stack Developer
You will typically find someone who is capable of this “in-between work” called a full stack developer. In that, they are comfortable working on frontend user interfaces, along with a wide range of backend systems, and how they are wired up. I tend to think of this as being a generalist, because you will know enough to get things done, but might not have a deep understanding of how they actually work, and have to learn some of it as you go. This is going to be a brief anecdote to by friend in Computer Science who I mentioned earlier. Generalists are extremely useful and typically get hired at smaller companies or start-ups, in that you will know how things are connected, especially so that you can offer suggestions, or quickly adapt across technology boundaries, and generally help to bridge gaps. I have found that the bigger the company, the more specialized things get, and you will have dedicated teams focusing on one of these areas, like programming the front-end, or the back-end, networking, security, operations, etc.
I just wanted to mention something quickly about learning on the job. Say, that this carpooling idea is not a side project, but something you are tasked with putting together at work. But, there is a minor problem, you do not know how to do it? Well, once you have a base level understanding of technology, companies are typically more than happy for you to learn new skills on the job. You might not explicitly say that is what you are doing, but you might report to your boss, that you are researching best ways to implement route matching algorithms, while you are actually learning about it for the first time. This does not mean I pull out a large book and kicking my feet up on the desk, but generally searching around for examples and working through them, to the point where I have a good understanding is acceptable. So, do not be intimidated by not knowing something, even on the job, as it is generally acceptable to learn as you go. Granted, some people have grumpy bosses, or there is a major deadline looming, and learning on the job is not acceptable. I am not talking about learning how to do for loops or something, as there is obviously a base level we need to be operating at, so take what I am saying with a grain of salt. But in most cases, in my experience across a handful of companies, learning on the job is an accepted practice. You hopefully you get the idea of what I am talking about.
I guess the root of what I am trying to say is that the technical unknowns should not intimidate you. A critical component of tech success is having a high capacity for problem solving, learning new skills, and basically grinding things out. This way of thinking, is not particularly hard, it is just time consuming. So, with this in mind, lets look at how we can break a larger problem of building this carpooling service into smaller chunks.
Working the Problem
Now that you know the idea we are going to develop, and the broad set of skills we are going to touch on, lets briefly chat about how this episode series broken up. Probably the best way to do that is through the use of a few diagrams.
It can often be hard to create something from scratch, in that you need somewhere to start, and it is not always obvious because you are starting from a blank slate. I have found that it helps to envision what the end product will look, illustrated by the minimum viable product on the right here, and then sort of reverse engineer it in our mind and on paper, back to the far left over here where we are sketching things down. So, lets briefly work though this and see what we get, as that will likely tell us how to logically group these episodes.
So, what do you think the major pieces, milestone, or problems will be? Well, we are going to need our carpool-matching algorithm for sure, so lets put that down. Then, what about solving the problem of turning a physical address into positional data we can put on a map, so we need to figure out the geo-location problem. Then, what about the route between our user’s house and the office, we need a way to put that on the map too.
We might have a second sketching step here, once we know more about our prototype and how things are going to fit together, so that we can use this as a feedback loop when building things for production. Then, we can use that sketching step to help plan the database design, programming the frontend, and also the backend.
We are also going to need to figure out how users are actually going to use the system. So, we will need to do some workflow brainstorming about having a signup page, where people can create group, how do they join group, what does that matching display page look like, etc. We can call this a type of interface design. Oh yeah, lets not forget actually building out the map and maybe a very simple prototype at the beginning.
So, hopefully you can see that we can kind of mentally reverse engineer steps to building what a product might look like without actually doing anything. Then we can use this as a map of the problems we are going to need to solve. These are kind of steps along to way to making our webapp.
You can also logically split the problems we need to solve into two phases. One might be a prototype phase, where we roughly prove the idea, and then feed in a bunch of example data to prove the idea actually makes sense.
The second phase, will be where we build something that is a much more polished product, and has things like user signup, group creation, group join, user login, password reset, email notification, etc. You know, all the streamlined user functionality that makes up a nice application. Basically, moving the idea from prototype to something people can actually use.
We never really chatted about how you solve these problems if you do not have the skill set needed. Worse yet, you might not even know all the steps, because you do not have enough technical experience to reverse engineer this fully. The unknown unknowns. So, here is the system I typically use to solve these problem steps, it is kind of a cycle I continually go through.
Most of the time, things start with a simple question, like, “how do I turn an address into position data that I can put on a map”? Well, I always keep a notepad of graph paper on my desk and continually jot down notes or sketch down ideas when working through a problem. I find that it help to write things down, drawing diagrams, and sort of put a face to the problem. Next, I suspect we all use Google to answer questions pretty much non-stop while working through problems. Personally, I find Stack Overflow, technical blogs, and Github search invaluable here. Then, I usually code and continually test my solution until I have solved each problem. It is just a matter of cycling through all these problems and chaining the solutions together into something that actually works.
It probably goes without saying, but we are going to run into many unanticipated problems not talked about here while actually building things out, it is just the name of the game when building anything for the first time. My suggestion, would be to keep it high level and not spend a crazy amount of time upfront planning, as things will change really quickly as you learn more about the problem. So, lets say on the far left here, that sketching down some rough ideas is problem number 1. We might have gone through this problem cycle 15 times before getting to where we actually have a working map, then a bunch more before we figure out how to turn an address into position data we can place on that map. You kind of get this idea. I guess my point of all this, is that even if you do not know how to do these things, just by asking questions and working through the problems, you can typically get to where you want to go. We might have solved a few hundred problems, gone down some dead ends, but by the end of this, we should have a bunch of solutions we can chain together. There is no need to be intimidated of the unknown here, it just takes lots of time, and you can basically grind it out.
So, as we talked about earlier, this is a high-level map of the problems we need to solve, and that translates nicely into a rough guide of episodes we can produce. We will likely tackle building out the prototype over the first few weeks, spending time proving the idea actually works, and looking at ways to improve match quality. Then, we will spend a few weeks polishing things for production use. All of this code will be posted to Github, so that you can hopefully learn how it all functions, as we work through it. Finally, the finished product will be hosted at https://groupcommute.com. If you haven’t already, I’d suggest signing up to get notified when it is released, so that hopefully you can save some money, and commute to the office with your co-workers.
I am also going to be posting these episodes to Hacker News, if you don’t have an account there yet, its and awesome place for developers, operations folks, and entrepreneurs to hang out. I have learned an incredible amount from that site. I’ll be watching the comments sections to answer questions. All right, lets jump back to the diagrams, I’m just about done, so lets wrap this up.
One kind of interesting thing to note is that you will typically have many of these little prototype side projects as you advance in your career. Most will rarely advance into the product phase as it just takes so much time and effort to create something that is polished. Most things that turn into products need to be maintained too, so there is an ongoing price to pay, that is why I mostly stick to the prototype area. There just is not enough mental bandwidth, to dedicate your time to too many of these product type ideas, so choose wisely.
It is time to end this episode. But, hopefully by now you understand the core idea, and how we are going to go about identifying and solving problems as we go.
I wanted to create this series, because I thought it might be cool to see the end-to-end lifecycle of how something actually gets built. Hopefully you will be exposed to something you have never seen before. I know, that as I work through these types of things, it is an opportunity for me to learn tons too. It is kind of a win-win, in that even if this product totally sucks, I am still producing something, which is hopefully useful, and learning lots in the process. So, you cannot really go too wrong. Last, but not least, I find that when you work on side projects, you can quickly amass experience and skills, along with your own repository of code. You can then fall back on these skills and code repositories throughout your career.