Exploring a large pile of someone else's code

I am occasionally approached by students who are currently learning to program and are sometimes taken aback by assignments that look like "here is a mostly working app - go fix this and that to make it actually work". What they are looking for is some simple and reliable way to approach that huge pile of code that someone else has written and make sense of it, so that eventually it is possible to address the problems they are asked to fix.

Funny enough this not a student only problem - any working professional faces it whenever they join a new project that has already been running for some time without them. So for the sake of everyone who runs into this here are several things that I like to bear in mind whenever I run into a new code base.

1. Entry points

Any program (literally) has entry points - a limited number of code sections which connect the application to the external world. A CLI utility will normally have just one - there would be something akin to the main procedure that gets executed, whenever a user runs the utility. A web backend application will have a number of HTTP endpoints/routes which get fired whenever a corresponding request arrives (plus, again, the main function the performs the setup). A frontend application will expose a number of screens/pages available under particular routes. For example, in a React application with a router these would be some larger components mounted at particular routes. Or in the same React application you can treat the root component (who in many cases bears this proud an meaningless name App) as another such an entry point. All of these are the points in the code which get invoked by some external actions and start the journey of data and control flow through the code.

For any kind of a program you will be able to locate a number of such "entrances" and these serve as a good place to start exploration of the codebase due to a couple reasons. First, they are usually limited in quantity, which helps to concentrate. Second, and more important, when you start there, you have the benefit of understanding what happens to one side of the entry point - there is either a user or another system that performs some actions against the program being explored and thus 'calls' these entry points. This gives your exploration a better sense of direction - from an entry point you can only go deeper. Plus, it allows you to experiment - you can be that external thing - run the program a couple times or throw a couple requests at it and see how it behaves.

2. Database / persistent storage

Many applications (here I can't use that nice term "all") utilize some kind of storage - maybe they save some stuff to files or manipulate records in a database. In any case, these may aide your exploration efforts. Here you get a chance to understand what kind of data the program manipulates, how it choses to represent and store it. You may also experiment: throw a request at a web app and see how it affects its database.

The goal here is to grasp the program's data model and see how it is utilized - that illuminates a lot about the program itself. You would want to see which classes represent which kinds of data here or which functions create or read which records, and who calls them. Manipulating data is what most of the code is about, so understanding what data is at hand inevitably helps understanding the code.

3. Find yourself a simple challenge

When it comes to action, find yourself a very simple task that actually involves changing some code and do it. Maybe the original set of problems that you came here to solve includes something simple. If it doesn't - just invent it. When you're dealing with a frontend app such simple task can be tweaking the order of controls or changing the colors of some UI elements. If you work with an HTTP backend, try to add an optional filter to a GET request handler or to prevent updates for records based on any kind of criteria, which you can control.

The common place here is to keep this first task trivial - without even the goal of doing something meaningful. That's a good idea because at this stage you're still focused on getting a feeling of the code, learning to navigate it and validating that your changes actually have some effect. Once you're done with that you'll have some stable grounds under your feet: the code will feel more familiar, you'll develop some understanding of what's available to you and feel more confident about solving more complicated issues.

4. Breadth, then depth

Here is another thing that I keep returning to again and again. When facing something large you'd normally want to study what is it on higher level first, without going into details. If we speak of the entry points, identify all of them before trying to make sense of the implementation details of any particular one. When you're exploring the data model list all the tables / collections and how they are related to each other before studying in depth what and how each of them stores.

This is useful because having a broad understanding of what the whole thing deals with will help to have better ideas about the implementation details when you dive into them - having some context is always valuable. On top of that, many bits of code will work with several higher level concepts at once, so it's useful to have a rough understanding of what these are in the first place. If you get and overview of the landscape first, exploring the details will feel more like connecting the dots, rather than just wandering in the dark - don't get yourself lost too quickly.

5. Use tests

If you're lucky enough, the project that you're coming to will have some tests, which would serve as a superweapon to study it. Tests encode developer's assumptions about the code, so just reading them carefully may shed light on a lot of things. Moreover, you can change them and see what happens. And if that's not enough, you can actually play with the code itself trying to break specific things - tests will show you whether your assumptions about what breaks what are correct.

If there are no tests your position is weaker, but there is still a wat to go - try rolling up a couple unit tests for some bits of logic on your own. It turns out, testing someone else's code is the second best way to understand it, so don't ignore that. Just don't take that too far - your goal here is not to ensure 100% test coverage, but rather to make a couple assumptions about the code that you're studying and proof them right.

6. Refactor

I said testing is the second best way to make sense of unfamiliar bits of code. The best one is refactoring. In practical terms that means taking that part of code that you're interested in - a function or a class - and trying to rework it in any way that you find plausible with the main goal of making every step of it clearer to you. Don't get me wrong here - I am far from stating that it is easy to make code better when you barely understand what's going on there. The point is to try to rewrite something and thus understand it better and then just discard all of your changes. If you already have tests to support you, the effect will be tremendous - essentially you will reimplement the sections of code, which you intended to understand, and it is quite hard (although possible) to do that without developing a degree of understanding on the way. (If you don't have tests, you know what to do first).

7. Write it down

The task of fixing a couple bugs or implementing a feature in an unfamiliar repository is akin to the quest of entering a labyrinth in hope to find a couple gems in there, when you know what the gems should look like but have zero idea about the place itself. If the maze is fancy enough, finding all the gems will involve a lot of going back and forth and getting lost sometimes. To make it easier make yourself a map while you're traveling. Write down what you're after, fix your assumptions, chart the course, pin down whichever discoveries you make on your way, note what doesn't work and why.

This may sound like a lot of extra work and a major distraction, but it is not as time consuming as it sounds, while the benefits are grand. One reason, why you want to do that is it helps to get back on track when you feel lost of trying several things that don't work. Another is that taking time to make notes will help you avoid the rush of just trying to hack a couple things together. When you're familiar with the codebase doing things quickly can be fine - your deep understanding helps you make the right decisions about what to do where fast. However, while you're only making yourself comfortable with the program a more thoughtful, slow but steady approach will be more apt and will also equip you with deeper knowledge for whatever comes next.

8. Be brave and humble

Above all, don't be afraid to start and start small. Don't take a big challenge at once - break it down into smaller, easier to approach problems and handle them one by one. As long as you are taking steps, you will eventually arrive at some place. And if you take small and carefully planned steps, you will get there sooner and will see that it's the right place.

Three Starters

During the last weeks among other things I've been repeatedly facing each of the following three completely unrelated tasks:

• Work with an HTTP API either to test it or to run a series of files/data-based queries, for example, to upload some data to a new service or to check how a service responds to particular kind of HTTP load,
• Configure basic HTTP endpoints monitoring with alerts sent to a messenger,
• Create a simple NextJS website.

I expect to run into each of these tasks again more than once, so I assembled three starter repositories, which I can simply clone and reconfigure according to the needs of the next projects. All three are available on github - feel free to use in your own endeavours.

Clojure HTTP Playground

Use REPL to run any kinds of http requests, utlizing full power and interactivity of Clojure to process the data being sent and received.

• Link: https://github.com/aturok/httpplayground
• Based on: Clojure

Prometheus Blackbox Probing Starter

Set up basic available/unavailable monitoring with Telegram alerts for HTTP(S) endpoints

• Link: https://github.com/aturok/promstarter
• Based on: Prometheus and a couple tools for it, wrapped in docker containers

NextJS Website Starter

Bootstrap a simple website/landing, starting with basic styling, some degree of responsiveness, a couple reusable components and (hopefully) easily reconfigurable color theme.

• Link: https://github.com/aturok/toolynextstarter
• Based on: NextJS, react, styled-components, react-markdown and a ton of other things

Image magic

I've been facing the task of manipulating images for web apps and websites from time to time - like cropping, scaling and reducing file size. While some of these tasks can be done with a handy GUI tool like Photoshop (you have to pay for it), Photopea (a very powerful web-based photoshop-like editor that works with a multitude of formats including .psd) or Paint.net, sometimes a shell-based tool will do so much better. 

There is a tool like that and it's called Imagemagick. 

The thing let's you manipulate various raster images in different ways. I would most often use it for resizing and compressing images in batches and sometimes for cropping (here a visual tool may be better, but really depends on the scenario). 

For my own and other's reference, here are some of my usecases. Of course, there is an extensive documentation on the tool's website. 

(Since I'm on windows, I use image magick in Powershell to process many files at once).

Crop

Crop all files in the current directory and save them with a .crop suffix. This one reduces the size of the images to 1080x1732 cutting off 62px on the top (android screenshot).

Get-ChildItem -File | Foreach {convert $_.fullname -crop 1080x1732+0+62 $_.fullname.replace(".png",".crop.png")}

Scale

Scale all files in a directory down to 50% their dimensions (preserves aspect ratio):

Get-ChildItem -File | % {convert -resize 50% $_.fullname $_.fullname}

Compress

Based on this stackoverflow discussion.

Just one file, saving a copy:

convert -strip -interlace Plane -quality 85% .\banner_1.jpg .\banner_1.comp.jpg

All files in the current directory, replacing the originals:

Get-ChildItem -File | % {convert -strip -interlace Plane -quality 85% $_.fullname $_.fullname}

A lot of other usecases - do explore them!

How to handle a programming problem

A couple students whom I mentor through a software development course recently asked me for a generic advice on a way to approach a relatively large software development - something that would help them know where and how to start producing a solution. The question definitely provoked several pretty deep discussions. Since I do expect other students to seek advice of a similar kind I decided to devote a bit more time to the framing my advice.

While contemplating and discussing the topic I came up with a number of points that depict the approach that I use to build software. Sometimes I would follow this process to the point, in other cases I just adhere to the general line of thinking that it suggests - it definitely depends on the kind and size of a particular project. Thus, even though I present the approach as a sequence of steps, it is most reasonable to treat it as a description of a way of thinking about a programming problem.

Note that it so happened that I faced the question soon after going back to the Code Complete book to check a couple chapters. Moreover, I was reading How to Solve It by George Polya roughly at the same time. Because both of these have a lot to do with the topic at hand, the post definitely builds on the ideas found there. I absolutely recommend both of these books in case you didn't give these a try yet.

So, you find yourself with the task of producing a piece of software that would solve a particular issue. What should you do?

1. First and foremost, state your problem in writing - what are you trying to solve?
2. What requirements should the solution satisfy? Jot every one of these down as well.
3. Imagine the final solution, state in writing what the thing that you are about to build will look like. Focus on the general shape of the final result here. See how it will solve the problem and meet the requirements.
4. Think over the data model. Decide what kinds of data you will be handling, how to structure it better, where it comes from and where it should go. Imagine how the information is split into tables and collections, spot the relations between them. Consider which bits you need to store and how. Do this even if you don't have any kind of database - maybe building only a frontend application - data is something that you work with in any software and it definitely plays a great role in shaping the solution. Write all of this down.
5. Decompose your future system or program based on what you have understood so far - think over how you can split the problem and the solution into pieces. Depending on the scale of the task you may be thinking on the level of services, modules or classes and functions here. The larger the problem - the larger should be the size of units considered on this design stage.
6. In this process follow the breadth-first approach - e.g. don't go into designing functions before you produce an overview on the level of modules. List the large-scale units first, state their purpose clearly and only once this is done permit yourself to descend to the next level of detail. Notice when your start jumping between different levels of design - like thinking of modules, then functions, then modules again. If this happens stop and pull yourself back to the highest level that still needs clarifying.
7. Don't overdesign and keep moving. Most of the technical design will happen while you're coding. Your key goal at the moment is to draw yourself a clear image of the whole solution, its components and relations that govern them and to see how you will meet every bit of requirements. Design can consume infinite time, so don't let yourself get lost in it - details will come.
8. After facing your components think over your past experience - each component poses a smaller problem to solve and many of these smaller problems you may have solved in the past. If so, check which bits of previous projects you can use now - this may mean anything from pulling in a dozen existing files to just reviewing how you approached a particular issue a couple months ago.
9. Throughout all these stages use pencil, draw figures, sketch stuff, write things down - it helps you think. Once you have started producing a diagram, your mind will follow. Draw how the system or its particular component is decomposed, outline relations between the pieces, sketch the data flow, chart user actions sequence.
10. Devise a plan. Decide in which order you will build your components. Write that down (in any form or medium) - that will help you see and maintain progress, especially with longer projects
11. Pick the first component on the list. Jot down the steps to build it. Think how to test it, list the functions to implement, decide on the order, add checkboxes. If you don't know how to pick the starting point, pick any single one of them. At this stage it’s most important just to start the job, so start anywhere. Even if you choose a wrong initial task, you will soon be able to see and correct that.
12. Go and start writing the code. The first lines may go hard, but with all the preparatory work you should be able to break through them and the rest will follow easily.
13. Don't try to write perfect code from scratch - start dirty. Starting is a challenge by itself, so simplify the task and relax the requirements. It is much easier to produce a dirty solution and clean it once you see it working. Just don't forget to always clean the mess.
14. Write tests as you go to stay confident in the state of your code. Confidence does matter a lot here and the lack of it may slow you down significantly.
15. Iterate. Pick small tasks, complete them, polish the solution, celebrate, step back to take a look at your design and plan, repeat.
16. When you face challenges and are not sure where to go from there, take a pencil and think. State the problem that you're trying to solve, write out the required characteristics of the solution, draw, see it and then implement it.
17. Yes, look for what others have to say about your problem - stackoverflow, github and google are your friends. Don’t just grab any code that you find on the web though - take time to understand what it does, why it works and what kind of new issues it may throw at you.
18. When all else fails, go away for some time - take a walk, wash the dishes, smoke a pipe, use your hammock time. Let your brain wander away. Most times you will come back with new ideas. In the worst case you will get some rest and have more energy to continue attacking the problem.

And finally, if there are no more tasks to do, everything is polished and you feel happy about the result, celebrate the success, reward yourself, think over the lessons to learn from this project and go look for the next issue to solve. If you can program, there is an infinite number of problems waiting for you!

Takeaways from the Preparing Slides Course

About a week ago I finished going through the Presentation Skills: Designing Presentation Slides course on Coursera. It proved to be a rather helpful guide for someone fully depraved of the ability to assemble anything that looks good in PowerPoint. If you belong to this kind like I did, I fully recommend the course - that's a moderate investment that offers quick returns. Below are some of the key lessons that I learnt from the course:

1. While working on slides I must ensure three things:
• Focus – I should draw the attention of my audience to the most important idea on the slide,
• Contrast – the slide should communicate what is most important and what is the detail of secondary importance,
• Unity – the slide should focus on one thing or idea; I should search for things that can be removed.
2. Slides should be functional, look professionally and entertain when possible. Order matters.
3. There’s no such thing as “too much text” – rather “too much text out of context”. In other words, if the text is important it can be properly arranged and styled in such a way that the slide will be readable – the problem is usually how we present the text.
4. Allowed level of complexity (e.g. of a chart on a slide) depends strongly on the readiness of the audience to perceive it. The size of the audience is a good proxy for that readiness: the larger the audience the less ready it is to try to understand complex stuff.
5. Less decoration is good. Adding decoration doesn’t make the slide look good – introducing structure does that. Overall, I should try to remove as much decoration as I can – in particular in tables.
6. You can achieve a lot in terms of readability by means of good structure and typography.
7. Outside of the branding-related decoration/slide template I may use a maximum of two colors.
8. One of which is the color of the main text – black, dark gray or dark blue.
9. I may use another color for a couple crucial words or, better, icons and focal points like that
10. Max 2-3 words should be bold.
11. Bold means more important, italics means less important. Sounds controversial, but visually looks reasonable: bold stands out from the slide, while italics sort of leans to the background.
12. Font size should be used to introduce structure, which should communicate importance.
13. Photos should be large and few, icons – small and numerous.
14. Bullet lists can be arranged horizontally and they look better and more readable this way – because it allows to introduce clearer structure and contrast.
15. There are the Align and Distribute tools in PowerPoint – save a lot of time arranging stuff on the slide when building structure.
16. Plus a dozen interesting and practical details about typography, colors and visuals on the slides.

Failure Checklist

Just about a year ago - early 2018 - I decided that I am ready to quit the job and build a business of my own in the wild. I won't go through the details of how I came to that and what happenned next - the important part is that after 8 months I have found myself running out of cash and not knowing what I can do except to look for a new employer.

By the end of 2018 I have successfully landed a job - a great one, by the way. So now I am safe and already had some time to contemplate my attempts at business. Here are the key conclusions that I made from the analysis of what I did and how that led me to the failure:

1. If you go into business, you absolutely have to design a sales proposition that would clearly show your potential customers what you are offering, how much it costs and exactly how much they would benefit from it. In other words, you have to present your customers a cost-benefit analysis of your offering. Once you have it, advertise it a lot and do be ready to pay for the ads.

2. Be sure to prepare a financial model of every business project that you are trying to build. If the model shows adequate income, don't abandon it even if you see only modest earnings per unit - you just have to scale it properly.

3. Your business model doesn't have to be cool or show few competitors, but it absolutely must allow for reasonable net income in the target market. Other things do not matter much.

4. You have to be ready to overcome difficulties, do something when it's not clear what to do and ask other people for help at solving the problems of your business. The only allowed reasons to abandon a business are poor cash flow or the act of discovering why your model can't generate proper income. No other difficulties can be a valid reason to quit trying.

5. You have to go and sell your product in each and every possible way - including those, which are least comfortable (or even painful) for you. You must not stop trying to sell until you understand why your attempts fail and why you can't fix that for an acceptable price.

6. Your work must be focused on closing the key needs or problems of your customers in the first place - not on something that you find important or cool. Whatever you add to your product or service must be helping you sell it. Of course, you have to start with identifying what are these things crucial to your existing and future customers.

7. It is important to extrapolate your positive experience and translate every successful deal into a clear offer that would be interesting for a wider market. Once you figure out how to adapt what you have done for one client to the needs of many others, go and sell it to them.

8. Focus on just one or two (absolute maximum!) projects at a time. Don't dive into any new shiny projects until you finish the current one. Remember, there is one good reason to abandon a project: either it fails to generate adequate net income or you absolutely can't find a way to live to the moment when it would finally start bringing that income.

These thoughts look obvious, but when it came to the actual work I managed to violate each and every one of them. I learned the lesson the hard way, but the next time I venture into something new, this will be a good checklist to validate what I am doing on a daily basis. If, on the other side, you are just about to start a business of your own, this may help you avoid the mistakes that I made.

The Moneyball Takeaways

I recently watched the Moneyball movie - a biopic after Billy Beane, the manager of the Oakland Athletics baseball team. It’s hard to imagine anything less relevant to me than baseball, but the movie is in large not about the game itself, but rather about how it’s managed and changed, so it did hit the right strings. 

The plot revolves around a baseball team manager trying to change the rules and build a team that would win under severe budget constraints - totally about management and, particularly, change management. Below are the key thoughts that got running through my head by the time the end titles made it to the screen.

1. If you gonna change the rules, there would be people who will oppose you. No exceptions. 
2. If you gonna change all the rules, almost everyone will oppose.
3. There still will be people who support you - keep them, teach them, trust them and respect them.
4. Building a winning team is not about picking the top players - it’s rather about picking the right ones, who cover the needs of the team.
5. The needs of the team is what it has to do to win - preferably, expressed in figures. Nothing else matters.
6. Winning is all about performance, performance is all about measurement - neither is about whom you like and whom you don’t.
7. While building a team you will certainly have to hire people, but likewise you will have to fire someone.
8. By the time the game begins the manager’s job is done - they can only watch how it develops and make a note of any required adjustments.
9. The fact that you succeed at changing the world doesn’t mean that you win.