Search Algortihms in the Real World

In the middle of my university life I had a very strong desire to build a computer game. I was studying for a software engineer and still understood little about most of the software applications. At the same time I loved PC games, which clearly brought me to the idea that I should produce one - the one with the best gameplay, of course. Many of the guys with the same background had that wish and like most of them I didn't succeed.

I spent a fair portion of my free time thinking over the game's design, writing the explanations in a textbook and studying computer graphics. The problem is that in these activities I focused on the things that were easier to comprehend for me. I liked to think through game mechanics - for example the way damage is dealt to the player. I worked this question out on a very detailed level - in many cases descending to the names of C++ classes and of course laying out the calculation rules. With graphics, I focused on trying to make primitives move around the screen using OpenGL, because moving stuff looked like an essential part of game development. This all happened where neither setting, nor even key game features were defined - I hardly settled with the genre. Instead of looking at these high-level concepts of the game, I grasped the details, which I could easily stick into my head, without even understanding what are the key building blocks that I will have to construct a video game from. Speaking in algorithms language, I was doing a depth-first search not wishing too see the entire tree that I will have to traverse eventually.

Recent observations at interviews and during the normal work process brought me to the idea that such a mistake is common among many younger developers. When put in front of an unfamiliar problem they tend to catch some detail that they understand and pursue it to the deep. Because they don't take time to step back and take a look at the problem as a whole, they may end up with weak solutions suffering from various illnesses that happen in the software world. Most likely you will see multiple leaky abstractions and dozens of lines of ad-hoc code in the programs produced this way. And of course don't be too optimistic about finding there clearly separated layers, constituted of the objects that comply with Single-Responsibility Principle.

After acquiring reasonable development experience we tend to replace the depth-first strategy with the breadth-first search. This happens not only because older developers are slower beasts, but more due to the acknowledgement of importance of the full context for the correct solution to the problem. The mistakes committed earlier make us approach both high- and low-level decisions with great respect to the surroundings and see the drawbacks of diving into details before thinking of the solution as a whole.

This obviously makes a lot of sense when we speak of software design and architecture definition, but actually the same breadth-first strategy is applicable to such activities as coding and bugfixing. For example, when analyzing a bug it is tempting to follow the first breadcrumbs seen in the data in an attempt to understand the issue, but this way one may end ep trying every possible hint and spend hours exploring dead-ends. Alternatively, one may take time to fully understand the context of the issue, appreciate its complexity and collect as many facts as possible before even trying to make assumptions regarding the cause of the problem. In this case he will be able to see clearly which guesses are worth a deep investigation and which can be ignored altogether.

There is a similar situation with, for example, writing blog entries. For some people it is natural to produce a good article  simply by letting their thoughts flow onto paper. For others producing a reasonable piece of writing requires planning it and doing a couple drafts first. I don't belong to the first kind and going the depth-first way always results in frustration and wasting lots of my time on a couple short paragraphs. To be honest, sometimes its very hard for me even to produce the first draft from the start to the end - details kill me on the way. Instead, I begin with a rough outline consisting of several key points, then gradually add several sub-points to each of them. Only after this two-tier plan is there, I would start expanding the sub-points to rough sentences, which make a foundation for the draft. This looks very much like traversing the tree of the future blog post with the breadth-first search algorithm, except for the fact that here nodes and leafs are themselves created by the search process.

While I have enough development and writing experience to understand these things, sometimes I still find myself using the details-first approach where breadth-first search is more appropriate. To some extent, this happens because there is always eagerness to produce results and the depth-first path always seems short and clear - until you start going. But at the same time, it may be the case that in absence of the right words to distinguish the two strategies nothing forced me to make a conscious choice - the analogy with the search algorithms came to me only recently. In addition to making the choice clearly visible, it also brings the criteria for picking one of the two approaches. When you need to devise a good solution or just to understand a large and potentially complex problem, breadth-first search will work better - the theory says it guarantees the best result possible. If, however, you need the result early and don't care about its quality too much - for example when building a quick prototype to get an idea of the look and feel of a future product - you'd better use the depth-first search, as it usually yields faster. Sounds obvious, but somehow just having this analogy with the algorithms world in mind seems to make the task of selecting the right approach to any problem easier.

Focus

There was obviously little activity in this blog during the recent months and I feel great returning here. The blog always gave me an opportunity to step back and think over my actions and choices. However, this year I was so focused on a single thing - my day job - that only one post had made it here. Now I want to check with myself what this attitude gave me and what it stole from me.

Last February my priorities changed dramatically. In about two months after becoming a team-lead I discovered that whatever I was doing was not enough to do good with new duties. I was both overwhelmed with the tasks and problems that fell on our team and saw clearly that we don't perform as good as we could do. Part of my natural response to this discovery was to ramp up the amount of time that I dedicated to the job.

There are lots of reasons why I'm happy with this choice. Firstly, putting more effort into work made me learn a lot and build up new skills in the areas of my responsibility. Because it was my duty to process all the requests that arrived at our queue, I learnt to do this efficiently and obtained a lot of domain knowledge. In other words, it turned out that it's enough to push me into a partially familiar field and show no way out to make me learn it deeply. My expertise grew enormously over the last year and being really focused on the job helped here a lot (yes, there are lots of things in my field that I still don't know or understand not good enough - I'm speaking only of the delta). Moreover, spending a lot of time studying and resolving various issues - some being totally unclear at first - I not only learnt new areas of the domain, but also developed a skill of learning faster.

At the same time, through spending extra hours on duty I came to see clear that sometimes one cannot address problems by simply working harder. When you do a lot and it doesn't help, you start to see deficiencies and acknowledge the need for changes in one aspect of the job or another. A different edge of this same idea is that one cannot do everything on his own. No matter how hard you try, there is always less done than left to do. These discoveries, combined with high exposure to the issues that we face, constant analysis of our work and search for means to improve it - all of these being enabled by having extra time - certainly brought positive results and helped me develop myself both as a developer and as a manager.

On the other side, the same willingness to work more than 5x9 had a negative impact on the manager role of me, because having more time I could in many cases take the responsibility for any new urgent issue or task. While this attitude helped me tackle problems in time and develop my own skills, it hampered the team's collective progress. Any problem brings new knowledge and shows ways for development, so I simply stole a lot of opportunities from my fellow team-members and hampered knowledge distribution.

Generally speaking, while my attitude helped me grow in terms of knowledge, skills and career, it also shadowed both opportunities and problems. It is a very important lesson for me: whenever one decides to tackle a surge in the amount of tasks by throwing in more (his own) manforce, he misses an opportunity to find more intelligent ways to solve problems and to use them as the points of growth .

These are the effects of the extra work on my job, but there are also those that go beyond my office life. First of all, I left the attempts to do programming and software development in the outside - this means all side jobs and projects. While these used to bring new opportunities and ability to study unfamiliar areas, that's not something that I regret much. Being focused on one area allowed me to grow more than attempts to handle both the main job and one or two similarly looking activities would permit.

What I do regret is that leaving less time for myself I started to pay less attention to continued study and learning new things - both around software development and outside this field. I totally stopped exploration of new programming languages and tools and did much less learning in other areas than I used to - namely, I didn't complete a single online course over the recent months. That's certainly something that I shouldn't have forsaken:  these activities could both help me do my job better and make me expand my knowledge and interests.

Another thing that I dislike is that I quit writing for the blog. Here the reason is not only that the blog has become a silent and lonely place - it was never crowded here. The real problem is that writing less I lost the habit to deeply think over whatever I do and to search for interesting problems and irregularities in my activities. Of course, I do reflect on my decisions and actions, but without regular writing I hardly do it in a systematic and efficient way. Moreover, without posting to the blog I avoid sharing my ideas, which is not a big loss for society, but a shameful cowardice of me.

As with any choice, there do exist both positive and negative sides to putting most of your effort into one area of your life. I see these and despite the cons I am going to continue along the road that I started almost a year ago. However, even though I will continue to pursue that sweet feeling of exhaustion, I do need to make certain adjustments to address some of the problems. In particular I will certainly pay more attention to the supporting activities like learning and blogging - these allow to widen perspective and spot issues that stay invisible when you are constantly inside the problem-solving loop. At the same time, staying focused on my job I will have to pay more attention to the things that I am busy with. After all, being totally occupied with the day-to-day activities is a direct way to keep doing wrong things. So I only need more thinking, more analysis, more writing and the same amount of job. Good to know the solution to one's problems, huh?

Fixing the Wrong Problem, Once Again

The last week I spent two or three hours looking for the cause of a bug really close to the place where it was actually located – but not close enough. From the exact moment when I opened the bug description in Visual Studio everything pointed me in the right direction – towards a couple lines in our code base, fixing which would have resolved the issue. However, when reproducing it according to the steps outlined in the issue tracker and exploring the related behavior of the system I have caught a glimpse of something peculiar, that could have been the reason of my issue, and it made me waste a lot of time on trying to fix the wrong problem.

A particularly important detail about this new problem that I discovered was that its incorrectness didn’t look like something trivial and minor. It seemed to me that I was looking at a very severe bug spanning a number of components of our system. At some point, I was almost sure that I have found a really big problem, much broader than what the reporter could have thought. This feeling of stumbling upon something significant that might have far-fetching consequences quickly made me forget my initial objective.

Moreover, I was excited in a very negative way by the idea that I see something that other devs on the team have overlooked, and the desire to show everyone how cool I am pushed me even further away from what I should have been fixing. The fact that the newly found problem looked much more complex than what followed from the original description also meant that it was more interesting. Hence, my eagerness to fight it instead of taking a broader look and examining all the aspects of the problematic process.

It took me a couple hours of digging and a relatively long break to persuade myself that I should try and go through all the steps that cause data corruption and the related error. Initially, I have stopped my attempts to find the exact source of the issue once I had spotted something strange and unexpected. Should I just follow the description of the issue, carefully crafted by the QA engineer, up to the point where the problem is clearly seen, I would have reached the root of the problem much faster – maybe in a matter of minutes.

What happened instead is actually very familiar to me – that’s not the first time I make this mistake. From time to time, when tracing down a bug I face something strange-looking in the middle of the way and start exploring it. I do this despite being fully aware of the idea that any problem must be solved from where it manifests itself down to its actual reason – without taking any shortcuts. I know that one should under no circumstances skip any steps on this way, but I still do this occasionally and it never brings me close to where I would be happy to find myself.

This exact problem is what the strategy of moving slowly and in tiny steps aims to solve. For me the nature of the issue began getting clearer once I pushed aside the keyboard and started writing – with pen and paper. These tools make one work slower, they don’t allow to hack one’s way through the code and skip crucial points. Moreover, they facilitate deeper and calmer thinking, while rapidly stepping over the lines of code in a debugger or executing numerous variations of an SQL query usually descends one into a state where only wild guesses and ‘let-us-try-this’ hacks are possible.

My takeaway is once again that one must not rush to solve any problem that they happen to see – frequently what we initially perceive as incorrect behavior turns out to be desired in the end. Instead, one should work towards fixing what they aimed at first. In many cases problem solving works better if one maintains a queue of the problems – not a stack. Fortunately, even though our minds seem to be using a stack internally, we can always push extra problems out of it – to a bug-tracker or to a good old sheet of paper – so that the original, the right problem can be fully comprehended and solved, giving way to new glorious fixes.

CodeProject