Search Algortihms in the Real World

In the middle of my university life I had a very strong desire to build a computer game. I was studying for a software engineer and still understood little about most of the software applications. At the same time I loved PC games, which clearly brought me to the idea that I should produce one - the one with the best gameplay, of course. Many of the guys with the same background had that wish and like most of them I didn't succeed.

I spent a fair portion of my free time thinking over the game's design, writing the explanations in a textbook and studying computer graphics. The problem is that in these activities I focused on the things that were easier to comprehend for me. I liked to think through game mechanics - for example the way damage is dealt to the player. I worked this question out on a very detailed level - in many cases descending to the names of C++ classes and of course laying out the calculation rules. With graphics, I focused on trying to make primitives move around the screen using OpenGL, because moving stuff looked like an essential part of game development. This all happened where neither setting, nor even key game features were defined - I hardly settled with the genre. Instead of looking at these high-level concepts of the game, I grasped the details, which I could easily stick into my head, without even understanding what are the key building blocks that I will have to construct a video game from. Speaking in algorithms language, I was doing a depth-first search not wishing too see the entire tree that I will have to traverse eventually.

Recent observations at interviews and during the normal work process brought me to the idea that such a mistake is common among many younger developers. When put in front of an unfamiliar problem they tend to catch some detail that they understand and pursue it to the deep. Because they don't take time to step back and take a look at the problem as a whole, they may end up with weak solutions suffering from various illnesses that happen in the software world. Most likely you will see multiple leaky abstractions and dozens of lines of ad-hoc code in the programs produced this way. And of course don't be too optimistic about finding there clearly separated layers, constituted of the objects that comply with Single-Responsibility Principle.

After acquiring reasonable development experience we tend to replace the depth-first strategy with the breadth-first search. This happens not only because older developers are slower beasts, but more due to the acknowledgement of importance of the full context for the correct solution to the problem. The mistakes committed earlier make us approach both high- and low-level decisions with great respect to the surroundings and see the drawbacks of diving into details before thinking of the solution as a whole.

This obviously makes a lot of sense when we speak of software design and architecture definition, but actually the same breadth-first strategy is applicable to such activities as coding and bugfixing. For example, when analyzing a bug it is tempting to follow the first breadcrumbs seen in the data in an attempt to understand the issue, but this way one may end ep trying every possible hint and spend hours exploring dead-ends. Alternatively, one may take time to fully understand the context of the issue, appreciate its complexity and collect as many facts as possible before even trying to make assumptions regarding the cause of the problem. In this case he will be able to see clearly which guesses are worth a deep investigation and which can be ignored altogether.

There is a similar situation with, for example, writing blog entries. For some people it is natural to produce a good article  simply by letting their thoughts flow onto paper. For others producing a reasonable piece of writing requires planning it and doing a couple drafts first. I don't belong to the first kind and going the depth-first way always results in frustration and wasting lots of my time on a couple short paragraphs. To be honest, sometimes its very hard for me even to produce the first draft from the start to the end - details kill me on the way. Instead, I begin with a rough outline consisting of several key points, then gradually add several sub-points to each of them. Only after this two-tier plan is there, I would start expanding the sub-points to rough sentences, which make a foundation for the draft. This looks very much like traversing the tree of the future blog post with the breadth-first search algorithm, except for the fact that here nodes and leafs are themselves created by the search process.

While I have enough development and writing experience to understand these things, sometimes I still find myself using the details-first approach where breadth-first search is more appropriate. To some extent, this happens because there is always eagerness to produce results and the depth-first path always seems short and clear - until you start going. But at the same time, it may be the case that in absence of the right words to distinguish the two strategies nothing forced me to make a conscious choice - the analogy with the search algorithms came to me only recently. In addition to making the choice clearly visible, it also brings the criteria for picking one of the two approaches. When you need to devise a good solution or just to understand a large and potentially complex problem, breadth-first search will work better - the theory says it guarantees the best result possible. If, however, you need the result early and don't care about its quality too much - for example when building a quick prototype to get an idea of the look and feel of a future product - you'd better use the depth-first search, as it usually yields faster. Sounds obvious, but somehow just having this analogy with the algorithms world in mind seems to make the task of selecting the right approach to any problem easier.

The Inmates Are Running the Asylum

I recently finished reading The Inmates Are Running an Asylum book by Alan Cooper, which dives deeply into the question of how software design should be performed. There are great many interesting ideas and a lot of thoughts that anyone working in a software company may want to consider. Even though I can't agree with the views of the book to the point, I liked it and will try to cover both the thoughts that fascinated me and those that seemed questionable.

At the core of the book lies a simple idea that when we are going to manufacture something - possibly that's a piece of software - we should design it upfront and do that properly. Design at its core means establishing the overall structure of the product mainly from the point of interactions with users - down to UI design, but well beyond it. The term 'properly' is an equally sophisticated thing and expands to many different recommendations, warnings and tools. Some that stand out are: finish working on the design before you start coding; detach designers from coders; use certain techniques to channel design and limit the scope effectively.

The idea that we should put some effort into design before development can hardly surprise anyone. Even those who preach that a programmer should just hack the code till the product is born would likely agree that certain amount of careful and structured thinking has to precede creation of the first lines of code. The explicitly stated suggestion that coders should not do design and even that the designers should better be somewhat isolated from those who program is more controversial. Still, it makes a lot of sense, because programmers (and I speak of myself in particular) tend to focus on implementation details and reduce the domain to the level of their apparent technical capabilities. This strongly affects the point of view and ability to see what we are going to build, for whom and why - we mostly pay attention to how it is going to be built. At the same time, software should be designed not just to be buildable, but to deliver value in an easy to use manner. Our ability to provide value to the users is usually limited by poor understanding of their needs - that's something that most of the programmers faced at least once during their career. With usability we fail in a more subtle way - we make the thing easy to use, but only for the folks that look like ourselves, because that's what our experience and customs drive us to. In most cases, however, the users are people with a very different background, so we unwillingly make sure that they have hard time with our product. While these claims may raise some disagreement, they look valid, even though are not always the best to follow from the practical point of view.

At the same time, I couldn't agree with denouncing of the act of prototyping. The reasoning for claiming it bad is that prototypes tend to stick and in many cases to become the first version and the core of the final product, being least suitable for that due to their nature. That happens because programmers don't like to throw their creations away and managers, avoiding what looks like wasted effort, like it even less. While it definitely can be very hard to get rid of the code that is already written, I still don't think that the technique of prototyping a part of the solution should be disposed of. We should simply use it with care and be ready to dump weeks of our work into a waste bin because the goal of a prototype is not to serve as an actual basement for the final version, but to provide grounds for decision making that brings us to the finished product.

What makes the book very practical and valuable is the design tool that Alan offers - personas. The idea boils down to making up a narrow set of characters that depict future users of the product and allowing these fictional dudes to wholly drive the design process. Interestingly, the book suggests that the design team should give each persona a name and photo, assign user roles and thoroughly describe their background. Such characters would allow to speak of the product in terms of the needs of concrete, although unreal, people who are expected to use it. I must admit that I have never tried this approach, but it seems very powerful to me, because it gives a framework to ask proper questions about the thing in the making and to ensure that design stays on the right track, as we develop the product exclusively for our personas. This means that if there is no persona that needs a particularly fancy button, the button will never make it into the final design. The book goes even further with this suggesting that in most cases it is not right to subside to the customer's attempts to squeeze another feature from the development team unless our personas justify it. This idea rests on the assumption that a good design based on the right set of characters would cover the needs of the user, while anything that is injected into the product in spite of the design will give birth only to inconsistencies that will eventually ruin the software. Sure, rejecting client's request is a tough thing to do, but there is something to the idea of declaring clearly what your user looks like and staying by that declaration - it could certainly make some of our creations more consistent.

While I liked most of the core ideas in the book, it still raises concerns in regards to its practical applicability. The problem with the entire design framework presented by Alan Cooper is that it requires great upfront costs with the possibility of getting nothing but experience as an outcome. The biggest share of these costs comes in the form of the time spent without delivering any kind of a marketable product. In the modern world, when everyone strives to squeeze time to market as much as possible, it is difficult to take a path that promises to double it (in the best case). It also seems nearly impossible to incorporate this design and development methodology into an existing process - it can work great from the beginning, but when there are established constraints of an existing product, anyone trying to inject a serious design into a working motor will likely fail.

Overall, I definitely recommend this great practical book to anyone working in the software industry, as it gave me a new perspective into the development process and explained some very useful tools like personas. It also conveys greatly the idea that usability is born - or buried - deep under the hood of any software and can't be just added on top of a finished product, which is something that you want to keep in mind. Finally, a ton of examples - both related to software and beyond the industry - show that many things that we use nowadays are designed poorly and are still hard to use, which means there are lots of opportunities for making the world a better place.

User's Expectations and Business Process

During a recent discussion with my friends I have faced an interesting contrast between what looked like the right thing to do in a mobile application and what a more traditional UX common sense suggests. While the way I thought the application should work seemed pretty reasonable from the business point of view, my friends were reluctant to accept it. The strength of their resistance to my approach descended me into doubtfulness and made me search for flaws and the ways to align the app’s behavior to the expectations of potential users without losing the emphasis on the business process.

My task was very simple and can be described in general terms without diving into details. The application deals with jobs, which user is supposed to create. In addition to that, he needs to do some work with the existing jobs – particularly assign workers to them. In terms of UI, this boils down to several application pages, only two of which are of interest now. The first one provides the inputs to set various attributes of a new job. The second view comes with a list of jobs, which were just created but haven’t been assigned a worker yet – user picks one there and selects workers for it. Once this is done, the job leaves the list and moves further along the pipeline. The problem sits right between these two pages: views on how the transition between them should happen when user fills all the properties of a new job and presses Create  might differ a lot.

One approach, which you will find in many widespread feed-based applications and which looks quite natural, suggests that once the job is created it should appear on the top of the list and be the first thing to draw user’s attention there. That happens, for example, when you post something to Twitter or Instagram – when you tap Post in the corresponding application, you are transferred back to the feed and see your new photo or message added on top of it. This is the approach that my friends were proposing. It actually has lots of benefits: user sees the item they have just added and can not only check that everything is correct, but also take some action. For example, right after publishing this entry I will want to look through it to locate things like typos or extra blank lines between paragraphs, and it is great that the blogging engine shows me the latest post first. However, for my application it didn’t feel suitable.

What I have chosen to do instead was to keep the list sorted by creation time in ascending order – the older the job the higher it appears in the list. Moreover, the app doesn’t scroll the list down to the newest item when the user adds it. For my friends this made an impression that when the list is already long new jobs just vanish unnoticed somewhere under the tower of older ones. While that certainly looks like that, there are some reasons to lean towards this approach. In my opinion, these fully justify the initial confusion that can arise from the fact that the feed does not work the way Twitter does.

Firstly and most importantly, the application’s purpose is to assist an employee in their interaction with a queue of customers and the pool of workers. In such a situation, when service time is critical, the older jobs are likely the hotter ones, meaning that if you have a job, which lies unassigned for 30 minutes already, you definitely want to process it before the one created a moment ago. Furthermore, since customers are served in FIFO order, in a situation when the list of unassigned jobs is already long and all workers are busy, arrival of a new customer should not interrupt the ongoing process. Under these circumstances, the manager handling the queue – our user – only has to put the new customer on the queue and forget about them. It is not necessary and possibly even harmful to draw too much attention to this new customer – instead of messing with a client, who has to wait for their turn, we should first process the jobs, which entered the queue earlier. (Of course, I don’t mean that the manager should literally forget about new customers, but he must be focused on the ones who have already spent some time in the queue.)

These considerations make me confident about the second approach, which pays close attention to the process that the application is supposed to help with – maybe in favor of complying with user’s initial expectations. At the same time, the discussion that we had made me thoroughly consider the problem. The least important and most obvious thought that I had in relation to it is that our views on how something should work and look might be affected too much by the things, which we interact with on the daily basis, despite the fact that the difference between the two can be vast. Here I don’t mean that my friends are totally addicted to Twitter. Instead, when they voiced their concerns over my design I have found myself in doubt and seriously contemplating adoption of the more popular approach. It looks like we are too used to feeds of tweets, photos, links and news growing from the ground up and sometimes oversee the fact that not everything should work this way.

More important takeaway is that doing the right thing is not always the same as taking a popular and well-established approach to UI. This flows naturally from the previous paragraph and merely suggests that before jumping on the bandwagon of any time-honored and user-approved UX pattern we must carefully study the goals of our products and how they can help users achieve their own objectives. This means that behind any application there is a process, which we try to enhance, improve and optimize and it is vital to take every detail of it into account when designing the application.

Last but not least, what we face here is the contradiction between user’s expectations and the goals of the software. When facing something like this, we should not forget that keeping users comfortable and amused is not the only purpose of a mobile (or any other) application. It should be easy to work with a product and the latter must not stay in the way of doing one’s job, but a business app occasionally has to make the user uncomfortable and draw their attention to something that they might be unhappy about. In the end, kittens, food and beautiful landscapes are for Instagram, while business apps are all about doing business.

Testing: Factory Method

These days I am working slowly on a Windows Phone application. Making my first steps in this direction turned out to be quite frustrating, although this is a topic for another post. As a disciplined developer (ha!) I started the project with unit tests and while writing these I noticed that there is a thing that I do automatically now. Including a kind of factory method into each and every test class which I write has become a habit for me, saving a lot of time and effort.