Over the course of my career in software development I got used to the idea that the key thing we deal with everyday is complexity. It shows up on different stages of our jobs, but most interesting cases are of course tricky bugs. Sometimes the issues reported by our QAs and especially by end-users get so messy that it takes a full crew of professionals to fix them. I have faced one such example recently and in addition to providing valuable experience and the feeling of satisfaction in the end, it served as an evidence supporting that all the roles we have in our team are of great importance and play best when assembled into a team.
About a year ago we have fixed a couple minor issues in our system. To be sure that we have that piece of functionality working properly despite any future changes we decided to create automated tests for it. Our QA engineers did get to this task, but because life is what happens when you have other plans, that happened only recently. Roughly at the same time another member of our team has found an issue in a distant part of the system, totally unrelated to that first bug, and fixed it using a certain code construct.
Once these two things met in our repository the new tests failed. It so happened that the developer who was digging into the failure wasn't in any way familiar with any of the two changes. The error conditions turned out to be tricky, so some folks joined investigation and it was eventually traced down to that seemingly unrelated last fix to be routed to the appropriate platform developer and fixed there. The importance of this event however took some more time to be recognized.
Another chapter of this story starts several months earlier. One of our customers filed an unpleasant data bug that we never managed to reproduce and understand. It was hanging there unresolved and made partners, so to say, unhappy. It's high severity kept it visible for us, but we still weren't able to do anything about the issue - till the moment when the events described in previous paragraphs took place. When I was doing a daily review of recent and priority items, I stumbled upon this nasty bug first and then spotted the one that caused all the mess outlined above. Something about them looked similar. The way the latter one was fixed suggested a new idea for repro steps for the former. Late at night this fresh approach was attempted and yielded positive result - we not only understood the reason behind the high-priority issue, but also knew that it is now fixed and could report that to everyone concerned - the problem was with the same code construct and the same platform issue that started the first bug.
This basically means that the data problem that we were so worried about took us a couple QA engineers, two developers, a product manager and a team-lead doing their usual jobs to track it down and fully understand - not to mention the significant amount of time spanned by all the activities that led us to this result. This might indicate problems with the product being developed or with the process, but the truth about software development is that such problems that eat enormous resources do happen and, if you are working on a large and sophisticated piece of code, you can only minimize them - not get rid of them completely. Complex issues also require different people to handle them - sometimes just to throw enough pairs of eyes and points of view at the problem. Even worse, there is an element of a slot machine to this requirement, because you can only partially design how and when someone of your team-mates will play his role in such a process - in most cases these revelations and discoveries happen merely because a number of people are doing their work properly.
Software development is sometimes messy work - especially when we speak of business applications. I would like to say that it doesn't have to be like that, but it seems that for most products getting complex enough to give birth to problems of this kind is just a matter of time. As a result we do need all of our people in their positions - QA engineers, developers, analysts and so one. This incident that we faced highlights the fact that even when I have no clue at all about the job that some people are doing at our office, it is still very valuable. The beauty of a development company and its processes is that one day any one of us - or rather an unexpected combination of our efforts - may save the company from something nasty or bring it to a new solid achievement. This simple idea is also a great motivator, because when I get tired with the routine and think that the work I am doing is useless, I can remember how almost the whole our team won the battle with that bug and see that one day me doing my job may save some thousands bucks for the company and countless hours of work to my colleagues.
