Tools: Command Line

As stated on my about page, I work as .NET developer in a company producing business software. I believe a traditional C# developer spends most of their time in a feature-rich IDE, which is Microsoft Visual Studio in most cases. However, during recent months I’ve been seeing myself gradually moving some of my activities to the command line, where in fact I can achieve a great deal of my goals at much lower cost than in a modern IDE.

My first conscious moves toward command line took place more than a year ago when I did Java development on my previous job. Like the .NET world has its most popular IDE, Java world has its own guys – for instance, Eclipse. Due to the simplicity of things that I built back then I haven’t faced most of the problems, which Eclipse-based developers lament about, though at some point it turned out that I actually want my job to be done a bit faster and a little different than the IDE allowed to. My primary concern was the need to quickly package a couple of JARs, obfuscate them and put the result into an appropriate place. It was easy to achieve something like this through doing all three steps separately: open project in Eclipse and assemble a JAR, launch obfuscator tool and feed the archive to it, put the result wherever it has to go – nothing complex. On the other hand, both launching Eclipse and the obfuscator took some time, with the additional seconds being spent on surfing through the Eclipse menus as well as locating the desired configuration files for the obfuscator, so after running the scenario several times manually I had no choice except for forming a strong desire to automate it. Because I also wanted to understand what Eclipse does under the hood when I ask it to compile the project I got some ant manuals and started automating everything away. In a couple of days I had what I wanted: I could open Powershell, type in ant ship and it would quickly do all the above steps on its own, so that I have only to decide what should be done with the obfuscated JARs. The economy of time that it awarded amazed me so strong, that I even went further and included into the build-file such targets as test and testncommit so that I could not only run tests from the shell, but also check changes into Mercurial in case the tests pass – sort of continuous integration on a single-man project. What made me stick with the command line then was the fact that it could save me from navigating through multiple IDE’s windows in addition to merely running things a bit faster.

Next time I searched for help from the shell during the work on my diploma project. At some stage of a major refactoring I still had a dozen or so very long and slow tests for fat components, which I was going to replace later. Since these classes were not touched during the most days until the time has come to deal with them, there was no reason to waste time on their tests during each run. However, it turned out that one cannot simply ask Visual Studio 2012 tests runner to ignore a couple of TestClasses or TestMethods. In addition to this, the IDE didn’t always successfully discover all the tests upon rebuilding. These little issues made me search for better options and thus find the vstest.console.exe tool, which simply runs the tests from command line. Not surprisingly, it allows to filter the tests being run, for example, by test classes names – that was precisely what I wanted at that moment. A nice thing to mention is that while the command required to run vstest utility with certain tests library and filters is quite long and awkward, most shells, including the PowerShell, allow one to set up different aliases to save some typing, so running tests becomes easier and faster than from the IDE – in addition to providing broader functionality.

Maybe the strongest push in the direction of command line I got from Visual Studio 2012 for C++, when I was experimenting with Cinder graphics library. Here the blocker was the environment’s speed: VS did everything deadly slow – sometimes so slow, that it was literally impossible to put in a line of code. I spent some time on tuning precompiled headers, which helped with compile time and even seemed to improve the situation in editor, still the state of affairs remained awful. What made me finally close Visual Studio and turn to compiling like msbuild.exe MyProject.vcxproj was the fact that in an attempt to increase the speed of IDE and fight the horrible lags I had turned off IntelliSense. While it didn’t help much with the original goal it made me ask myself why do I want to stick with the environment if I get no assistance from it? With quite modest support of C++ in the IDE part of Visual Studio (or without it after disabling IntelliSense) I merely could not find a reason to stay in the slow environment when I could switch to the Sublime + PowerShell combination boosting things considerably. I acknowledge that my Acer Aspire P3 slate with i5 processor is not a powerful machine, while C++ is more difficult to handle than, say, C# with which I did not face such awful quirks – I can even assume that it would be faster should I use a Professional version of Visual Studio instead of the Express one. Nevertheless, I can’t believe that it is so bad that not being able to enter a line of code in the editor for 5 minutes (let alone the timeframes when it builds the project) is considered OK. On the other side, once you turn to the shell, you are able to get binaries in a matter of several seconds after typing in the msbuild.exe MyProject.vcxproj command. In case you change the contents of the precompiled headers, you’ll have to wait a bit more, but at the same time you are not blocked by the build process and it won’t prevent you from looking through your files or switching to the browser. In contrast, if you build from Visual Studio, the only option you have is to get some coffee and take a walk, because it somehow manages to slow down the entire system significantly. This is another crucial characteristic of the command line tools – they tend to do precisely what you ask for, while the complex and shiny IDEs in their pursuit for the users’ satisfaction seem to be doing lots of stuff behind the curtains even when you don’t ask for it. Furthermore, since IDE has this beautiful and handy user interface, it also has a great option of doing some hard work on the UI thread, thus making user face unresponsive windows and feel frustrated, because their own tool stands in their way to getting the job done.

Another area where I quite often use shell-based tools instead of IDE functionality is talking to TFS. One particular example is checking out a file for edit. When the desired file belongs to a solution you work on, the task is pretty easy to accomplish in Visual Studio – you just locate it in the Solution Explorer and chose check out in its context menu. The situation, however, gets a bit worse if you want to edit a file, which is not included into any solution at all – just a file in the repository. In this case you have to open the Source Control Explorer thing, navigate it to the desired file (as far as I remember, unlike Solution Explorer it has no handy Search function) and check it out through the context menu – that is in fact a bit too slow because you have to switch to a certain window and do some folders joggling there. Does command line have something on offer to make the process easier? If you are used to navigating the folder trees like ‘first 2 letters of the desired folder – TAB, … ‘ then absolutely yes, because all you have to do to check-out a file through the command line is type tf checkout path/to/your/file.extension. Although, this is not that significant problem of the IDE and, in fact, I can hardly imagine the way it could have been done better, there are other tasks where Visual Studio clearly sucks in terms of working with version control – most notably, merging.

Since we support several versions of our product, merge is frequently used on a changeset basis, where you fixed a bug in one version and want to merge the fix into another one. What you have to do to accomplish this in Visual Studio is going to cause some pain, because before letting you pick a changeset the IDE will first ask the server for the entire list of those available for merging between the selected branches (or locations inside them). Moreover, in my experience the process of assembling this list sometimes takes up to a minute and in case there exist broken changesets it may finish with a failure message not letting you merge anything. Now, let me ask you a question: why would one want such a list of merge candidates in case they’ve just checked-in the changeset, which they want to merge and know its number? Why couldn’t there be a Merge button in the Changeset Details window in Visual Studio? I don’t know the answer to the second question, but I am pretty sure about the first one – I don’t need this list at all, because I can go and type something like tf merge 4_10 5_00 /version:C159001~C159001 /recursive in the shell and it will instantly merge the changeset 159001 from branch 4_10 to branch 5_00 if that is possible. Certainly, this example does not serve to prove that command line is vastly more powerful than IDEs – instead it just highlights the fact that there are ways to go in making the developers’ tools better and sometimes one doesn’t have to go very far to introduce a major improvement.

Overall, my programming activities, even those started in powerful modern IDEs like Microsoft Visual Studio, sooner or later led me to doing at least a portion of work in command line. While there were always serious reasons for switching from the fancy windows of the former to the awkward, 80s-style text-based environment of the latter, I still don’t believe that this is a sentence to the IDEs and their developers. Instead, I think programming environments are just in the process of becoming better and easier to use tools and sometime later, particularly when the human-computer interaction technologies will afford that, IDEs will give developers as much power and freedom as shells do for several decades already. Still, currently command line offers significant advantages and here are just some of them:

• Shell is a lot faster;
• It’s more controllable and does precisely what you ask for, leaving out a lot of yak shaving;
• In some sense it is easier to navigate;
• Shell also serves as an entry point to a great deal of things, including your text editors, compilers, diff tools, revision control systems, testing frameworks and whatnot;
• It allows to set up and hold a piece of context through the use of aliases, functions and variables, which also means that shell is quite customizable;
• Command line keeps your fingers at the keyboard, which is especially important when coding – even the numerous hotkeys in an IDE can hardly compensate for this;
• And finally, yes, shell gives one a feeling of being a hacker in one of those legendary movies from 90s. 

What makes you trade a comfortable IDE for a good old command line? Are there any particular tasks that you can’t accomplish in the modern environment, but which are crazy simple to do in the shell? Or maybe you never had to leave the IDE because it satisfies all your needs and never stands between you and your goals? Please, share your opinion!

Our Predictions Are Bound To Fail

Some years ago, when I did pay much attention to politics, I have found multiple articles by Valeriy Panyshkin (here is some info about the journalist, alas it's in Russian only), who wrote a lot about politics as well as numerous other problems, which one could see in Russia. Despite the fact, that many of his articles, opening my eyes to the things, which I didn't necessarily want to see, were somehow depressing, I read almost all of them. Moreover, even after I had become less concerned about these things - closing my mind once more - I continued to follow the man's writing. Interestingly, Valeriy gradually switched his attention from politics to a narrower set of issues - he got focused mostly on various problems that children and those who try to help them face in our country. He still highlights many other things too, although, as I see it, now, working with organizations that raise money for healing badly ill children or try to protect orphans' rights, he devotes most of his time, efforts and writing to these particular issues.

 

Recently I have read his article (the link - in Russian too) discussing a set of principles, upon which, according to Valeriy, every charity should base its work. The most surprising thing about the article was the idea that any organization of this kind must aim at ceasing its existence. Although this may seem a bad mission, the idea is quite simple and makes sense when elaborated. Valeriy doesn't claim that every charity should give up helping people and go find a more reasonable niche. What he states is that everyone who works in such a foundation should dream of the day when the problem, that he fights with, is finally solved: a treatment is found, which heals blood cancer in a matter of minutes, orphans receive such a crazy amount of support from government that life is as easy for them as it can be and so on. This way the main purpose of any charity is not only to help people, but to put its best efforts into improving the situation so that one day there will be no need for their help at all. This notion of aiming at becoming obsolete, although looked interesting, seemed to me something local to the field of charities at the first glance. However, it is easy to see that it is inherent (or, at least, should be) to many other kinds of activities and institutions - from medical research centers and police departments to ministries of Justice and parliaments.

 

Because I am a programmer it was particularly interesting for me to notice a trace of this idea in the world of software. First of all, there is a multitude of software development jobs, which pursue a goal of solving a particular problem, and once the solution is brought to life, its consumers, roughly speaking, don't need the awesome guys, who built it. However, this kind of relation between producers and consumers, although widely present in my field, is quite common elsewhere. Maybe its because it looks so usual, that it does not seem that interesting. It stays hardly exciting even if we consider all the benefits that many of us could get in case we did our best to make working with our products and extending them absolutely independent of our own presence. Still, I have finally stumbled upon another idea that looks somehow reminiscent of the principled proposed by Valeriy.

 

There is a system at my university that eases collection and aggregates information about the progress of students in the course of a semester as well as their performance during final exams. As I have undoubtedly mentioned in this blog, my diploma project is going to involve analyzing this data and making predictions based on it. What I want to do is to build a system that is capable of predicting the results of exams for a particular student from their performance during some part of a semester. Such a tool may be useful for faculty staff - for instance, they might want to know who's chances to be expelled are high way before exams. Wait a minute... The people, who teach and do different kinds of administrative work at the university pursue the goal of providing every student with quality education - they barely want anyone of their students to be expelled. So, for what sake would they want my software to say that certain students perform poor enough to fail examination and thus put an end to their education? The answer to this question is not subtle at all, because once you know that someone in your responsibility has serious problems - with study in our case - you gain a chance to help them. Thus, my system is not designed to allow teachers and other faculty staff to concentrate only on those students, who work well, but to draw their attention to those who may fail and to do this before it is too late to change anything. This way I am working on a system that will hopefully predict events that are not supposed to happen thanks to these same predictions. Furthermore, I am quite satisfied with this situation and will work hard to make its failed predictions as precise as possible. Clearly the notion of forecasts, that we make with the purpose of making them fail, is not equivalent to the idea of charities aiming at self-elimination, still I feel there is something similar to the nature of these two things.

 

Not surprisingly, I am not the only one who strives to make precise forecasts to see them fail in the end. The most popular example of applying machine learning techniques, demonstrated in various courses devoted to the subject, is predicting whether a patient is going to have cancer (or has it already.) When such a thing is done by medics outside the lecture-room, their main goal is to save a patient from the disease. The same holds for exploiting data on cardiac arrests or CVAs - healthcare field is rich with examples of this kind, as many other areas are. For instance, analysis of constructions is usually performed to find flaws in them and discover where the thing should be strengthened so that no disaster will happen - not to know where to strike. Aircrafts and vessels are overseen to avoid collisions, which are predicted first. The list is quite long and there is no need to continue it, because my point should be clear already.

 

Actually activities of this kind took place much earlier than the first computer was built. People always tried to foresee things, which they wish didn't happen, and they did this not only because of the desire to know the future, but because they knew that as soon as one is aware of bad things ahead, one can escape or prevent these. So I am not opening something new - I just feel fascinated by the simplest idea that was here since the first days of humanity or even earlier. The invention of computers merely gave us new powerful tools and techniques to foresee things, and it seems astonishing to me, how frequently we try hard to make precise predictions that are bound to fail just because we do succeed.

Software Bucket

I have recently come across a short blog post devoted to having a personal list of software pieces, that one wants to develop - a software bucket. Of course it was interesting for me to think over the possible contents of my own bucket. The first lines of my list are occupied by the things I am currently working on. The topmost is definitely a simple tool for arranging personal tasks. I know that there are plenty of such applications, nevertheless I have decided to create one more - mainly because I want a tool, which is as convenient for me as possible, and hope that someone else might also find it useful.

 

I have originally come to the idea of creating such an application almost a year ago, but it was only in the end of October when I have finally made up a deadline for the first version of the tool. Being fascinated by the new Windows 8 OS I have chosen to deliver my app through the Windows Store. The key advantage of this approach is that it will provide me with both the means of distributing my creation and the feedback channel - in case someone downloads the application. Beside that, Windows Store applications development feels quite suitable for me because I have got reasonable experience with .NET development and, particularly, with Windows Presentation Foundation.

 

Despite having done a significant share of work, I have missed the initial deadline, but feeling ashamed for this I have already set up the new one. I feel determined not break one more promise given to myself. After delivering the first version, on which I am working now, I plan to continue expanding this project and add some more features to help me and  potential users plan activities in even more effective manner. I will struggle to avoid making the app overloaded with details and inconvenient, so that other Windows 8 users will be able to consider it helpful and easy to use.

 

Another thing in the bucket is my current job project - I am developing a little toolset for generating manufacturing-related reports from the data provided by a complex PDM-system. Although the task may seem not very interesting, I feel quite absorbed with it, because it offers me a great chance to practice TDD and refactoring techniques to build and deliver an efficient and reliable piece of software, which is capable of giving valuable help to our customers. Additionally, I believe that the need to continuously align the thing to fresh portions of requirements will build me up towards multiple challenges that I will face in future.

 

On the other side, I was stupid enough not to test drive the application from the first steps, so it developed in a way which is far from perfect. However, I feel surprisingly glad about that, because for me this is an opportunity to feel all the pain caused by my own code and design decisions - now I am refactoring the project and supporting it with automated tests. Lots of interesting details of this piece of software and decisions made while working on it, as well as the desire to clean and develop it further, make it an important part of my software bucket.

 

One more item on the list, which is easy to come by, is my diploma project. This one is going to be a data analysis tool for monitoring students' progress data in my university and making predictions based upon it. I am not sure whether there is a substantial chance for it to be adopted and used by the university staff, nevertheless it provides me with important experience and offers a lot of interesting opportunities. First of all, I would like to explore existing data analysis tools and techniques and to adopt some of these to real data. Besides, while working on the project I have to use an external system, which is aggregating and providing the data consumed by my application. The fact that I have been working with it for a considerable period has taught me to put significant effort into shielding my own creations from the fluctuations in the consumed systems' behavior - positively, that's an important lesson. In this sense this looks a bit like my job project, where my tools retrieve data from an external system. My diploma and job feel even more similar because both have something to do with reporting activities, although the projects focus on different problems. The key aspect of the former is data analysis, while the latter revolves around providing flexible data retrieving and processing tools to bridge the gap between the PDM-system, that our customers use, and the reporting engines. My work on the diploma project, as I see it, will involve facing multiple software development problems, learning new techniques and, hopefully, obtaining valuable results - the project is worth the effort.

 

The rest of my software bucket looks much less clear. Being fascinated by the world of software development and its numerous applications, I cant say where exactly I want to direct my work. Moreover, I love the process of designing, writing and testing programs so much, that it is very easy for me to build things merely for the sake of building things. I feel that's a problem and I have to fight this passion sometimes, because it prevents me from focusing on the issues that are truly important, and not only is it harmful to me, but it can also become a source of pain for my employers and customers. On the other hand, this same passion for software allows me to make out a couple of things, which are attracting me as a young developer.

 

Among these things one can find code analysis tools. I have come across multiple mentions of such software recently and it so happened, that the idea to contribute to some open-source project to taste the idea of open-source myself has come to me almost at the same time. That's why I plan to search for a project in Clojure, which will allow me to practice understanding others' code and design, to make contributions to a system that may feel alien to me and to better understand functional programming and Clojure itself. I don't really limit the search to code analysis projects only - I acknowledge these may be absent - but I feel eager to work on a project of this particular kind, as this may help both in understanding the ideas behind code analysis and writing better programs.

 

All the items listed above are united by the concept of tool - they are designed to assist users with their work and provide them with some solutions to the problems, which they face in some area of their life. At the same time any tool is marked by the lack of autonomy and relies heavily on the signals from its consumer. No hammer drives a nail into wall itself - it always needs a person who will pound it on the nail. Definitely, that's not the only way software can function, and I really wish to participate in building an entity, that is capable of analyzing situation on it's own, making sensible decisions and acting correspondingly. I cant say that I am fully determined to work on, say, self-driving car, but I suppose that the area of autonomous (and autonomously developing) systems - either robotics or less physical applications of AI - may turn out quite suitable for me. Anyway, it won't be easy for me to reject an opportunity in the field.

 

The key aspect of such a 'self-driving' system is the ability to function in a self-directed manner without a flow of instructions from human operator, who serves as a link between the thing and the world around it. The idea that 'reality' of this world may be varied - in the sense that the world of the thing need not be the same as the world where I write these paragraphs - combined with my affection for video games leads to another piece in my software bucket: creating an AI for a video game is a challenging task. While I can hardly imagine earning my living in the game development industry, I could sooner or later participate in some open-source game project. Although the game development field may seem not serious, I still think that it can sometimes serve as a source of inspiration and insight for other, more serious areas. Furthermore, my minor attempts to create some simplistic computer games showed me that this kind of work is highly paying in emotional sense, because it is always amazing to play with your own creations.

 

So these projects and ideas constitute my software bucket today, although due to the reasons mentioned above the growth and expansion of the list are barely constrained. This said, I will have to practice saying 'no' a lot in the near future to avoid getting torn apart by my own curiosity and desire to build things. Still, I hope that willingness to say 'yes' will guide me towards doing what I can do best and therefore must do.