Picking a simpler approach to aggregate data in Clojure

The only purpose that I use Clojure for now is talking to our Jira to extract some statistics for analysis and even this still brings a lot of opportunities for discoveries and revelations. The most recent task that I set for myself was to get data on bugfixing activities over some period of time and store it as a table for further analysis with Excel.

I wanted to transform a list of Jira items into a table that shows how many items of different severity each team member had fixed on a certain day. I already had a way to talk to Jira so the key part of the task was to aggregate the list into a table with 3 attributes and one numeric value - the count of items holding this combination of attributes. Aggregation is easily done with reduce so I only needed to chose the form of the result. My first natural response to a problem of this kind is to assemble a structure of nested dictionaries with values of attributes as keys and summed count of items as leaf values, something like this:

{ "2016-02-10"
    { "Ivan Petrov"  { "Major" 1 "Normal" 2 }
      "John Stone" { "Critical" 1 "Normal" 1 }}}

It turns out that Clojure 1.7.0 offers the update-in function that works greatly with nested maps. The thing takes the hashmap, a sequence of keys and a function. It would first retrieve the value currently stored in the nested map under the specified sequence of keys, apply the function to that value and store the result back under the same keys. Thus transformation of the list of items into an assembly of nested maps holding aggregated values will look like this:

(defn resolved-bugs-stats [issues]
    (reduce
        (fn [report {date :resolutiondate assignee :assignee severity :severity}]
            (update-in report [date assignee severity] (fnil inc 0)))
        {}
        issues))

This piece of code yields the figures that I want - the only step left is to transform it into a sequence of rows and this one took me much more thinking. Despite the fact that I was able to find a solution, I also realized that I don't need the nested structure at all. (That is availability of update-in turned out to be a misfortune).

The essence of my revelation was very simple: why would I build a map of maps of maps if I need a list in the end? Since Clojure lives great with vectors as keys I could just use a composite key and go with a one-level hashmap. This single level is still required if I want to have an easy way to sum up the count of items with certain values of attributes, but unlike a nested map it transforms very easy into a simple table. Here is the new function - it looks almost the same but produces a simpler result and, what's more important, makes the code that uses it way cleaner:

(defn resolved-bugs-stats [issues]
    (reduce
        (fn [report {date :resolutiondate assignee :assignee severity :severity}]
            (update report [date assignee severity] (fnil inc 0)))
        {}
        issues))

; generates a result in the form:
{
    ["2016-02-10" "Ivan Petrov"  "Major"]   1
    ["2016-02-10" "Ivan Petrov"  "Normal"]  2
    ["2016-02-10" "John Stone" "Critical"]  1
    ["2016-02-10" "John Stone" "Normal"]    1 }

While modern programming languages offer powerful tools to make complicated solutions real and cheap, there are few cases where the form of intermediate data structures need to be significantly more complex than the form of the desired result. Simply keeping this idea in mind and evaluating our solutions against it may help avoid some of the excessive complexity that we introduce when building systems.

Simple Models

Recently I've started exploring Microsoft XNA platform. In an attempt to get familiar with numerous powerfull tools it offers I've decided to use it to accomplish some relatively simple but interesting task that will make use of graphics but won't prompt me to focus on sophisticated graphics stuff. Due to some strange reasons I've choosen to start with developing Rubik's Cube puzzle game. The idea for application here is quite simple - the only thing my game must offer to a user is an opportunity to solve the well-known puzzle on a computer screen.

As for development,the task accomplishment, roughly speaking, requires two smaller problems being solved: firstly I should design a Rubik's Cube representation for storing and modifying tha state of the toy in line with player's actions and secondly I need to introduce routines for showing the Cube in an appropriate state on the screen. I haven't started writing graphics-related code yet, but I feel I'm almost finished with the first stage - that means I have developed a couple of classes representing the puzzle which provide convenient ways for performing any required in-game operation on the toy (e.g. rotating one of the cube's faces).

These recent efforts to develop a simple and convenient model of a real world object encouraged me to think over Leonard Susskind's words across which I had come some time ago. Speaking about the legendary physicist - Richard Feynman - Susskind had said:

"He [Feynman] truly believed that if you couldn't explain something simply, you didn't understand it."

[Leonard Susskind: My friend Richard Feynman - 9:30]

My intuition supported this idea long before I've heard these words, although I have never thought about it deeply. However this time the attempts to design a representation of a real world thing combined with revisiting Susskind's speech made me contemplate the thought.

The above quote suggests that to understand something one needs to have a simple model of the thing in mind - only possessing such a model one can explain something - that is share the model - and the explanation will be simple only in case the model is. In fact having a complete model of some object or process means understanding it - and vice-versa - so the only way to understand something is to study or design a model of the thing. A task of designing a model is usually a hard one and requires a lot of work to be done. Such a design process involves overcoming numerous problems which actually help the developer to form a better understanding of the entire object or process being described. Obviously it is difficult to design either a simple model of an object or a sophisticated one, but surprisingly the former are usually much harder to discover and develop. That means one often needs to do much more work, to face much more problems to achieve a goal of designing a simple but workable model - that is precisely one of the reasons why simpler models correspond to a better understanding.

However we actually don't design all the models we use ourselves - we tend to take ready ones and use these for our purposes. In this case one can't directly benefit from the difficulties connected with developing a simple model, though one benefits from the model's simplicity itself and really understands the things better than in case he would use a large and sophisticated one. That's because our mind's power is finite, hence it may be quite difficult for our brain to handle a sophisticated thing - roughly speaking it may fail to 'cache' a complex model completely with all of it's numerous although important details. I suppose we tend to unconsciously simplify the model if one is too sophisticated for us to handle, but in case our mind is not familiar with the model such a simplification may result in losing some important details which are just cut away and eventually this will lead to mistakes.

In this way simple models really produce better understanding through representing corresponding objects in a complete and convenient (particularly for our mind) manner. Moreover such models have another advantage over the complex ones: if your model is simple it is easy to share it because it can be explained simply.