Combining Liberator and Korma to Build a Web-Service in Clojure

In a recent episode of the Code Speak Loop podcast I mentioned two Clojure projects: Liberator, designed to build REST services, and Korma, allowing to talk to relational databases easily. I’ve been working with these libraries lately and it turns out they play quite nice together. In this post and the related repository on GitHub I will show the way I combined Liberator and Korma to build a simple RESTful application so that anyone who wants to do something similar has an example. I did not put much effort into separating concerns and making code clean in this sample, still I think it conveys the general ideas properly.

Here we will set up a task-list application, which would allow to view, add and edit tasks over HTTP. There is a bit more to it on Github, but I won’t cover many of the details here. For a database I used a local Postgres installation with a very simple table structure – there is a schema.sql script in the repository. It should not matter much whether you use Postgres or not, although if you pick some other DBMS you will have to change the DB connection configuration in the application (see below). Besides, some problems may arise with timestamps.

Let us start with the database. To talk to it we use Korma, and Korma in turn uses entities. These are the descriptions of the database tables written in Clojure with a defentity macro. Entity definition normally includes a set of keys, a list of fields to select from the corresponding table by default and possibly a name of the table – if it differs from the name of the entity. Additionally – and that’s the coolest part – entities might include relationships, which allow to extract linked entities seamlessly.

(declare tag)

(defentity task
  (pk :task_id)
  (entity-fields :task_id
           :title
           :description
           :is_done
           :is_cancelled
           :created_time
           :finished_time)
  (many-to-many tag :tasktag))

(defentity tag
  (pk :tag_id)
  (entity-fields :tag_id
           :tag)
  (many-to-many task :tasktag))

(defentity tasktag
  (entity-fields :task_id :tag_id))

In our application there are only three entities – task, tag and tasktag. Both in task and tag we specify that tasks are related to tags with a many-to-many link – to do this we only need to specify the second entity and the name of the linking table (:tasktag) in our case. We don’t define any relationships for the tasktag entity – that’s because we need it only to insert and delete records, which link tasks and tags together. To achieve our other goals the relationships defined on task and tag are pretty enough. (Note however, that I don’t show tag and tasktag entities at work in this post – take a look at the code on Github.) You can find a lot of info regarding entities on the Korma site.

Once we defined the entities, we have to tell Korma where we want to get them from – that is specify a database connection. In Korma you do this by means of defdb macro passing it a connection description generated from a dictionary. I use postgres function provided by Korma that will setup all the required parameters for connecting to a Postgres database. There are plenty of other helpers like that in Korma – check them in the docs.

(def dbcon 
  (postgres {:db "libekorma" :user "postgres" :password "Aw34esz"}))

(defdb dbconnection dbcon)

Now that we defined the entities it’s time to give access to them through a resource. Resource is the fundamental concept in Liberator, which binds together various handlers and parameters, that define what will it do under which conditions. We create a resource with defresrouce macro and for our simple case we will specify only the :available-media-type – we deal with JSON, :allowed-methods – GET is enough so far, and a function to :handle-ok – this one will get tasks from the database and encode them in JSON format – that’s what users will get. Even this simple example shows that Liberator allows to manage a lot of HTTP-stuff without much ceremony. The most important part of the resource for now is the :handle-ok function, called when the resource thinks it should respond with 200 HTTP code – that’s what happens when user sends GET request because we don’t have any restrictions and in this case we should respond with a list of tasks.

(defresource tasks-r
  :available-media-types ["application/json"]
  :allowed-methods [:get]
  :handle-ok (fn [_]
        (json/write-str
          (select task (with tag)))))

To make it all work we have to do only one more thing: define a Compojure route that will expose the tasks-r resource. Its definition starts with ANY, which means that the route accepts any HTTP method. Liberator handles allowed methods through its own mechanism (:allowed-methods) and thus there is usually no need to make Compojure expect a specific verb.

(defroutes app
  (ANY "/tasks" [] tasks-r))

(You can check the project.clj for a list of libraries that we use.)

However, if you try to request some data from /tasks, you will likely run into an error message telling that the app can’t produce JSON output because it doesn’t know what to do with timestamps. Even though this does sound scary, thanks to the extensibility of clojure.data.json this problem is pretty easy to deal with – we just extend the Timestamp type with a simplistic implementation of the JSONWriter protocol:

(extend-type java.sql.Timestamp
  json/JSONWriter
  (-write [date out]
  (json/-write (str date) out)))

Now you can check that everything, including this last trick, works fine. That means one can retrieve the list of tasks with GET request and observe the records stored in the database (be sure to insert some for testing – there is a sample.clj, which can do this for you). This is not too impressive though, so let’s proceed and allow for tasks creation with POST in the same resource:

(defn tasks-request-malformed?
  [{{method :request-method} :request :as ctx}]
  (if (= :post method)
    (let [task-data (util/parse-json-body ctx)]
      (if (empty? (:title task-data))
        [true {:message "Task title missing or empty"}]
        [false {:task-data task-data}]))
    false))

(defresource tasks-r
  :available-media-types ["application/json"]
  :allowed-methods [:get :post]
  :malformed? tasks-request-malformed?
  :post!
    (fn [{task-data :task-data}]
      (let [data (into task-data {:created_time (util/cur-time)})]
        (insert task (values data))))
  :handle-ok (fn [_]
        (util/to-json
          (select task (with tag)))))

Here we add 2 things. First, when someone’s posting data to us we want to check that it complies with our requirements. Validation of this kind can be done in the malformed? handler of the corresponding resource. Particularly, for tasks we don’t allow empty :title, so for requests with bad title our tasks-request-malformed? function returns a vector of true (yes, the request is malformed) accompanied by an error message. If, on the other side, a proper title is present in the posted data, the function will return false – not malformed – and a dictionary including the parsed request data under the :task-data key. This illustrates the proper way to pass data between various decision points in Liberator: if along with the result of the check (true or false) you return a map from the handler, liberator will merge it into the context and downstream handlers will have access to whatever there is in the dictionary.

In our example the data provided by the malformed? is used by post! handler, which gets it from the context by means of destructuring. Beside this, in post we add the :created_time field to the same dictionary and call Korma’s insert with it. That’s it, we enabled creating tasks – core functionality is here!

One particular thing to note are the calls to the cur-time function. There is nothing magical about it – I just use it to abstract away instantiation of the Timestamps for the time columns in the database:

(defn cur-time []
  (Timestamp. (.getTime (Date.))))

If you take a closer look at the malformed? handler above, you’ll notice that it uses the parse-json-body utility function. This one combines two other functions and json/read-str to get the JSON body of the request from context, turn it into a Clojure map and transform its string keys into keywords. In other words, the function creates an easy to handle dictionary from a raw stream buried deep in the context. Be aware that the keywordify function used here is not recursive, so only the top-level keys will become keywords, while nested dictionaries will still have string keys.

(defn body-as-string [ctx]
  (if-let [body (get-in ctx [:request :body])]
  (condp instance? body
    java.lang.String body
    (slurp (io/reader body)))))

(defn keywordify [dict]
  (into {}
    (map (fn [[k v]] [(keyword k) v]) dict)))

(defn parse-json-body [context]
  (if-let [body (body-as-string context)]
    (keywordify (json/read-str body))
    {}))

Liberator allows to define a lot of various handlers thus opening doors for managing any particular condition in proper place and time. The general idea is that when processing a request Liberator will navigate the decision graph and execute handlers defined for visited nodes. In the example above we used only the malformed? decision point to parse and check the incoming request. Next, we will implement a separate resource for deleting and updating individual tasks and see how one can implement other handlers.

Let us start with something simple – deletes. We actually need to define only two handlers: delete! and exists? As a bonus, we will also implement one under :handle-ok to allow for getting tasks by ID – just because it is very easy:

(defresource one-task-r [task-id]
  :available-media-types ["application/json"]
  :allowed-methods [:get :delete :put]
  :exists?
    (fn [_]
      (if-let [task
        (first
          (select task
            (with tag)
            (where {:task_id task-id})))]
        [true {:task task}]
        [false {:message "Task not found"}]))
  :delete!
    (fn [{{task-id :task_id} :task}]
      (delete task
        (where {:task_id task-id})))
  :handle-ok
    (fn [{task :task}]
      (json/write-str task)))

Here we remove tasks in the delete! handler with a simple call to Korma’s delete with task entity and a where clause restricting the ID of the task. However, the function provided under the :delete! keyword gets called only in case the one specified with :exists? yields truth or a vector starting with truth – there is little sense to deleting missing tasks. Our implementation of the exists? handler attempts to select the task from the database by its ID and upon success returns it together with true. Here the pattern is the same as in the malformed? handler – we use the dictionary to pass data around so that , for example, downstream handlers don’t have to query database once more. In case you update or delete records it makes a lot of sense to retrieve them in the exists? handler and then use them when they are needed.

What makes this resource very different from the previous one is that it takes an argument – task-id. This might seem strange because otherwise resources look more like dictionaries, but that’s the thing that Liberator handles without any work required from us – we just accept this gift. As for passing the parameter in, we do it in the route definition like this:

(defroutes app
  (ANY "/task/:task-id" [task-id] (one-task-r (Integer/parseInt task-id)))

Now we can remove the tasks – and only the existing ones. Deletes, however, are very simple in comparison to updates, which we are going to implement next. First thing that we need is a malformed? handler that will parse and validate the request – we have already seen something like this in the previous resource:

(defn task-update-request-malformed?
  [{{method :request-method} :request :as ctx}]
  (if (= :put method)
    (let [task-data (util/parse-json-body ctx)]
      (cond
        (empty? task-data)
          [true {:message "No new values specififed"}]
         (and (contains? task-data :title)
            (empty? (:title task-data)))
          [true {:message "Empty title is not allowed"}]
        true
          [false {:task-data task-data}]))
    false))

The new thing is the conflict? handler. In our case, it is pretty simple and just verifies that the task doesn’t end up completed and cancelled at the same time – this is a forbidden state:

(defn task-update-conflict? [{new-task :task-data old-task :task}]
  (let [combined (into old-task new-task)]
    (if (and (:is_done combined) (:is_cancelled combined))
      [true {:message "Invalid state after update"}]
      false)))

As you might guess (or discover from the decision graph), Liberator invokes the conflict? handler for put requests somewhere between the exists? and put! handlers. This means that you already have access to data extracted by exists? and can check whether the update can cause any problems here, without turning the actual put! handler into a state validation mess. Note that the OK return value here is false – meaning no conflict.

Having this handler separated is cool because there are usually quite a few other things that you have to decide upon when executing update, so it might end up messy by itself:

(defn update-task [{new-task :task-data old-task :task}]
  (let [just-finished?
      (or
        (and (:is_done new-task)
           (not (:is_done old-task)))
        (and (:is_cancelled new-task)
           (not (:is_cancelled old-task))))
      finished-time-dict
      (if just-finished?
        {:finished_time (util/cur-time)}
        {})
      updated
      (into finished-time-dict
        (filter
          (fn [[k _]] (#{:title :description :is_cancelled :is_done} k))
          new-task))]
    (update task
      (set-fields updated)
      (where {:task_id (:task_id old-task)}))))

Our update-task function evaluates the :finished_time field if needed and passes it to Korma’s update together with the values coming from the user. It is important to filter the latter and exclude the fields that should not be updated under any conditions – e.g. :task_id and :created_time. That’s what we do when assembling the updated dictionary.

(defresource one-task-r [task-id]
  :available-media-types ["application/json"]
  :allowed-methods [:get :delete :put]
  :can-put-to-missing? false
  :malformed? task-update-request-malformed?
  :conflict? task-update-conflict?
  :put! update-task
  ; + a bit more - see above
  )

In the resource we specify these routines under the malformed?, conflict? and put! keys. Additionally, we have :can-put-to-missing? set to false, which prevents updates to non-existent tasks. The update-related functions do look quite complex, but they’d be absolutely dreadful without the separation of concerns offered by Liberator and that is its main strength.

Another cool thing about this library is that it automatically manages the execution flow in a fully HTTP-aware way. That is, when building an app, you don’t need to think, for instance, of producing appropriate error codes and giving them back to user in the required form – Liberator will do everything on its own. On the other side, it doesn’t stand in your way and allows to fine-tune the resources the way you need them.

As for Korma, it’s key advantege is simplicity – at least that’s what I love about it. It isn’t an overweight ORM or something – it just allows to run queries against a database from Clojure, but it does it in a very natural way. However, I also can’t leave out the relationships feature of Korma, which permits linking the entities together in a straightforward manner.

Our simple example also shows that these two libraries work great together thanks to how Liberator makes you consider only one thing at a time and how Korma simplifies access to data. I began using these about a month ago and I must admit that I truly love this way of producing REST services.

The sample project that I show here is a bit bigger than I managed to stick into the post. For example, it allows to add tags to the tasks and to ask for a full list of tasks tagged with a particular word. This doesn’t introduce many new concepts apart from demonstrating Korma’s relationships in action – I just tried to go further along the road of showing Korma and Liberator. You can find full sources on GitHub – do clone the repo and play with the code!

I will appreciate your comments – both regarding the examples here and about your experience at building web services with Clojure! If you spot an error in the code or just can’t figure out what a particular piece does – please let me know. I’ll be happy to fix the mistakes and explain what I meant!

CodeProject

Aging

I was pretty fortunate to be born in the middle of summer as this gives me two very well defined and perfectly positioned points in a year when I am willing to reflect on the results of recent months. In January everyone’s year ends to give way to a new one and in July my own age gets incremented – both events suit well for peeking into past and future. So, a couple weeks ago I turned 25 and this somehow made me think about what does it mean to get older apart from the fact that every year you have to tell different numbers when answering a question “How old are you?”

I mean I am not a complete idiot an I know that over the time of my existence the Earth has done roughly 25 full circles around the Sun, cells in different parts of my body died off and got replaced with new ones many times and all that stuff. In other words, something changed in the world and something changed in me as well: I used to be an infant, then a little boy, then kept growing larger and stronger and now I am this guy with a laptop in a park. I also know that from this point on I will likely grow weaker – not stronger, not until I put some serious effort into avoiding this. Still, physiological aging feels as boring as it is inevitable.

Being a bit more intelligent than a rock, I also acknowledge that there is a social aspect to aging. When I was born I wasn’t actually a member of any society at all and was hardly capable of being one. I gradually started interacting with my parents and other family members and being a part of a small group of people. After learning some stuff from them as well as other guys of my age one day I became eligible to enter a larger community – an elementary school. From there I advanced to the high school, then university – both of them taught me some crucial skills and knowledge making me potentially valuable for Humanity. Finally, somewhere close to the 7th of 6 years of my study at the university I learned that I can not only consume what society gives me, but also give something back. That’s when I took my first part-time job in software development. A year ago I finished study at the university and working as a programmer became my fulltime activity, so that I moved a bit closer to the production side of the consumption-production gauge. This is certainly a very important aspect of aging, yet the same as the biological part of the matter it doesn’t seem interesting.

Continuing the search for the “essence of aging”, I also remember of responsibilities: the older one gets the more responsibilities they are supposed to carry. Until some time you have only to be doing what others tell you, so you are only responsible for not breaking the simple rules they set for you. Later, one takes the new, sometimes less clear responsibilities associated with study or job. Furthermore, one may take responsibilities for others' well-being and well-fare when they get married and give birth to children. So, the general idea is that children are responsible for nothing, while grown-ups are in charge of everything and that’s what I was taught in school and elsewhere. Even though I believe that’s true – to the extent anything about the way human society functions can be true – that’s not something that I have been facing a lot during the recent years. I mean I did get new responsibilities when starting to work as a developer or when obtaining the driver’s license, but all these feel so natural that it is hard to define aging in these terms.

What does seem to be essential to aging for me is learning and, most importantly, learning that most things won’t do themselves. Learning itself is crucial and it is often true that the older you get the more you learn – sometimes that’s what makes you capable of going further. Still, it just happens to come along with aging. On the other side, understanding that some things are just not going to happen the way you want them until you push them hard into the desired direction comes to me as a manifestation of the fact that I am getting older. 

There are plenty of examples of how this simple idea might jump at one here and there. For instance, long ago the laces on my shoes used to get tied properly without any help from me. Later the magic vanished and I discovered that one has to put effort to get out of home without losing their shoes. Something very similar happens with keeping shoes clean – for some time while you are trying to accustom yourself with the laces, magic keeps cleaning your shoes, but later it evaporates as well. You notice that shiny footwear becomes bleak and dusty, dirt sticks to it and doesn’t go away overnight any more. This way every year one sees some magic being drained away from their world.

This thought is not something original or new, but it struck me this year. I mean my parents stopped cleaning my shoes long ago – that’s other things that drew my attention to it. For example, I always used to be a very slim person. During some periods of my life this was supported by my attempts to do running consistently, but even when I was much less physically active I remained slim and hardly seen any body fat. However, the situation changed dramatically about a year ago. Somehow, after I took my current position and started to work full-time at the office my body began to build up fat. I don’t fully understand why did this happen – the amount of physical activity didn’t change much between the university and the job. One of the reasons may be that my life became more determined than a couple years ago when I had much more options for where to sit with a laptop and study and when to change the place. Anyway, some months ago I noticed that I got much fatter than I have ever been before and that my body is not going to remain slim or fit without some help from me anymore.

I also made similar discoveries closer to the software development field. While I already knew that code doesn’t write itself (although, there is meta-programming!), it turned out that there is still some place for surprise. My Windows Phone development adventures taught me one thing for sure: the products that I make won’t market themselves – not until I put significant effort into making them stand out and forcing potential customers to see them. Put in other words, my customers won’t come to me until I reach out and find them. So, basically nothing is going to happen by itself in terms of distributing and marketing my products – I have to do something to show them, explain them and sell them.

All these “discoveries” of things, which I believed to be driven by magic, but which can actually be driven only by me, do feel obvious. At the same time I’m pretty sure I will have a lot more of them in future. None of them is too difficult to predict now – almost everything that I might rely on will one day require some payment from me and stop working silently the way I am used to. However, like with shining shoes, testing my code and marketing my apps, I will definitely be very surprised by each and every thing that didn’t seem to need any help to get done but suddenly starts to do so when I get a little bit older.