MagnionLabs.com - Blog

Rails testing 101

2009-11-29T14:00:00Z

From my completely unscientific survey, it appears a lot of people have still not taken to writing tests for their Rails projects. Mostly because either they don’t know why and how much they should be testing or unfortunately, because they feel overwhelmed by the myriad options: Should I checkout Rspec? or may be Shoulda? what is autotest? Should I learn about mocks and stubs first?

If you are one of those feeling overwhelmed, firstly, don’t feel bad. Here’s a guide about why, what and how of testing for the completely uninitiated. Simplified, especially for you. (The rest of you can scroll over to the next article please).

I will use the Test::Unit framework that comes built-in with Rails and at the end, cover all the other options available.

However beautiful the strategy, you should occasionally look at the results. —Sir Winston Churchill

As promised, firstly, the whys

As you probably know by now, Ruby is an interpreted language and along with the upside of faster development cycle, there is a downside of not having a compiler: there is no syntax or type checking after you write the code. Executing the code is the only way to know if its valid and tests are a great way to execute code.

Tests also provide you a guarantee against any regression in your code. If you have ever added a “feature” which broke something which was always working in the past, you know what regression is. Tests allow you to guard against such breaking changes.

A gentle introduction

If you have been testing your app through the browser (of course you do, silly me) or by trying things out in the console, writing test takes only a little bit more effort than that. Writing tests basically automates your steps and thus saves your from repeating the same steps in the browser or console, later. If you have never tried testing your Rails app, this is how you should start:

First, prepare your test database:


$ RAILS_ENV=test rake db:create 
$ rake db:test:prepare

Next, write some fixtures: Fixtures are the initial contents of the test database, before a test runs. Just like you keep some data in your development database while you are testing your app through the browser, fixtures are a way to populate some seed data in the test database. Rails automatically creates a fixture file for each model in your project.

Let’s say you are building a bug tracking system where you have a model called Ticket. The fixture file for Ticket should contain an entry for each row that we want to insert into the database. Following fixture file has two entries. Each entry is given a name. the name is significant while setting associations.


# fixture for Ticket
# file: test/fixtures/tickets.yml
bug_filed_by_john_customer:
  title: "Signup form does not retain previously filled data on user error" 
  description: "some more explanation here" 
  type: "bug" 
  state: "open" 
mary_managers_question:
  title: "How can I change the color of the shirt once it has been added to the cart " 
  description: "I want to ..." 
  type: "question" 
  state: "assigned"

Fixtures are as simple as they seem – each of these records gets loaded as rows in the table. There is just one thing to learn here: do not initialize the id columns and if you have associations, the names you have given to the rows can be used to fill in the foreign key columns. e.g. suppose, you also have a Update model which captures updates on tickets by different users.


# fixture for Update
# file: test/fixtures/updates.yml
question_by_peter_developer:
  ticket: bug_filed_by_john_customer
  description: "please provide steps to reproduce" 

response_by_john_customer:
  ticket: bug_filed_by_john_customer
  description: "here's how you can see the problem for yourself ..."

So, specifying the row’s name (bug_filed_by_john_customer) would populate the foreign key (ticket_id) in updates table correctly.

Various types of tests

Rails recommends that you divide you tests into three categories. Unit tests, which test your models; Functional tests, which test controllers and integration tests, which tests almost the whole application end to end. We will learn about all of these.

In the directory test/unit you would find a file corresponding to each model and similarly in test/functional a file for each controller.

Unit tests

Writing unit tests in very similar to trying out stuff in Rails console. Continuing with our example, let’s say the Ticket model is defined like this:



# file: app/models/ticket.rb
class Ticket < ActiveRecord::Base
  has_many :updates
  validates_presence_of :title
  validates_presence_of :description
  validates_inclusion_of :type, :in => %(bug enhancement question)
  validates_inclusion_of :state, :in => %(open in_progress not_reproducible closed)
end

You should be testing all the code you have added, like the validations above.


# file: test/unit/ticket.rb
class TicketTest < ActiveSupport::TestCase
  test "should have title and description" do
    ticket = Ticket.new
    assert !ticket.valid?

    ticket = Ticket.new(:title => "some title", :description => "some desc")
    assert ticket.valid?
  end
end

Then run the test:


$ rake test

You can of course use the data that you have put into the database through fixtures. Suppose, your model code put restriction that you couldn’t have two tickets with the same title:


  test "each title should be unique" do 
    ticket = Ticket.new(:title => tickets(:bug_filed_by_john).title)
    assert !ticket.valid?
  end

As in this test, you can access the models loaded by the fixtures as: table_name(:fixture_name).

While writing code and trying it out, you should add a test whenever you discover a bug. Suppose you got a customer complaint and figured description should be made into a text column rather than a string so that it can hold more than 255 characters. It would be a good idea to create a test for that. It guards against description ever being made into a string again.


test "description can be longer than 255 chars" do
  ticket = Ticket.new(:title => "some title", :description => "a" x 500)
  assert_not_raised ticket.save!
  assert_equal 500, ticket.description.size
end

There are a number of such assert functions. Here’s a list of the useful ones:

assert(boolean, “optional message”)
assert_equal, assert_not_equal
assert_match, assert_no_match
assert_nil, assert_not_nil
assert_raised, assert_nothing_raised
assert_valid(ar_object) # calls valid? on the ar_object
flunk(“message”) # always fails
assert_difference ‘expr’ do … end, assert_no_difference

Since tests are normal Ruby code, you can use usual techniques to organize your test code eg. Loops, separating out common pieces in a function, etc.


test "ticket when created should have a correct type" do
  valid_types = %(bug question enhancement)
  invalid_types = %(not_reproducible incomplete)

  valid_types.each do |valid_type|
    t = Ticket.new(:title => "some title", :type => "valid_type")
    assert_valid t
  end
end

Functional tests

Functional tests are meant to test controllers. Since, you have already tested the models in unit tests, you should focus on test only the logic that’s present in the controllers. For example, here’s a sample functional test:


# file: test/functional/tickets
test "access should be restricted to only logged in users" do
  get :show, { :id => tickets(:bug_filed_by_john).id }
  assert_redirect_to login_path
  assert_equal "Please login", flash[:notice]
end

As you can see, functional tests give you a method called get (also post, put, head and delete) using which you can simulate a web request to an action. Then, use the assert methods to check up the response. On top of what you saw with unit tests, functional tests also have these additional asserts: assert_response assert_redirected_to assert_template

For the most part, views are not tested here. You can test them (see assert_select) but it can get cumbersome.

Integration tests

Integration tests simulate a continuous session between one or more virtual users and our application. It gets the closest to how the application will be used in production.

In integration tests, you can send in requests, monitor responses, follow redirects, and so on. Ideally, a integration is something your customer should be able to understand – there are no models and controller and other internal details.

Instead of using the Rails build in integration tests, you can use webrat to make writing the tests lot more easier as it lets you describe a test in pretty much the same steps as you would do things with a browser.


$ sudo gem install webrat

In test/test_helper.rb, right at the end of the file, add this:


require "webrat" 
Webrat.configure do |config|
  config.mode = :rails
end

Rails does not generate any integration test skeleton automatically. So, use the generator:


$ ruby script/generate integration_test ticket_life_cycle


# file: test/integration/ticket_life_cycle_test.rb
  test "ticket life cycle" do
    # a users signs in
    visit new_session_path
    fill_in "Email", :with => "john@customer.com" 
    fill_in "Password", :with => "john's password" 
    click_button 'Sign in'
    assert_redirected_to tickets_path

    # creates a new ticket
    visit new_ticket_path
    fill_in "title", "new problem..." 
    fill_in "description", "some description about it..." 
    click_button 'Save'

    # someone adds an update to that and marks it fixed

    # customer adds another update that it works

    # engineer closes it
  end

As you can see, with webrat you can write the tests in a clear declarative style. It works like a web crawler similar to Mechanize or Watir. Webrat also does some verifications on the page. For example, here’s what click_button does according to the documentation: click_button verifies that a submit button exists for the form, then submits the form, follows any redirects, and verifies the final page was successful.

Here are some useful webrat commands:

visit <route_or_url>
fill_in “name_or_label”, :with => value
click_button “name”
click_link “name”
select “option”, :from => “name or label”
choose “id of option”
attach_file
check “name”
reload

All the asserts you have seen earlier, like assert_response, assert_template etc. are also available. You can also access the database, like in unit tests, to verify stuff.

More testing

There are some more kinds of testing you may want to. Most importantly, performance testing, where you determine how fast your application is performing and where are the bottlenecks in getting a higher performance. Put it simply, you first have to do benchmarking to determine your application’s current performance and then run a profiler to see the relative performances of methods for that run. Rails also allows you to test routes and mailers.

Additionally, you can do browser testing to determine whether it runs reasonably in all major browsers.

Ready to move ahead

There are few more tests related concepts to be familiar with:

Stubs: Often you have a module or over the network call in your application that you don’t want to invoke everytime the tests are run. e.g. Credit card gateway or an external storage. Stubs allow you to mask those calls. Create a file like test/mocks/development/file_name.rb where file_name.rb should be the same name as model, controller or lib file that contains the class you are trying to stub out. In this file, just reopen that class and refine the relevant methods as stubs.

Mocks: As you can see the functional tests also load the fixtures and execute the model classes, although, models have already been tested in unit tests. So, you can substitute model classes by mocks and avoid loading fixtures and make the tests run faster. On top of this, mocks check if they are used as specified or else they raise an error, giving you an additional level of testing. Have a look at Mocha if you want to use mocks.

Once you have mastered writing tests, you can try to write your tests before you write the code. This is similar to the idea of developing HTML/CSS mocks before writing backend code for your application. If you write tests first, you would be writing the code to a predetermined specification. This practice is called “Test driven development” or TDD.

Now, some people have gone ahead with the idea of tests serving as specification and called them, well, specs. This style of development is also called “Behavior driven development” and most popular framework that supports this is Rspec.

There are several other testing frameworks you can try e.g. Shoulda and Zentest. There are some useful plugins like Factory Girl, Machinist (both are replacements for fixtures) which makes writing tests easier.

There are also tools which help with testing like autotest which runs your tests automatically whenever the code changes. Of course, continuous integration packages also run tests.

If you are interesting in seeing how much testing you have been doing:


$ rake stats

will show you how much test code you have written compared to application code. For more serious mesurement you can try calculating code coverage using rcov.

Tips

Aim of developer testing is not so much to find bugs – if its does, consider that a bonus. For a complex application, you should definitely prepare a test plan and do more comprehensive testing, preferably by involving a test engineer.
Organize your test code. Use setup method, create helper methods in test classes
Keep each test method small and test just one thing in one test
If you are following RESTful architecture and putting all your domain logic in models, write unit tests and do integration testing thoroughly. Functional testing can be de-emphasized.
In the beginning, don’t bother testing pathological cases, start simple.
For integration testing there are a variety of choices. e.g. use selenium if you are already familiar with it.

Conclusion

Testing a Rails application using the built-in testing framework is easy and fast. Although there are lot of things to learn in testing, it is easy to start small and build gradually as you gain more knowledge and experience. Happy testing.

Web development is hard

2009-10-28T18:39:00Z

Despite a lot of progress, from what I see, developing a web application and running it successfully involve a lot of hard work. Here’s what I think are reasons for that.

On the Bangalore OCC mailing list which has a fantastic mix of entrepreneurs, consultants, VCs and like, somebody while advising to be careful while hiring web developers commented : “Web development is not… one time development…”. That got me thinking: Web development is still tricky. Yes, we do have our productive frameworks and tried agile methodologies, but there is still something which makes developing a web application inherently hard.

Never, never, never believe any war will be smooth and easy, or that anyone who embarks on the strange voyage can measure the tides and hurricanes he will encounter. —Winston Churchill

Alright, not as difficult as a war may be. But, here are some reasons why I think it is difficult:

Developing a web application needs multiple skills: Your web developer would need to know HTML/CSS, a server side language and relational databases; Javascript and some good library to add AJAX and some zing. Additionally may be, Flash/Silverlight. Depending on your application, you may need to do image manipulation or some video processing. If you are going to scale, you also better build your architecture well add later add caching, tune performance of your stack, shard your database or add a nifty nosql database. Aah! It like being a multi-handed Hindu goddess.
Of course, you will not find a single engineer who can do all this but a good web developer will know few of these well and have sufficient knowledge of others to know when to bring in more help.

It is like gardening: You cannot develop a web application and just leave it like that. It needs continuous care to change and adapt to usage patterns, server loads and optimize work flows.

Security is top priority: In a web application you are effectively putting your database online with only the application layer protecting it from the big bad internet. Good security is not an add-on, its a requirement.

You will change it: Like the war quote above says, it is very hard to predict what directions you would take once you put an application online. Because of the very short feedback loop of what you put, how your users react and then how you would adapt, web applications need to be nimble. It would be only prudent to develop it in a way so you can change it fast enough.

A real life RESTful Rails application

2009-10-06T09:11:00Z

There are a lot of tutorials on the web explaining how to use Rails to build a RESTful application but very few that goes beyond toy applications. Here, I present a slightly more complicated example and show how to build it using Rail’s RESTful architecture.

“Soon you will know, soon you will be one of us” — an ancient interwebs proverb.

Introduction

If you have spent some time learning RESTful way of developing applications with Rails, you’ve probably been through some tutorials on developing basic CRUD applications in the RESTful way.

However, a lot of people struggle with applying the RESTful concepts to anything beyond simple toy applications. So, here I am presenting an example application which shows slightly more involved use of RESTful architecture.

At this point I should point out that it is a moderately complex example and so if you are new to Rails or RESTful architecture in Rails, you should brush up on these first. Here are a few good starting points for REST:

and for how it applies to Rails:

OK, let’s start.

While using RESTful architecture in Rails, only controller and routes are different. What you already know about models and helpers and all of the other plugins and libraries you are used to, work exactly the same.

Controllers become more streamlined and there will be lot more controller classes than earlier, as you will see soon. However the code becomes much more simpler.

Requirements

Let’s start by building an example Rails application. Here are the requirements:

Our application should allow users to register and then login using their login and password. Fairly standard stuff. After authentication, they should see an inbox with messages – very much like your Gmail inbox, and a big button called “send a message”. On clicking it, any user should be able to compose and send another user a message. A message consists of subject and body. Again, quite similar to email but these messages are within the closed system i.e. only users in the system can send a message to another user in the system. Yes, quite like what social networking sites do. Additionally, a user should be able to archive a message or delete it; view archived messages and sent messages.

Is requirements part clear? Good.

Preparation

Let’s start by building user accounts and authentication:



$ rails -d mysql demo
$ cd demo 
$ ruby script/plugin install git://github.com/technoweenie/restful-authentication.git  
$ ruby script/generate authenticated user sessions
# add 'include AuthenticatedSystem' to application_controller.rb as recommended by restful-authentication plugin
# edit file: config/database.yml and add password for the database if you have one
$ rake db:create
$ rake db:migrate
$ ruby script/server

Point your browser to: http://localhost:3000/users/new and create a new account. Then go to: http://localhost:3000/sessions/new and sign in with the new user account which you just created and you should see the “Welcome aboard” page.

So far you have built a basic Rails app, with restful auth. You might want to register 2 users and keep their login and passwords handy – let’s say with login names venkat and suresh. We would next be building the messaging part so that these two users can send messages to each other.

Models and the Database design

You can reflect for a moment about what kind of DB design would suit this problem. But, just to speed us up, here’s something I thought of. Let’s have these following 3 models (and hence 3 tables in the DB) and their attributes:

MessageDetail: subject, body, sent_message_id

ReceivedMessage: message_detail_id, recipient_id, state

SentMessage: sender_id, state

The idea being that MessageDetail object represents the actual message – the subject and body; ReceivedMessage represents the view of the message as visible to the recipient; similarly SentMessage is sender’s view. ReceivedMessage and SentMessage are separate objects from MessageDetail so that senders and each of the multiple recipients can independently archive or delete their messages.

This means, even if sender decides to delete a message, we won’t delete MessageDetail record from our table because otherwise message from recipient view would be lost. We would delete the message when both sender and receiver delete their messages.

Let’s create each of these models:


$ ruby script/generate model MessageDetail subject:string body:text sent_message_id:integer
$ ruby script/generate model ReceivedMessage message_detail_id:integer recipient_id:integer state:string
$ ruby script/generate model SentMessage sender_id:integer state:string

Then run the migrations to create the DB tables:


$ rake db:migrate

Now, fill in the model code:



# model classes omitting validations

class MessageDetail < ActiveRecord::Base
  attr_accessible :subject, :body, :sender, :recipient

  belongs_to :sent_message
  has_one :sender, :through => :sent_message
  has_many :received_messages
  has_many :recipients, :through => :received_messages

  def sender=(user)
    self.build_sent_message
    self.sent_message.sender = user
  end

  def recipient=(user)
    self.received_messages << ReceivedMessage.new(:recipient => User.find_by_login(user))
  end

  def recipient
    self.received_messages.map { |r| r.recipient.try(:login) }.join(", ")
  end
end

class SentMessage < ActiveRecord::Base
  has_one :message_detail
  belongs_to :sender, :class_name => 'User'
end

class ReceivedMessage < ActiveRecord::Base
  belongs_to :message_detail
  belongs_to :recipient, :class_name => 'User'
end

You will also have to modify the User model a little bit to add this association:



class User < ActiveRecord::Base
  has_many :received_messages, :foreign_key => 'recipient_id'
  has_many :sent_messages, :foreign_key => 'sender_id'
  ...
end

Now, that is a lot of code. But, hopefully you can make some sense out of it. Even if you are not able to, don’t worry. We will mostly treat this as black box. Our main aim here is to learn about the controllers which is coming later.

To test that our models work as we expect, do this in your Rails console:



message = MessageDetail.new(:subject => 'test subject',
                            :body => 'test body',
                            :sender => User.find_by_login('suresh'),
                            :recipient => 'venkat')

message.save # this should return true

suresh = User.find_by_login('suresh')
msg = suresh.sent_messages.first # suresh's first sent message
msg.message_detail.recipients # list of all recipients of this message

Restful architecture

Now, our aim is to expose the messages system through our UI. REST is methodology of specifying an external interface (or a network API in other words) of a system. Constructing that API is also quite systematic.

In review, REST is about 3 things:

resources, which are identified by URLs
fixed vocabulary of verbs (HTTP verbs: GET, POST, PUT and DELETE)
representations of resources e.g. the type of representation is specified by HTTP header: content-type

Our first task is constructing a URL scheme by identifying the resources in the system. Think of resources as the nouns. Then each resource can be acted upon by one of the standard HTTP verbs: GET, POST, PUT or DELETE.

Let’s say we decide on following URL scheme:

/messages where GET /messages returns all messages in inbox, POST /messages sends a new message

/messages/archived similar to /messages but shows all messages in archive folder

/messages/sent similar to /messages but shows all messages in sent folder.

/messages/1 GET /messages/1 shows the message id 1

/messages/1/archived GET /messages/1/archived tells us whether this message is archived and PUT /messages/1/archived with a true in the body changes the status of the message id 1 to “archived”.

Rails actually constraints the REST even further. It generates the following set of URLs:

/messages

/messages/1

/messages/1/edit

/messages/new

Each of these URLs can be generated by a named route helper (if you don’t know named routes, see here: http://guides.rubyonrails.org/routing.html). The helper names for the above URLs are:

messages_path

message_path(:id => 1)

edit_message_path(:id => 1)

new_message_path

Then Rails maps certain combinations of HTTP verbs with these URLs to controller actions:

GET /messages index

GET /messages/1 get

POST /messages create

PUT /messages/1 update

DELETE /messages/1 delete

GET /messages/new new

GET /messages/1/edit edit

Now, creating the four named routes, and mapping the above 7 combinations of HTTP verb and URL to controller actions, can be accomplished by adding just this one line in the config/routes.rb file:


map.resources :messages

Go ahead and add this to your config/routes.rb. Then, create a controller called MessagesController with the above 7 actions, and the actions will be called whenever the user fetches the corresponding URL.


$ ruby script/generate controller messages

Filling the controller with the 7 actions is very much boiler plate code. But, unfortunately Rails does not provide a generator for that (yet). If you use Textmate or Emacs, you can use a snippet (available here).

You can also generate the controllers using scaffold or resource generator.

However, let’s do it by hand:


$ ruby script/generate controller messages index show new edit create update destroy

As an aside, if you are confused about difference between new and create: when a user wants to create a new message, she GETs the /messages/new URL which is rendered by new action, fills in the form, and then POSTs to /messages to create the object in the database. The POST /messages request is handled by create action. edit and update actions work similarly.

Now, all that is remaining is to code up the 7 actions in the controller.

There are rules about what can the web application do for each of these actions. Most importantly, REST specifies that GET, PUT, DELETE are idempotent. Which means they can be repeated, say by user hitting reload on their browser; where as POST is not. So, all actions except create must be idempotent, that is even if they are repeated, there should be no harm.

Now, there are a few things to take care of while coding the controller. Firstly, put a filter so that the actions can be executed only after authentication:



class MessagesController < ApplicationController
  before_filter :login_required
  ...
  ...
end

Now, we have to skillfully map our restful controller’s actions to calls to the models we already have built. So, for example, code of index action would be:



# GET /messages
def index
  @messages = current_user.received_messages
end

Now, create a view messages/index.html.erb and display the messages from @messages



<table>
<thead>
  <th>Sent by</th>
  <th>Subject</th>
  <th>Date</th>
</thead>
<% @messages.each do |message| %>
  <tr>
    <td><%= message.message_detail.sender.login %></td>
    <td><%= message.message_detail.subject %></td>
    <td><%= message.created_at %></td>
  </tr>
<% end %>
</table>

Go to http://localhost:3000/messages to look at the inbox for the logged in user.

Now, it is quite straight forward to code up all the controller actions and corresponding views. Let’s try create next:



  # file: app/views/messages/new.html.erb
  <h1>Enter your message</h1>

  <% form_for(:message, :url => messages_path) do |f| %>

  <p>From <strong><%= current_user.login %></strong></p>
  <p>
      <%= f.label :recipient, "login-id of the recipient:" %><br />
      <%= f.text_field :recipient %>
  </p>
  <p>
      <%= f.label :subject, "Subject:" %><br />
      <%= f.text_field :subject %>
  </p>

  <p><%= f.text_area :body, :cols => 60, :rows => 8 %></p>

  <p><%= f.submit "Post Message" %></p>

  <% end %>

  # file: app/controllers/messages_controller.rb
  # GET /messages/new
  def new
    @message = MessageDetail.new
  end

  # POST /messages
  def create
    @message = MessageDetail.new(params[:message].merge(:sender => current_user))

    if @message.save
      flash[:success] = 'Message was successfully sent.'
      redirect_to messages_path
    else
      render :action => "new" 
    end
  end

Now, go to http://localhost:3000/messages/new and send out a messages to the other user in your database.
Go to http://localhost:3000/session/new and login as other user
Go to http://localhost:3000/messages and see the new message in the inbox

There, you have a message system whose external interface is completely RESTful.

Recap:

We built a skeleton Rails app with restful_authentication
Then we added messages models and tested from console that we can send a message from one user to other
Lastly we exposed the messages through a RESTful interface

Conclusion:

Rails supports RESTful architecture out of the box. There is no need for any plugin to make it work.

REST learning resources:

REST wiki
Sam Ruby’s blog

Rails:

Webservices to make application development easy

2009-09-27T09:41:00Z

Recently Web Services have been a major driving force towards making application development easy, modular and more robust. Here’s a list of some popular and proven ones for Rails development.

Last week there was an article on six revisions about Website Features That You Can Easily Offload listing how you can take advantage of web services to lesser the burden on a site.

That prompted me to make a list of web services that an application developer, especially a Rails developer can use to similarly, offload functionality and get away with less code, less maintenance and develop quickly. Here’s what I could think of:

Authentication – Even though creating an authentication system in Rails is easy thanks to plugins like Resftful authentication, your users probably hate having to create and remember another login and password. By using OpenId you can allow them to sign using a login and password they already remember, not worry about handling forgotten passwords and actually be more secure.

Client side analytics – Google analytics has made client side analytics dead easy. But there are others too: Clicky, Crazy Egg and Mint.

Server side metrics – If you want to collect metrics within your application, there are web services that help you there as well. See Mixpanel and Nuconomy.

Server monitoring and alarms – If you don’t want to build a dashboard yourself using Nagios or Munin, one of the best web services product to monitor your server performance is Scout

Application performance monitoring – Among the hosted services, NewRelic seems to the current leader for Rails application performance monitoring. Scout would also do it for you in a limited sense. this also looks like a promising upcoming alternative.

Uptime monitring – To know if your site is up and get notified as soon as it goes down is absolute key to any business. Keynote, Pingdom are two good choices. Of course, there are many others as well.

Exception reporting – probably unique to Rails, is getting notified whenever a user gets the dreaded 500 server error page. If you want more than what Exception notifier plugin can do for you, Hoptoad and Exceptional are two services which you can look at.

Sending emails – If sending emails from your server seems like too complicated, or you are seeing your mails end up in people’s spam folders, you can try Google’s SMTP. For a more full featured alternative, try Campaign monitor or Mailchimp.

Videos – With excellent services like Youtube, Vimeo and blip.tv You no longer really need to handle video upload, processing and serving. In fact, using one of these services will be faster for both development as well as for your users.

Images – Similarly, with services like Flickr you don’t need to handle photos if you don’t want to. You can even offload profile pictures to Gravatar

Payment – Handling payment is one of the trickiest pieces. Almost all the apps end up using one of these services to make it easy. Take your pick from: Paypal, Google checkout and Amazon payments. Then there’s Spreedly especially to make handling subscriptions easier.

E-commerce – If you want a service for handling checkouts, order management dashboards, fulfillment and the whole nine yards, check out NetSuite and Shopify

Geoocoding and maps – You are probably already using Google maps as a service to embed maps in your application. If you need to do geocoding, try this

DB backup and storage – There are umpteen choices for offsite stroage. Couple of good ones are Amazon S3 and Jungle Disk but there are many others too.

Amazon Web services: I thought AWS deserved special mention since they really have been pioneers of web services. You can find many interesting services here from distributed queue, hosted databases, metering and even on-demand work force. Disclaimer: I used to work for AWS.

Besides, there are Rails plugins of course which make developing applications easier ever further. But, since we are considering only web services, they are not listed here.

Beautiful code

2009-09-23T21:18:00Z

Joel Splosky once defined beautiful code as: code which does something in much less lines than what it seems like the task should take. I think these examples fit the definition exactly.

A long time back, browsing through a book, I came across code for strcpy function which is part of the standard library in C:


strcpy(char *s1,char *s2)
{
  while (*s1++ = *s2++)
}

Just the beauty of the code made me fall in love with C and programming all over again. Then, I got busy with courses, jobs, projects and releases.

Recently, while programming with Ruby, I had to randomize an array of objects before it got displayed. I thought, I should consult Google’s brain before I whip up anything of my own. This is what I found:


(1..10).sort_by { rand }

Randomizes an array of numbers 1 to 10. Oh the beauty!

More details here if you want to learn about strcpy and here is where I spotted the Ruby code.

Web 2.0 application development with RoR - slides

2009-08-15T16:33:00Z

Slides from the recent training I delivered at Hewlett Packard India.

Web 2.0 Application development with Ruby on Rails<object height="355" width="425"><param /><param /><param /><embed src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=slides-090815112157-phpapp02&rel=0&stripped_title=web-20-application-development-with-ruby-on-rails" height="355" width="425"></embed></object>

View more documents from akmags.

Before you write that code...

2009-07-22T16:18:00Z

There are several important things for a business owner to do before letting engineers start developing code. Here I give a comprehensive list of such tasks. It will make your project more controllable and eventually will make your developers happy as well.

So you have decided that you are going to implement your big idea and may be you have also hired your favorite developer. There is nothing that a programmer loves more than writing code – they would just want to get hacking. But in my experience, there are some essential things you should do before you get to writing code.

Prepare a page flow diagram: You should have a clear enough idea of all the important pages and how they link together. You should be able to walk through the steps a typical user would go through while using your application.
There’s a whole field devoted to dividing the user flows into pages and laying out information on those pages. But, you don’t have to get fancy. As the 37Signals guys recommend in their fantastic book Getting Real, just a rough sketch on paper is enough.

Ideally you should also get the graphic design done based on this page flow diagram. But, at least get the page flow diagram prepared before you write any code.

For a lot of projects, the page flow diagram serves as good as a requirements doc; and it is a lot less work. You certainly want to make sure you got your vision into your dev’s brain. It is a good way to do it.

Make a high level task list: Based on the page flow diagram, sit down with your engineer and make a list of tasks. My favourite tool for capturing task level breakdown is Pivotal Tracker. But, you could easily use anything else or even paper.
Try, Redmine, Unfuddle or Mingle

All of these tools can also serve as the bug database once the project gets going.

Get estimates: You should try to get an idea of the effort estimate, in terms of days or weeks, for all the tasks you have listed. It will be rough and would most likely prove to be inaccurate. But the actual numbers are not important. Rather, it will help you see the relative cost of different features.
In any case, it will help you prioritize tasks. You already knew what was important and not so important for your business. Now you know how much they roughly cost. Do it. Quite often, it is surprising.

Get detailed estimates: Just before you are about to start implementation, ask for detailed estimates for at least the first few tasks. You should try to fill at least the next two weeks with work.

Ask for a status report: Most likely, whatever estimates you have got at the beginning is going to change as project progresses. Also, you are going to change your mind about features that you want implemented. A weekly status report is a good way to keep track of these things. Make sure to include the following in the reports:
1. List of tasks completed last week with amount of time spent on each task
2. Tasks which will be worked on in the next week
3. Important discussions that happened and decisions taken
Of course, these reports will help you settle invoices later.

Set up a regular call schedule: This is very much dependent on the momentum of the project and what suits the team. The underlying point is take regular stock of how things are progressing.

Arrange for a staging server: While your app is under development, you should arrange for it to be deployed some place where you can review it. It could be some shared hosting or a cheap VPS plan.
Make sure to protect the staging server with an htpasswd to prevent casual visitors from any unauthorized peek and stop crawers and search engines. Also, if you are developing a data driven app where you are storing seed data in the staging server while you are developing the application, make sure it is backed up.

Arrange for source code repository: The code that your engineers are developing for you are really the only “thing” you are producing. It is worth protecting it safely. Make sure you arrange for a code repository (Subversion and Git are popular these days) and make sure it is safely backed up. Good ones are: Github, Unfuddle and Assembla). It also serves as an important development tool – most developers worth their salt won’t write any code without it.

Final words

If you are interested in being more thorough, you can start by reading the Joel test and of course no discussion of software development can be complete without a reference to The Mythical Man Month and The Code Complete

Although, this is a discussion on things to do at the beginning of a project, I should mention that you should make sure to arrange for QA soon after you start.

Of course, this is just the beginning and similarly there are other important things for the in-progress and the at end of a project to take care of as a business owner. On that, sometime later.

DB Backups

2009-07-07T19:18:00Z

Having a robust database backup scheme is essential for any production application. Here’s a brief summary of popular schemes used with Rails applications and a description of my own favorite.

Every application in production needs some sort of database backup scheme. There are plenty of methodologies and several Rails scripts or plugins to do just that. Here’s a brief summary of some of the popular ones:

ar-backup is a Rails plugin which saves data as fixtures. The data of course becomes database independent. However, to store it safely you will have to push them to your SCM or manage backups separately.
A comprehensive backup system of DB data, code and SCM to Amazon S3 is on Ruby Inside
here is a simple rake task to take a mysqldump and push it to S3. There is no scheme for rolling backups, but if you want a simple solution, this may be enough.
For a large database where full backups may be expensive, see this. There’s also a Rails plugin using this, however at the time of writing it seems incomplete.
The awesome guys at Collective Idea have a polished Rails plugin: for backing up the database as fixtures. It also has some nifty features like ability to pull data back into development server. Read about it here
Finally, a novel approach of backing up the database in a Git repository which seems will work for databases which are not growing very rapidly.

However, surprisingly, none of them fitted exactly what I needed and so I decided to roll my own. Also, it is quite straightforward as you will see. This is what I wanted from a DB backup system:

Take periodic dumps of the MySQL database and push them to S3 for safe storage. However, keep the old backups so that if the database gets corrupted or I want to look at the history, I have old ones available. But, I don’t want all the old ones (to cut down on storage charges), perhaps just last couple of weeks’ ones.

Courtesy of this solution, which provides rolling backups but saves to disk and some help from Amazon, I wrote this rake task. If you want to use it, copy this in lib/tasks/db_backup.rake of your Rails project


require 'find'
require 'ftools'
require 'aws/s3'

namespace :db do
  desc "Backup the database to a file. Options: RAILS_ENV=production MAX=28" 
  task :backup => [:environment] do
    AWS::S3::Base.establish_connection!(:access_key_id => 'your-S3-access-key', :secret_access_key => 'your-S3-secret-access-key')
    BUCKET = 'A-S3-bucket-name-you-created'

    datestamp = Time.now.strftime("%Y-%m-%d_%H-%M-%S")
    base_path = ENV["RAILS_ROOT"] || "." 
    file_name = "#{RAILS_ENV}_dump-#{datestamp}.sql.gz" 
    backup_file = File.join(base_path, "tmp", file_name)
    db_config = ActiveRecord::Base.configurations[RAILS_ENV]
    sh "mysqldump -u #{db_config['username']} -p#{db_config['password']} -Q --add-drop-table -O add-locks=FALSE -O lock-tables=FALSE #{db_config['database']} | gzip -c > #{backup_file}" 
    AWS::S3::S3Object.store(file_name, open(backup_file), BUCKET)
    puts "Created backup: #{file_name}" 
    FileUtils.rm_rf(backup_file)

    bucket = AWS::S3::Bucket.find(BUCKET)
    all_backups = bucket.objects.select { |f| f.key.match(/dump/) }.sort { |a,b| a.key <=> b.key }.reverse
    max_backups = ENV["MAX"].to_i || 28
    unwanted_backups = all_backups[max_backups..-1] || []
    for unwanted_backup in unwanted_backups
      unwanted_backup.delete
      puts "deleted #{unwanted_backup.key}" 
    end
    puts "Deleted #{unwanted_backups.length} backups, #{all_backups.length - unwanted_backups.length} backups available" 
  end
end

Then set up a cron job on your production host to call this:


rake db:backup RAILS_ENV=production MAX=8 # take a backup of production DB, push the file to S3, keep past 8 copies

You can use S3Fox to keep a watch on backup files as they get pushed to S3. If you ever need to import back the database into local desktop – use S3Fox to download the file and do this on your development machine:


gzip -dc production_dump-YYYY-MM-DD_HH-MM-SS.sql.gz | mysql -u root -p <development database>

Rails migrations revisited

2009-03-23T20:34:00Z

Rails Database migrations is a great piece of technology and deservedly popular. However, there are aspects of it which are not quite well understood. Here I discuss couple of such points: how to manage the schema.rb file and how to handle data migrations.

No, we are not back to 2007 again. There are just a few things about migrations that I think some people have still not got right. So, I thought I will highlight them here.

Rails have had migrations for quite a few releases now. Here’s a brief review:

Recap

When you want to do a change in your database schema, like add a column or change a table definition, you generate a migration, populate the code for forward and backward migrations, and run rake db:migrate. It let’s you change your DB incrementally. Everybody gets this part I think. See Rails guides for a wonderful overview.

schema.rb file

Running a migration updates the db/schema.rb file. It is required to put this in your source control and check it in with the migration. A lot of people actively ignore this file (by using svn:ignore or .gitignore). Don’t.

Migrations are useful when you are upgrading your database. Where as if you are setting up a new instance of the database (say a new user joins your team or you are setting up a new staging server), you should not be running the migrations. Instead do this:


  $ rake db:create
  $ rake db:schema:load

Running a long winded migrations, in most non trivial projects, is almost bound to fail (because of name changes in your source code, since you are not going back and updating your migrations). This is official Rails policy (see the comment on top of your db/schema.rb or DHH’s code commit here)

See here for a summary of the opposite view.

Data migrations

Data migration is porting the data in the tables to the new schema. Unfortunately, Rails does not have a clearly specified way of dealing with this. Usually developers do not treat this as a separate issue but handle it as part of usual Rails migrations.

There are two cases when you may find yourself dealing with data in migrations:

while seeding the database with some data, or
porting the data in older tables to newer schema.

Most of the times it is possible to handle both using migrations. However, for the second case, you may sometimes need to create special migration scripts.

However, if you are not running migrations to start a new database, you still need a way to seed the database with the required data. Here is something I have used which has worked well for several projects which I picked up from here

In addition to creating a migration, put seed data in db/fixtures/*.yml
load the seed data from db/fixtures when initializing a new database



$ rake db:create
$ rake db:schema:load
$ rake db:seed # see link above for Rake task definition for this

conclusion

Most projects ignore data migrations during development, which is not entirely wrong. Who cares for lost data during development? But, you will need a robust strategy for handling data migrations once you are on production and you should prepare yourself with a methodology well in advance. And a last final tip, backup your production database before running any migrations.

Background job processing in Rails with delayed_job

2009-02-28T20:08:00Z

Running jobs in background is part of every non-trivial web application – for either speeding up user response, or running periodically scheduled tasks. Here, I explain how to install, configure and use the elegant delayed_job plugin to process background jobs.

Why background processing is needed

Since Ruby 1.8 only supports green threads, if you want to process something longish as part of a web request in Rails, it will delay the response. Of course, if the processing is not critical to returning a response to the client, it’s wiser to push the processing to a background queue and return a response immediately.

There are other incidental benefits to using a background job processing queue like, letting a dedicated server, other than the customer facing app server, handle it, assigning priority to jobs, retry on failure etc.

Most common situations where such a background processing is needed are: sending emails to users, rebuilding search indices, image and video processing, etc.

Available options

There are several ways to process background jobs in Rails. In fact, So many that Geoffrey Grosenbach joked on the RailsEnvy podcast about the number of continuous integration servers finally being equal to number of queuing servers available in Ruby.

My favorite one so far is: delayed_job. Because its simple to set up, simple to use and has lot of powerful features. However, don’t forget to look at others to see what fits you needs and style better. For an overview see here. In the rest of the article, I will explain how to install and use delayed_job.

Installation

Firstly, install it in your rails project just as you would any other plugin:


$ ruby script/plugin install git://github.com/tobi/delayed_job.git

Delayed_job stores the jobs in a table in the database. Create the table using a migration:


$ ruby script/generate migration create_table_for_delayed_job

class CreateTableForDelayedJob < ActiveRecord::Migration
  def self.up
    create_table :delayed_jobs, :force => true do |t|
      t.integer  :priority, :default => 0
      t.integer  :attempts, :default => 0
      t.text     :handler
      t.text     :last_error
      t.datetime :run_at
      t.datetime :locked_at
      t.datetime :failed_at
      t.string   :locked_by
      t.timestamps
    end
  end

  def self.down
    drop_table :delayed_jobs
  end
end

$ rake db:migrate

Note: Compared to the documentation, I have changed last_error column to text from string to handle larger error messages

Delay jobs

Restart your server, and you can start delaying jobs. There are two ways to push a job to background. First, the easy way:

You have seen Ruby’s send method, right.


"Bangalore".length # returns 9
"Bangalore".send(:length) # same as above, returns 9

send just invokes the given method on the object.

Analogous to that, delayed_job adds send_later method. Say, if you were sending a mail to user on registration:


NotificationMailer.deliver_welcome_user(@user)

Just change that to:


NotificationMailer.send_later(:deliver_welcome_user, @user)

And you are done. delayed_job will push this into the queue (the database table you created above), to be executed later.

There is another way to background a job. First, create a job class.


# put this in lib/delayed_notification.rb
class DelayedNotification
  attr_accessor :user_id
  def initialize(user_id)
    self.user_id = user_id
  end

  def perform
   NotificationMailer.deliver_welcome_user(User.find(@user_id))
  end
end

# Add this where you were sending the mail earlier
Delayed::Job.enqueue DelayedNotification.new(@user.id)

Executing the jobs

Now, there are several ways to run the job which are there in the queue. In development mode, you can just issue the following command in your terminal:


$ rake jobs:work

This will cause the worker to run in a loop. Stop it by pressing Control-C.

On production, you would want the worker to be running all the time. Also, it will be nice to have the ability to stop the worker just before the deployment and start it again once the deployment finishes. It sounds complicated, but its pretty easy to set up.

First, install the daemons gem:


# Add in environment.rb
 config.gem 'daemons'

$ rake gems:install

Then, copy this script in your rails project in file script/delayed_job and give it execute permission. This creates a worker daemon, which when started keeps running as a background process.

Alternately, instead of daemons gem, you can use daemon-spawn gem.

Install the gem on the host where you want the delayed_job daemon to run:


$ sudo gem sources -a http://gems.github.com
$ sudo gem install alexvollmer-daemon-spawn

And then copy this script instead in script/delayed_job. Rest all of the following steps are identical, whichever gem and script/delayed_job you choose.


# start the worker daemon
$ ruby script/delayed_job start

# stop it
$ ruby script/delayed_job stop

You would of course want to start and stop it on your production server using Capistrano. Here’s a recipe to do that (courtesy collectiveidea)


# add this to config/deploy.rb
namespace :delayed_job do
  desc "Start delayed_job process" 
  task :start, :roles => :app do
    run "cd #{current_path}; script/delayed_job start #{rails_env}" 
  end

  desc "Stop delayed_job process" 
  task :stop, :roles => :app do
    run "cd #{current_path}; script/delayed_job stop #{rails_env}" 
  end

  desc "Restart delayed_job process" 
  task :restart, :roles => :app do
    run "cd #{current_path}; script/delayed_job restart #{rails_env}" 
  end
end

after "deploy:start", "delayed_job:start" 
after "deploy:stop", "delayed_job:stop" 
after "deploy:restart", "delayed_job:restart"

And you are done. Just deploy as usual.


$ cap deploy:migrations

Further tips and tricks

Get a separate server to process your delayed job so as not to take CPU away from customer facing app server.

You can customize some parameters like how many times to retry on failure and prune runaway tasks. See here.

Delayed_job allows you to assign priority to jobs.

In development mode, if you just want to discard pending jobs instead of running them:


$ rake jobs:clear

If you want to control the time at which the job gets executed, you can specify it as an argument to enqueue while delaying the job:


Delayed::Job.enqueue DelayedNotification.new(@user.id), 0, 15.minutes.from_now

Here, 0 denotes the (default) priority of the task. This task will get executed at least 15 minutes later than it would have been executed normally.

References

For more about this wonderful plugin, refer to these resources:

A nice tutorial covering the basics.
Home of delayed_job (also look at the wiki).

Why I am thankful for all the maintenance programming I did

2009-02-17T19:09:00Z

Although maintenance programming is no one’s dream job, they can help you learn a lot of practical lessons and mature as a developer. I have done my fair share and here’s how I think it helped me grow.

Recently, an article bubbled up in delicious popular that caught my eye: Why I Don’t Mind Maintenance Programming

As the author notes, almost every developer wants to work on a new project. So much so that, it is invariably brought up in job interviews either by companies to sell their position or by candidates trying to decide whether to take up an offer. I was no different when I graduated from college and was duly disappointed when couple of months into my first job I realized I would be doing maintenance of several legacy systems. Only later did I realize the whole lot of good that it did to me:

I got to see what really good code, written by experienced developers, looked like. Common idioms and styles which make a code base maintainable for long (there was one product which was over 12 years old and still pretty clean).

I got to see what really bad code looked like. Code most likely written by rookies or people in hurry to get their little feature done. I probably cannot emphasize enough how illuminating it is to actually look at bad code

Although it is easy to dismiss something as “bad code”, it is much harder to realize that it is pragmatism which is most important. Ultimately, we are building a product to be used by customers. Having clean code is important only as much as it helps you do other future changes quickly enough. Doing maintenance of other people’s code makes you keenly aware of what real life code looks like. You will never cringe at some small stylistic gotchas in code, but rather focus on the bigger picture of getting it correct and efficient.

One other important thing that it taught me was importance of writing tests. When you are doing maintenance, you live or die by your test suite. Also, only later I realized that the nifty trick of writing tests first before doing the bug fix or a feature that I was using had a name for it already (TDD).

Of course, while doing maintenance, you get to see the difference you are making to the life of a customer, almost immediately.

I think doing maintenance programming was a significant factor to help me grow as a developer and shaped my judgment ever since. I strongly recommend every beginner developer to spend at least a year or two doing maintenance. With the spread of open source, it has become all the much easier to look at other people’s code.

What's a computer?

2009-02-14T18:13:00Z

Back in the days, how one of my teachers defined a “Computer” for me and it has stuck with me since then.

My wife once told me about her first lecture in Computer Science back in her college days. The teacher walked in and started talking about history of computers, data, files, computer languages and “lots of other hyperbole”. She didn’t understand anything out of the one hour lecture. Soon after that, she switched to Political Science.

I have heard similar stories of people getting a rough introduction to computers. That got me thinking about how I got introduced to computers. I remember in my very first class (I was 12 then), my teacher Mr. Jhulan Chakraborty started by explaining what a computer is. What he said, stayed with me ever since:

A computer is a machine – just like a ceiling fan, a sewing machine, or an engine in a car. But, it’s a different kind of machine because of two important properties:

You can provide a step by step instructions to it and it will follow the instructions exactly to the letter – without any error. Just as you provide instructions to your housemaid for a task. However, unlike a human—
A computer while following the instructions, can do repetitive tasks, without getting bored, without any error, and with substantial speed.

That’s it. That is all a computer is.

I think, that is a brilliant introduction. It is concise and accurate. It also serves as a baseline for introduction of more advanced topics. For example, programming languages are just a language for specifying the instructions to a computer (see step 1 above).

File Formats

2008-07-13T19:20:00Z

File formats are crucial to how digital data is saved and their longevity. This is an introductory article for some not very familiar with what they are.

What is a file format

File Format is a way of putting your software application's data into a file on the disk when you do a save.

For example, if you create a document using a spreadsheet application, when you save the document, the file format should have ways to store how the data is organised in rows and columns, where you have put colours etc. so that you can later re-open the saved document in the application and it should look exactly the same as you created it. File formats are ways of encoding the information in the document, so that they can be stored in files on disks.

Why are they needed

As stated above, they are needed so that you can save your application to disk. The disk files are really just one long string of bytes - like a very long sentence. So, file formats specify how how the information in a document is converted to a string of bytes.

There is another reason for file formats: they allow documents created in one application to be opened using another application when the two applications understand a common file format. For example, you can create a drawing using the drawing application, save it, then open it in your word processor.

OK, what's the controversy

Well, my goal is to just make you aware of idea of file formats and not go into any controversy. However, you should be aware of the controversy. So, here it is.

There are two types of formats: open and closed. Open formats are those that have a written specification for the format which is available to everybody. Anybody can write an application to read or write a file in that format. Closed ones are those whose specifications are not public. There are some formats which fall some where in between - their specifications not public but available from the creators for a fee and some legal formalities.

But, why should I care

As mentioned above, file formats are vital for data exchange between applications. For example, your digital camera stores an image in TIFF format but you can later convert it to JPG to put it on your blog. The camera need not be programmed with producing every possible image format because it produces the image in an open format. Anyone can write an application to read the camera's captured image in TIFF and convert it to a different format. It also helps with future-proofing. Since your camera produces images in an open format, there is a good chance that it will be supported by new image editing softwares even when your camera vendor stops supporting your camera, since anybody can read your image files because its in an open format.

File formats are used extensively in EDA (Electronic Design Automation), which is where I became aware of them. EDA vendors produce software to create electronic designs. Electronic design work happens in steps: first comes the schematic capture, followed by a simulation and then synthesis and then placement and routing. The software to do each of these steps can come from a different vendor and hence the tools need to talk to each other. However, when a software is being written, the developer does not clearly know what exact software will come before and after it. So, the industry has agreed to use file formats, each software tool is build with ability to import a standard file format and export data in that standard file format. Problem Solved.

Postscript: One of the popular EDA file format is LEF & DEF created by Cadence Design System and later open sourced. I worked on the Cadence's original LEF & DEF reader and writer which was also the first project I worked on as a software professional.