Rails makes doing the right things easy and the wrong things a bit more difficult. One of those right things is testing, and the Rails 1.0 release candidate has a new set of defaults and a couple new goodies to help your tests go faster. And when tests go faster, they tend to get run more often.

Herewith, an overview of the new testing stuff, a how-to for upgrading your existing tests (though you don't have to), an obligatory math formula or two, and a performance comparison that's worth about as much as you paid to read this.

What's New?

A quick peek inside the test/test_helper.rb file that's generated for all new Rails apps reveals shiny new defaults:

ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
require 'test_help'

class Test::Unit::TestCase
  # Turn off transactional fixtures if you're working with MyISAM 
  # tables in MySQL
  self.use_transactional_fixtures = true

  # Instantiated fixtures are slow, but give you @david where you 
  # otherwise would need people(:david)
  self.use_instantiated_fixtures  = false

  # Add more helper methods to be used by all tests here...
end

Here we see the Test class being opened and two defaults being set.

self.use_transactional_fixtures = true
self.use_instantiated_fixtures  = false

In previous releases, transactional fixtures were turned off by default. Going forward, they're turned on by default. Conversely, instantiated fixtures were turned on by default in previous releases. Now they're turned off for all new Rails apps. In other words, the defaults for the two primary Rails testing options have been flipped.

All of your existing unit and functional tests are potentially affected by this change because they ultimately extend the Test class. And if your existing tests rely on the old defaults, as mine did, then upgrading your Rails app may break your existing tests. That said, you don't have to upgrade your tests and simply running gem update rails won't break your app. More on that later.

So why the complete reversal of defaults? One word: performance.

The Old Way

To appreciate what's new, let's recap how the old defaults worked.

Consider the following unit test:

require File.dirname(FILE) + '/../test_helper'


class CartTest < Test::Unit::TestCase

  fixtures :products

  def setup

@cart = Cart.new


  end

  def test_add_one_product

@cart.add_product @version_control_book
assert_equal 1, @cart.items.size


  end

  def test_add_duplicate_products

@cart.add_product @version_control_book
@cart.add_product @version_control_book
@cart.add_product @automation_book
assert_equal 2, @cart.items.size


  end

end

The tests use the @version_control_book and @automation_book instance variables. These variables are instantiated automatically when the products fixture is loaded. The test/fixtures/products.yml fixture data file describes three products, as follows:

version_control_book:
  id:              1
  title:           Pragmatic Version Control
  description:     How to use version control
  image_url:       http://.../sk_svn_small.jpg
  price:           29.95


automation_book:
  id:              2
  title:           Pragmatic Project Automation
  description:     How to automate your project
  image_url:       http://.../sk_auto_small.jpg
  price:           29.95

unit_testing:
  id:              3
  title:           Pragmatic Unit Testing
  description:     How to write better code
  image_url:       http://.../sk_ut_small.jpg
  price:           29.95

Now let's see what happens behind the scenes when we run the test. (I had to make a tiny hack to Rails to get the fixture-specific output.)

SQL (0.000296)   BEGIN
Fixture Delete (0.000778)  DELETE FROM products
Fixture Insert (0.000828)  INSERT INTO products (...) # first product
Fixture Insert (0.043348)  INSERT INTO products (...) # second product
Fixture Insert (0.004889)  INSERT INTO products (...) # third product
SQL (0.000923)   COMMIT
Product Load (0.000992)  SELECT * FROM products WHERE (products.id = 3) LIMIT 1
Product Load (0.000884)  SELECT * FROM products WHERE (products.id = 2) LIMIT 1
Product Load (0.000949)  SELECT * FROM products WHERE (products.id = 1) LIMIT 1


setup runs

test_add_one_product runs

SQL (0.000263)   BEGIN
Fixture Delete (0.001630)  DELETE FROM products
Fixture Insert (0.000754)  INSERT INTO products (...) # first product
Fixture Insert (0.000863)  INSERT INTO products (...) # second product
Fixture Insert (0.000992)  INSERT INTO products (...) # third product
SQL (0.002623)   COMMIT
Product Load (0.000887)  SELECT * FROM products WHERE (products.id = 3) LIMIT 1
Product Load (0.000890)  SELECT * FROM products WHERE (products.id = 2) LIMIT 1
Product Load (0.000830)  SELECT * FROM products WHERE (products.id = 1) LIMIT 1

setup runs

test_add_duplicate_products runs

Interesting. There's a lot going on as a result of including the following line in the test case:

fixtures :products

Consequently, Rails automatically does three things before each test method:

1. Deletes all the test data in the <tt>products</tt> table of the
   test database. 
   (<tt>DELETE FROM products</tt>)  
   

2. Inserts a row into the <tt>products</tt> table of the test
   database for each product listed in the fixture data file.  
   (<tt>INSERT INTO products</tt>)
   

3. Finds the instance corresponding to each product in the test
   database and assigns it to a instance variable of the same name.
   (<tt>SELECT * FROM products</tt>)
   

This is good because it means that each test method is isolated from database changes made by other test methods. That is, the fixtures are restored to their original state in the test database before each test method runs. What's the cost of this isolation? Let's do the math.

minimum # of SQL calls per test case = T * (F * (1 DELETE + R INSERTS + R SELECTS))


where T = # of test methods

  F = # of declared fixture files
  R = # of records specified in each fixture

So, for the example test case above, the math works out as:

2 * (1 * (1 DELETE + 3 INSERTS + 3 SELECTS)) = 14 SQL calls

That's not a huge number, but the example test case is a pip-squeak. Any respectable test case starts to rack up SQL calls faster than you can scream "In-memory database!". And pretty soon you aren't running the tests any more because you feel like you don't have time to test.

It doesn't have to be that way. Let's chip away some SQL calls before the tests get too slow.

Transactional Fixtures

Transactional fixtures use database transactions to isolate tests. Rather than deleting and re-inserting fixtures for each test method, transactional fixtures are loaded once at the beginning of the test case. The fixture data in the test database is restored to its original state after each test by doing a transaction rollback.

Let's re-run the test case, this time with transactional fixtures enabled, per the new defaults:

self.use_transactional_fixtures = true

The test log is noticeably shorter:

SQL (0.000493)   BEGIN
Fixture Delete (0.000805)  DELETE FROM products
Fixture Insert (0.001194)  INSERT INTO products (...) # first product
Fixture Insert (0.000824)  INSERT INTO products (...) # second product
Fixture Insert (0.000793)  INSERT INTO products (...) # third product
SQL (0.000982)   COMMIT


SQL (0.000208)   BEGIN
Product Load (0.002081)  SELECT * FROM products WHERE (products.id = 3) LIMIT 1
Product Load (0.007071)  SELECT * FROM products WHERE (products.id = 2) LIMIT 1
Product Load (0.001213)  SELECT * FROM products WHERE (products.id = 1) LIMIT 1

setup runs

test_add_one_product runs

SQL (0.015225)   ROLLBACK

SQL (0.000236)   BEGIN
Product Load (0.000963)  SELECT * FROM products WHERE (products.id = 3) LIMIT 1
Product Load (0.000793)  SELECT * FROM products WHERE (products.id = 2) LIMIT 1
Product Load (0.000787)  SELECT * FROM products WHERE (products.id = 1) LIMIT 1

setup runs

test_add_duplicate_products runs

SQL (0.002332)   ROLLBACK

This time the test data in the products table is deleted and re-inserted from the fixture file exactly once. A transaction is started before each test method (BEGIN), then rolled back at the end (ROLLBACK).

Here's the math for transactional fixtures:

minimum # of SQL calls per test case = (F * (1 DELETE + R INSERTS)) + (T * R SELECTS)


where T = # of test methods

  F = # of declared fixture files
  R = # of records specified in each fixture

So we've spared ourselves the trouble of 4 extra database hits.

(1 * (1 DELETE + 3 INSERTS)) + (2 * 3 SELECTS) = 10 SQL calls

To take advantage of transactional fixtures, your database must support transactions. I realize this seems obvious, but it tripped me up because MySQL uses the MyISAM database type by default, as far as I can tell. And my favorite MySQL database demo tool (YourSQL) follows suit. Unfortunately, MyISAM doesn't support transactions. So if you're using MySQL, then you'll need to make sure your tables use the InnoDB table format. Here's an example of how to convert a table from MyISAM to InnoDB:

alter table products type=InnoDB;

The only drawback to using transactional fixtures is when you actually need to test transactions. Since your test is bracketed by a transaction, any transactions started in your code will be automatically rolled back.

On-Demand Instantiated Fixtures

By default in Rails 1.0, fixtures aren't instantiated and assigned to instance variables until you need them. Whereas previous releases would automatically create a @version_control_book instance variable, for example, before each test method, you can now use the fixture accessor method products(:version_control_book) to load fixture data into variables on demand.

So before running the test with instantiated fixtures disabled, we need to change the tests to use fixture accessor methods:

def test_add_one_product
  @cart.add_product products(:version_control_book)
  assert_equal 1, @cart.items.size
end


def test_add_duplicate_products
  @cart.add_product products(:version_control_book)
  @cart.add_product products(:version_control_book)
  @cart.add_product products(:automation_book)
  assert_equal 2, @cart.items.size
end

Calls to products(:version_control_book), for example, are cached within a specific test method. That is, the first time you call it in a test method, a SQL query loads the corresponding model. The second time you call it within the same test method, it returns a cached result. Calling products(:version_control_book, :refresh) will force a reload.

OK, now let's run the test again, this time using both new defaults in Rails 1.0—transactional fixtures enabled and instantiated fixtures disabled:

self.use_transactional_fixtures = true
self.use_instantiated_fixtures  = false

The test log is even shorter:

SQL (0.000296)   BEGIN
Fixture Delete (0.001028)  DELETE FROM products
Fixture Insert (0.001111)  INSERT INTO products (...) # first product
Fixture Insert (0.003815)  INSERT INTO products (...) # second product
Fixture Insert (0.000737)  INSERT INTO products (...) # third product
SQL (0.000864)   COMMIT


SQL (0.000227)   BEGIN

setup runs

test_add_one_product begins

Product Load (0.000807)   SELECT * FROM products WHERE (products.id = 1) LIMIT 1

test_add_one_product ends

SQL (0.001663)   ROLLBACK

SQL (0.000209)   BEGIN

setup runs

test_add_duplicate_products begins

Product Load (0.000850)   SELECT * FROM products WHERE (products.id = 1) LIMIT 1
Product Load (0.001064)   SELECT * FROM products WHERE (products.id = 2) LIMIT 1

test_add_duplicate_products ends

SQL (0.001645)   ROLLBACK

Just as before, the test data in the products table is deleted and re-inserted from the fixture file exactly once. As well, database transactions still bracket each test method to keep them isolated. But notice what happens inside of each test method. Rather than going to the trouble to instantiate all three products before each test method, Rails simply hands you the reigns. If you use a fixture accessor in a test method, the respective fixture data is loaded and cached. If you don't use a specific piece of fixture data in a test method, you don't pay to have it loaded.

I'll spare you the math on this one, and instead show the results on a real-world project a bit later.

Upgrading Your Tests

First, you don't necessarily have to change your tests when you upgrade to Rails 1.0. You can simply choose not to have Rails update your test/test_helper.rb file when you run the rails command in your project directory.

Or you can simply change the generated test/test_helper.rb file and reset the new defaults back to their old settings, like so:

self.use_transactional_fixtures = false
self.use_instantiated_fixtures  = true

Or you can add those two lines to the top of any existing test case and override the new defaults just for that test case, like so:

require File.dirname(FILE) + '/../test_helper'


class CartTest < Test::Unit::TestCase

  fixtures :products

  self.use_transactional_fixtures = false
  self.use_instantiated_fixtures  = true

  # Your tests go here.
end

Finally, you can bite the bullet and upgrade all of your tests by:

• Making sure you're using transactional database tables and
• replacing all occurrences of fixture instance variables with fixture accessor methods. Example: change @fred to users(:fred).

What's the Bottom Line?

You've been patiently waiting for a performance comparison on a real-world project, if only because you'd enjoy something to throw rocks at. Let me help you keep your arms pointed at the keyboard by saying that the numbers that follow don't relate to your project. As you've seen in the parenthetically-challenged math formulas, there are a few variables. Thus, you will get different results depending on how many fixtures you have, how much data is in each of those fixtures, how you write your tests, and how hard you press the Enter key to run the tests.

With that disclaimer behind us, I present you the performance numbers from a Rails project I'm working on.

Using the Old Defaults

> rake


(unit tests)

Finished in 13.426457 seconds.
57 tests, 134 assertions, 0 failures, 0 errors

(functional tests)

Finished in 20.938738 seconds.
53 tests, 183 assertions, 0 failures, 0 errors

Transactional Fixtures On, Instantiated Fixtures On

> rake


(unit tests)

Finished in 7.942351 seconds.
57 tests, 134 assertions, 0 failures, 0 errors

(functional tests)

Finished in 17.832574 seconds.
53 tests, 183 assertions, 0 failures, 0 errors

Transactional Fixtures On, Instantiated Fixtures Off

(This is the new default in Rails 1.0.)

> rake


(unit tests)

Finished in 5.667295 seconds.
57 tests, 134 assertions, 0 failures, 0 errors

(functional tests)

Finished in 14.663263 seconds.
53 tests, 183 assertions, 0 failures, 0 errors

Unit tests are almost 3x faster; functional tests are about 25% faster. I suspect it would be a lot more dramatic on bigger suites and fixtures. I've heard rumblings of total times being increased as much as 5x.

Summary

Simple: The combined effects of the new testing defaults in Rails 1.0 make your tests go faster. And when you're running tests after every change, every second counts...

(We'll be talking about this and other tantalizing Rails goodies in the Pragmatic Studio.)