[Ruby] Scalability tests

[ben wiseley]

After the last meeting (which was partially about "don't worry about
scalability") I'm pretty hesitant to bring this up - but we're (
http://activerain.com) growing fairly quickly, we've been appallingly bad
about unit testing in general (this was against my wishes - I'm a big fan of
TDD but I don't write the checks, thankfully it looks like they might be
able "afford" this soon) and are probably a month out from having
scalability issues (mostly DB and memory related).

I know that http://railsexpress.de/blog/ has a lot of information on this
and I think RailsBench would do a lot of this and we'll be watching
Geoffrey's httperf podcast but I'd be interested in hearing your thoughts on
this if any of you have a few minutes...

We'd basically like to run load tests that mimic X users doing various
scenarios and then try to improve the numbers through tweaking code based on
some combination of httperf and RailsBench output.  Is this the general
idea?  Or is there a better way to go about this?

We're running multiple mongrels on multiple web servers against a single
PostgreSQL db right now but would be testing probably on a single box and
coming up with a scale to let us know how that relates to production.  Or
isn't that such a great idea?

Without zillions of rows in fixtures (which seems silly) the base output
from the unit tests we do have don't seem to be too helpful (because all of
this stuff flies on small datasets under light load)

Basically just looking for some direction if anyone has a few minutes.

-ben