[Ruby] Scalability tests

[Zed A. Shaw]

On Mon, 19 Mar 2007 14:31:00 -0700
"ben wiseley" <wiseleyb at gmail.com> wrote:

> After the last meeting (which was partially about "don't worry about
> scalability") I'm pretty hesitant to bring this up - but we're (
> http://activerain.com) growing fairly quickly, we've been appallingly bad
> about unit testing in general (this was against my wishes - I'm a big fan of
> TDD but I don't write the checks, thankfully it looks like they might be
> able "afford" this soon) and are probably a month out from having
> scalability issues (mostly DB and memory related).

<rant>
There's an old saying:  Your lateness does not constitute my emergency.
I never let others jeopardize my reputation for quality with their
demands of efficiency.  In the end, I end up having to pay for it with
exactly these kinds of situations, not the guy who writes the checks.
And when the shit hits the fan, who do you think the guy who writes the
checks blames?

Definitely not his accountant.
</rant>

> We'd basically like to run load tests that mimic X users doing various
> scenarios and then try to improve the numbers through tweaking code based on
> some combination of httperf and RailsBench output.  Is this the general
> idea?  Or is there a better way to go about this?

Users won't help you since you can't accurately define it and produce a
metric.  You have kind of two approaches to ease your scalability risk:

1) Write good Watir tests that run through the app and record timings
of the pages in a DB or something.  Then run this test in a ton of
threads to see if your site eats it and to spot pages that have a wild
change in their behavior under load.

WARNING:  This will NOT give you a measurement of speed, only tell you
if you have a problem when an insane user who has 20 arms, 7000
fingers, and a constant IV of meth jacked into his brain decides to use
every page on your site.

2) Go through your app and pick your most important pages, possibly by
usage potential.  Develop a measurement plan that includes target
metrics based on *definable* criteria that is the simplest possible.
This is always requests/second or a similar singular metric per
second.  Then do what topfunky's peepcode tells you to measure these
pages and make sure your changes make them faster.

The reason you can't do both #1 and #2 at the same time is because of
confounding.  #1 adds tons of complexity on purpose, so it adds too
much to your measurements to be useful.  #2 targets specific requests
against specific pages with specific data, so it can't tell you if your
site will stay up should #1 happen.  By doing them in an iterative
cycle you can then just get more assurance your site is tight.

Also, don't forget that you can do quite a lot of simple stuff in your
page to just make people think it loads faster.  Like, take all
your .js and .css files and make a single .js and .css file.  Using
firebug and tamperdata help a lot with this.

> We're running multiple mongrels on multiple web servers against a single
> PostgreSQL db right now but would be testing probably on a single box and
> coming up with a scale to let us know how that relates to production.  Or
> isn't that such a great idea?

Once you've got a good way of measuring your top pages (#2 from above)
then it'll apply to all your deployments and act as a good check.
Until then you'll be lost.
 
> Without zillions of rows in fixtures (which seems silly) the base output
> from the unit tests we do have don't seem to be too helpful (because all of
> this stuff flies on small datasets under light load)

Unit tests won't help you validate performance since that's dependent
on too many configuration elements.

Good luck.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/