[Ruby] Scalability tests

[Eric Hodel]

On Mar 19, 2007, at 14:31, ben wiseley wrote:
> We'd basically like to run load tests that mimic X users doing various
> scenarios and then try to improve the numbers through tweaking code  
> based on
> some combination of httperf and RailsBench output.  Is this the  
> general
> idea?

It depends on what you're looking to optimize.  Trying to optimize  
Rails from the HTTP level may not give you the information you need  
to make proper decisions.  (Yes, this is slow, but why?)  I find the  
log files much, much more informative, for example I found a 25MB  
memory leak in 43 Things due to sloppy coding through additional  
logging that no amount of GET requests could illuminate.

> Or is there a better way to go about this?

There's no sense trying to figure out what to optimize by mimicking  
usage when all that information is easily extractable from your logs.

Use the production_log_analyzer gem to figure out what pages get hit  
the most and how much they cost and look at where they spend their time.

pl_analyze will tell you if time is spent waiting for the DB,  
processing a view or elsewhere (controller/model before render), and  
then help you figure out if you need to change DB queries or add  
indexes, cache renders, or optimize some code respectively.

Here's trimmed output from 43 Things from about a year ago:

Request Times Summary:     Count    Avg    Std Dev  Min    Max
ALL REQUESTS:              2035035  0.521  1.218    0.000  1387.964

ThingsController#view:     420323   0.974  0.803    0.000  55.557
PeopleController#view:     302102   0.466  0.527    0.000  46.119
PeopleController#progress: 254735   0.710  0.422    0.000  42.280

The above shows that ThingsController#view is dog slow, and needs  
some work.  Looking below you'll see that half the time is spend  
rendering, and the DB times for all three actions are close.

I doubt DB work could improve things here.

DB Times Summary:          Count    Avg    Std Dev  Min    Max
ALL REQUESTS:              2035035  0.106  1.034    0.000  1387.940

ThingsController#view:     420323   0.136  0.417    0.000  52.436
PeopleController#view:     302102   0.183  0.363    0.000  44.985
PeopleController#progress: 254735   0.154  0.225    0.000  41.713

Render Times Summary:      Count    Avg    Std Dev  Min    Max
ALL REQUESTS:              2035035  0.211  0.401    0.000  17.207

ThingsController#view:     420323   0.535  0.522    0.000  5.996
PeopleController#view:     302102   0.230  0.263    0.000  8.691
PeopleController#progress: 254735   0.129  0.172    0.000  8.491

Here we see that PeopleController#progress renders really fast  
compared to the other three, but takes .7 seconds overall, so the  
controller is either doing too much work or needs optimization.

> We're running multiple mongrels on multiple web servers against a  
> single
> PostgreSQL db right now but would be testing probably on a single  
> box and
> coming up with a scale to let us know how that relates to  
> production.  Or
> isn't that such a great idea?

There are many, many factors to tune.  pl_analyze will tell you if  
your DB is overloaded and needs tuning (indexes or memory), or if  
your code is crappy.  It won't tell you if you don't have enough web  
servers or if you're running too much load on your web server.  (I've  
found that Rails seems to have a sweet spot around 3 processes per  
CPU, and adding more processes has diminishing returns.)

pl_analyze won't tell you if you're losing speed by proxying requests  
through some other web server rather than handing them straight off  
to mongrel (when experimenting on 43 Things, Apache 1.3's mod_proxy  
gave a 20% slowdown when proxying to localhost).  You'll need to use  
something like httperf or railsbench or ab to find this out.  If you  
need to load balance, get some hardware.  Its less painful in the  
long run, you should be able to get your hosting company to take care  
of it for you.

Neither pl_analyze nor any benchmark tool will tell you if your  
processes are churning rapidly.  Starting up a Rails app on a loaded  
system is an expensive operation, check your logs and make sure they  
don't die until they've run for a couple hours.