Phasor Burn

Lets pretend you have a stock ticker application that is getting hammered. A lot of people are interested in one particular stock for some reason.

The poor little app server is running some lumbering hulking piece of code written in a legacy language (java). It can’t keep up with all the requests for this same stock symbol over and over.

Furthermore, the developers haven’t been able to put their own caching code into the application just yet.

How do we fix this, from a system administrators perspective?

Perhaps the better way would be to use mod_cache.

Unfortunately, this option did not exist on our web servers and we needed something PDQ to take the load off the app servers. We went with mod_rewrite instead and a small script to do the caching.

Cron runs a script every 5 minutes, which calls the app server, caches the results in a file. Then mod_rewrite rules tell Apache to use that instead of going to the app server (via existing rewrite rules)

In the apache virtual host entry :

RewriteCond %{REQUEST_URI} ^/ticker/s=AAPL$
RewriteCond /app/ticker/cache/AAPL -f
RewriteRule ^(.*) /app/ticker/cache/AAPL [L]


A script to maintain the cache :

#!/bin/bash

The cron entry to make it go :

0,5,10,15,20,25,30,35,40,45,50,55 * * * * /app/ticker/bin/update-ticker-cache >/dev/null 2>&1


Now when the outside url http://www.example.com/ticker/s=AAPL is hit, apache looks for the file /app/ticker/cache/AAPL and delivers its contents instead of passing the request through to the app server (via other rewrite rules).

Cron and the script keep this cache updated every 5 minutes.

The graph speaks for itself.

This is based on a real world example. Details have been changed to protect the guilty parties.

Excel computed it’s brains out for 30+ seconds before giving me a partial graph and the complaint that I had too much data for it to continue. I guess you can’t really do anything *serious* with Excel for charting data, such as a month of 1-minute interval data points.

(60 minutes x 24 hours x 30 days = 42,200 points)

I had to resort to gnuplot, which is a bit funky but easily handles large datasets such as this.

What was I trying to graph anyways?

Web server hits per minute and average response time per minute, over a month.

I had custom apache log lines that looked like this :
63.214.229.120 - - [01/May/2009:00:00:00 +0000] "GET /s HTTP/1.1" 302 381 - - - "http://muy/url/here" "MOT-SPARK/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0" "-" "-" 922us 0s -


I piped the logs through distillation code :

#!/bin/sh

END {
for (timestamp in hits) {
print timestamp " " hits[timestamp] " " timeavg[timestamp]
}
}
' | sort


Which after about a minute resulted in data looking like :

Time, Hits, Average Response (ms)
2009-05-01-00:00 357 508
2009-05-01-00:01 363 607
2009-05-01-00:02 357 589
2009-05-01-00:03 381 693
2009-05-01-00:04 405 576
2009-05-01-00:05 391 369


( That was the data set that Excel choked on… )

Now, create a gnuplot script for hits per minute

#!/opt/local/bin/gnuplot
set terminal png enhanced size 1024,768
set xdata time
set timefmt "%Y-%m-%d-%H:%M"
set format x "%d-%m-%Y"
set xlabel "time"
set grid
set style data points

set output "web01-05-hits.png"
set ylabel "Hits per minute"
set title "Web01 - May - Hits per minute"
plot "web01-05.dat" using 1:2 title ""


which created this graph in under a second :

Finally, the response time averages per minute

#!/opt/local/bin/gnuplot
reset
set terminal png enhanced size 1024,768
set xdata time
set timefmt "%Y-%m-%d-%H:%M"
set format x "%d-%m-%Y"
set xlabel "time"
set grid
set style data points

set output "web01-05-response.png"
set ylabel "Average Response Time (ms)"
set title "Web01 - May - Average Response"
plot "web01-05.dat" using 1:3 title ""


[ Click on the graphs to expand them to a readable size. ]

So yeah. I tried to use Excel as these were to be one-off charts and it seemed like the easiest way to do it at first.

Seduced by the simplistic, (and simpleton) microsoft way at the start…

In the end, I learned a bit about gnuplot and how dead simple it really is to ask it to chew through a decent sample size and produce a nice enough looking graph in under a second.

Does The Linux Desktop Innovate Too Much?

<kosh>Yes.</kosh>

I’ve been waiting for this kind of article to come from inside the FOSS camp. I hope it is just one of many pebbles voting for the usability avalanche that leads to stability and greater traction.

In the mean time, I’ll stick with my proprietary locked down GUI which is based on a certified UNIX platform underneath, both of which do not get in my way and merely help me get my daily work done without getting in my face.

What you talkin’bout Willis?

Ever hear of OS X? 10.5 Leopard is UNIX certified and has more a respectabledesktop share ( approx 9% ) than all the other Unix and Unix-wannabe (Linux) variations out there.

I can’t say I am unhappy with the office’s new internet pipe. It has been in place maybe 6 months now.

Enmax 100Mbit fibre. Yummy except for the bandwidth caps, so no frequent large file transfers (measured in tens of GB) over this to the datacentre. That’s what portable hard drives are for anyways.

Office -> Datacentre via public internet
8 hops, mostly Enmax, BigPipe, Shaw, Telus, Telus internal.
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.815/2.334/10.166/0.900 ms

We talk mostly to the machines at the datacentre via VPN. One would think VPN would slow things down with the encryption, connection tracking etc. Especially since the VPN devices on each end are actually CentOS virtual machines with OpenVPN.

Yes, virtual machines.

The only time there is noticable impact seems to be when backups are running and hammering the VMWare host or guest on either end. Aside from that, we’re nice and zippy.

100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.278/2.943/8.176/0.733 ms

I really can’t complain about this.

Much better than that crappy AllStream ADSL and Shaw Cable we had in the office previously.