Phasor Burn

Archive for the 'Work' Category

A Cautionary Tale

Friday, March 5th, 2010

Stay Focused, Stay Healthy Part 1, Part 2

Reminds me of the previous workplace. I didn’t get quite as far gone as this guy, but it was definitely a road I was heading down.

Thankfully, I’m in a much better place now.

Posted in General, Work, System Administration | No Comments »

Apache Bandwidth from access.log

Wednesday, March 3rd, 2010

Quick and dirty Apache Bandwidth graphs with GnuPlot

#!/bin/bash PATH=/opt/local/bin:$PATH


TMP=/tmp/`basename $0`.$$

trap "/bin/rm ${TMP} >/dev/null 2>&1" 0 1 15
gzcat *access.log.*.gz |\

 sed -e 's/\[//g' -e 's/\]//g' |\

   awk ' 
	{

		day=substr($4,0,2)

		month=substr($4,4,3)

		year=substr($4,8,4)

		hhmm=substr($4,13,5)

		if ( month =="Jan" ) month="01"

		if ( month =="Feb" ) month="02"

		if ( month =="Mar" ) month="03"

		if ( month =="Apr" ) month="04"

		if ( month =="May" ) month="05"

		if ( month =="Jun" ) month="06"

		if ( month =="Jul" ) month="07"

		if ( month =="Aug" ) month="08"

		if ( month =="Sep" ) month="09"

		if ( month =="Oct" ) month="10"

		if ( month =="Nov" ) month="11"

		if ( month =="Dec" ) month="12"

		timestamp=year"-"month"-"day"-"hhmm

		hits[timestamp]++

		bytes[timestamp] += int($(10))

	}
    END { for (timestamp in hits)

                  printf("%s %10d %20d\n",

                             timestamp,

                             hits[timestamp],

                             bytes[timestamp]*8/60/1024)

}
' | sort > ${TMP}
# Now we have YYYY-MM-DD-HH:MM hitspermin and Kbps
cat < < EOF_GNUPLOT_SCRIPT  | gnuplot

  set terminal png enhanced size 1024,768

  set xdata time

  set timefmt "%Y-%m-%d-%H:%M"

  set format x "%d-%H:00"

  set xlabel "time"

  set grid

  set style data points

  set xrange [ "2010-02-24-00:00" : "2010-03-02-23:59" ]

  set output "hits.png"

  set ylabel "Hits per minute"

  set title "Apache Hits"

  plot "$TMP" using 1:2 title ""

EOF_GNUPLOT_SCRIPT

cat << EOF_GNUPLOT_SCRIPT | gnuplot set terminal png enhanced size 1024,768 set xdata time set timefmt "%Y-%m-%d-%H:%M" set format x "%d-%H:00" set xlabel "Time" set grid set style data points set xrange [ "2010-02-24-00:00" : "2010-03-02-23:59" ] set yrange [ 0 : 6000 ] set output "bandwidth.png" set ylabel "Kbps" set title "Apache Bandwidth Past Week" plot "$TMP" using 1:3 title "" EOF_GNUPLOT_SCRIPT

Ow, that smarts!

Wednesday, February 17th, 2010

Today just gave me a small moment of Imposter Syndrome

Mid point of week three of the new job, everything going swell and all except and I’ve shot my foot off today. Something that doesn’t happen very often, but does now and again to remind you that the Universe is unforgiving.

I was using udpcast to reimage a bunch of new developer workstations. They were all attached to a 1 Gbps switch, and were reimaging at approx 600 Mbps.

That same 600 Mbps of multicast traffic tried to squeeze itself out onto the internal 100Mbps network. I had forgotten to disconnect the uplink.

Whups.

LAN connectivity for everyone went to hell, access to the data centre over the wan went bye bye, and the voip telephone system went awol. Not good considering that included a customer call centre in our other office the next building over.

It was about a 5 or 7 minute outage total, before I figured out I was the cause of the issue and ran to unplug the cable. Yes, I did own up to it immediately.

Now really, with QoS this shouldn’t have happened. Supposedly the voip system is on a separate VLAN and has QoS enabled.

Definitely need to look into that after hours sometime…

Posted in General, Work, System Administration | No Comments »

Dev vs Ops

Monday, February 1st, 2010

Velocity 09: John Allspaw, “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr”

“In the last week there were 67 deploys of 496 changes by 18 people” – Flickr Dev Blog, December 17th 2008.

For deploying new code, planning projects, and crisis management, communications and cooperation between development and operations is paramount. As web infrastructures grow, the line between systems and software become quite blurred. Operations and development are disciplines that historically have been limited to a predictable list of responsibilities and have suffered from a culture of finger-pointyness.

We’ll talk about why and how these roles have mixed in Flickr’s environment for the better.

We’ll discuss tools, techniques, culture, and the processes of transparency that enables a healthy respect for the Big Red Deploy Button™ as well as incident management, escalation, and troubleshooting.

Tools, techniques, and culture:
• Overview of our one-button code deployment, application-level logging, and dev/ops communications tools.
• Examples of adaptive monitoring and metrics collection, capacity awareness and planning must-haves.
• Rethinking the ops/developer relationship and mutual respect for responsibilities and expertise.

Curiously compelling idea…

There’s a video of their talk. Watch it.

Posted in General, Work, System Administration | No Comments »