rrdtool tidbits
Introduction
rrdtool (round robin database tool) by Tobi Oetiker is a small database & graphing tool for many kinds of data. If you know, mrtg, you´ll have an idea of what its output may look like; just think of rrdtool as far more flexible (and thus a good deal more complex ;).
On this page, I have put up some snippets that I used in the past to create graphs about different kinds of data. They´re provided as-is -- i.e. you might have some work to do before they work exactly the way you want them to.
These scripts are released under the GNU GPL and therefore Free software; you may copy, redistribute, or base your own work on it, provided the license stays the same. Crediting wouldn´t go amiss, either.
The guts of the matter
Overview
- Requirements
- A "simple" traffic graph
- TCP connections per minute
- Quick´n´dirty Rackshack bandwidth stats aggregator
- Round Trip Times
- BTrackalyzer
- Closing
Requirements
- Perl (anything after 5.005 should do)
- rrdtool (also the shared module for Perl)
- an SNMP daemon or a device that supports it and accompanying tools
- Most scripts here assume you have a *IX-like operating system or at least an emulated environment thereof.
A "simple" traffic graph
Description
This will be a graph showing the input/output bandwidth utilization of a single interface, say, on a webserver or on your internet connection at home; you would achieve similar effects with mrtg; although you would loose some flexibility in the graphing ... In this case, the graphs are created by BASH scripts, and the data is gotten from SNMP via a commandline utility. This is fine if you just have a couple of graphs, but once you get into the hundreds, another approach will yield better results; maybe that´ll find its way into this page as well at some point ;)
Database Creation
First, you need to create a database that will contain the measured data; The script below will create one named interfaces.rrd with a starting date of "now" (date +"%s" will return the current time as a unix timestamp) with two datasources, eth0_in and eth0_out. We assume that they are 100Mbit/s interfaces and are thus able to handle a maximum of 12.5 MBps. For these two datasources, we save the average bandwidth and the peak bandwidth for 5 minte, 30 minute, 2 hour, and 1 day intervals. (MAX in 5 min intervals is superfluous here, but diskspace is cheap ;)
Data Gathering
Next up is the script that will get the needed values out of SNMP and into
the database. It looks scary and is probably easy to implement in another
way, but hey ;)
A few notes on the below script : I assume your community string is
COMMUNITY, which it most likely will not be. Check your snmpd.conf to see
what it actually is (this needs to be at least a read only community with
access to the interface tables); also, I assume that the server to be
queried is sitting on localhost. Change accordingly if needed.
First, I set the Timezone to Europe/Berlin (which is my own), in case the server is located somewhere else; you can leave out this step if you want. Next up is getting the data from snmp with snmpget ... Depending on which version of snmpget you have, the following sed command may not be neccessary and it will return the bytecounter for the specified interface (to get a list of interfaces, enter
That should tell you which index you are interested in.
Next up is some simple error checking; if the returned string is empty,
replace it by U (for unknown). Last, but not least, call rrdtool to update
the interfaces database with the newly found values.
Graph Generation
Next up is the largest part : the graph generation. I will assume that the graphs are to be placed into a subdirectory named "graphs", that you want daily, weekly, monthly, and yearly graphs of the same size, and that your format of choice will be PNG.
The below code will generate the four graphs; their code is mostly
similar, so I´ll only explain the daily graph.
First up is the call to rrdtool :
--start -86400 tells it to start a day back, ending 5 minutes back (for
which we may or may not yet have data).
Next is the DEFinitions of what datasources we are going to use.
Fairly straight forward, using the AVERAGEs for both eth0_in and eth0_out from interfaces.rrd with the names eth0_in_bytes and eth0_out_bytes.
The first two lines get us the Mbps as opposed to MBps by multiplying
the values by 8.
Line 3 goes on to create another temporary variable with
the negated value of the current egress bandwidth; it´ll come in
handy when graphing.
Lines 4 through 5 so some more magic; the first check whether or not the
current speed fits into 12.5MBps, then check whether it´s unknown
(if so setting to zero for the computation), then multiply it by the number
of seconds the graph covers. What you get here is for each second a value
86400 times as high as it should be; this can later be used for determining
the total amount of data that has flown through the interface in the
graphed interval by just taking the average over these values, as we will
see below.
Line 6 just adds in+out for later use.
This, then, starts graphing two areas; one going up (ingress) and one
going down (egress), in green and blue with labels eth0_in and eth0_out.
Next up are some comments giving a legend and the current date, then
Which basically just tells the speeds over the graph (peak, average, and current). One point of notice is
This will use the above inflated values and average over them, giving the total amount of data transferred in the interval. Note that these figures are given in Metric units (i.e. 10 to the power of X) instead of binary units (i.e. 2 to the power of X). I have not yet found a way to tell rrdtool to selectively use different scales.
Some more parameters for the graph.
-v specifies what is shown on the side,
-t what is shown in the title,
-h the height of the graph.
-w the width of the graph. If you wanted to use the space perfectly, -w 288 would be
better -- one pixel per datapoint; I like this size though ;)
-x tells about where vertical lines are to be drawn, and what the legend
should look like. Consult the rrdtool manual on how about that works, if
you are interested ;)
You might also want to set -u and -l, which specify the initial
upper and lower bounds of the graph (i.e. you could make the scale always
show 100Mbit/s, even if you only use 2). Also of interest in this is
--rigid, which will bound it no matter what (even if the values are too
high to display)
Download the script below (only showing the daily graph here, the script contains weekly, monthly, and yearly as well)
Getting it to work
To get this to make some nice graphs, you need to first use create_interface.sh, and add update_interfaces.sh and graph_interfaces.sh to your crontab. You may need to include a
at the top of the scripts called by cron, so that your working directory is always right. Personally, I use
In my crontab.
TCP Connections per Minute
Description
This graph will show the average incoming/outgoing TCP connections per minute; the data is again gotten from SNMP, and the structure is much like the one for the simple traffic graph; due to that, I´ll focus on the differences instead of saying everything again :)
Database Creation
Pretty straightforward; like above; I chose the limit of 1000 connections/sec max. This should suffice for most applications.
Data Gathering
Same as above, just getting a different value from the SNMP daemon.
Graph Generation
Download the script below (day, week, month, year, not just what´s shown below ...)
Most of this should seem familiar from the traffic graph. Some notable differences :
Here, we also grab the saved MAX values from the database to be able to graph peak values in addition to average values. This is useful if each column in the graph graphs more than one datapoint, which is usually averaged.
To get at the connections per minute, we take the connections per second time sixty. Remember that the graph has a 5 minute granularity; We are not loosing information here.
First, the peaks get drawn (in shades of gray), then the averages (on top of the peaks, so that they are visible). STACK stacks the two graphs on top of one another.
Getting it to work
The approach here is no different from the traffic graph -- call create_tcpconns.sh once, and update_tcpconns.sh and graph_tcpconns.sh every five minutes from cron.
Quick´n´dirty Rackshack bandwidth stats aggregator
Description
RRDtool may also be used as a Perl Module. The other day I used that approach to generate a combined graph of the network utilization pages offered by Rackshack.net, where a few servers I adminstrate are hosted. The result can be seen on the right; click on it for a full view.
Database Generation, Data Gathering, Graphing
All this is done by a single perl-script. It´s neither commented nor particularily well written (as the title suggests, it really is quick and dirty); It´s avaiable for download below.
The script first grabs the network overview page, parses it for links to the sub-mrtg pages; it then grabs those, stripping out the interface name and cuin d/cuout d for use with rrdtool. It keeps a list of known interfaces so that they´re used in the same order even if new ones should appear (which has been known to happen ;) and uses some reasonably simple list processing to create the rrdtool calls.
Source
Download [6k] or see the highlighted sourcecode
Round Trip Times
Description
This is a script that generates a graph depicting round trip times to various hosts, as gathered by fping.
Sorry, I haven´t had the time to put this up, yet.
BTrackalyzer
Description
BTrackalyzer is a script to parse BitTorrent tracker logfiles, extracting lots of statistics and some graphs. These graphs are generated using rrdtool. Its website offers more details, the script itself, and some commentary ;)
Currently, btrackalyzer calls an external rrdtool process for updating the databases and graphing to reduce dependencies; this should not be necessary anymore once I get some coding on that done, though. For now, it´s just another example of how one might use rrdtool.
Closing
Maybe these scripts or explanations have helped you; if so, that´s good; if you have suggestions, comments, etc., please drop a line and I´ll try to get back to you as soon as possible.
If this page helped you and you´d like to help me out financially,
please consider donating via PayPal... It doesn´t have
to be much; I´m happy with anything that helps. Of course you will
be mentioned here (if so desired).
