Friday, 13 April 2012

Installing Graphite on CentOS - Part 3 - Posting some data to Graphite

In this final section I'll talk about some utilities out there which easily enable you to post data to Graphite. While not strictly CentOS related, it finishes off this series of posts on graphite quite nicely.

With your graphite installation up and running you should see you have two listening ports open - TCP 2003 and 2004. Carbon listens on these ports for incoming connections and translates the data into datapoints which it inserts into whisper files held under /opt/graphite/storage.

The format of the data is object value timestamp where timestamp is simply `date +%s`.

The object is simply how you want to structure your data in graphite - e.g. servers.hostname.mem would be a "folder" for memory stats on a particular host and may contain substats such as swap util, cache, buffers etc. Graphite doesn't care how you structure the data or what data you send it - its merely a presentation tool at the end of the day!

A basic way of writing some data into graphite could be through netcat - the following script run every minute by cron captures the temp of my graphics card and sends it to graphite running on my local system:

#!/bin/sh
CARBON_PORT=2003
CARBON_SERVER=localhost
temp=`cat /proc/acpi/thermal_zone/TZ00/temperature | awk '{print $2}'`
echo "servers.`hostname -s`.graphiccard.temp $temp  `date +%s`" | nc $CARBON_SERVER $CARBON_PORT


Which ends up looking like this in graphite:



Of course, writing some scripts like these isn't really scalable - both from an data collection perspective and managing those endpoints, and the fact your graphite system may be handling thousands of tcp connections per second.

Graphite provides a pickle interface on port 2004 - pickle simply allows you to batch lots of stats in one string and seperates them out on delivery, reducing your TCP overhead. You can further improve performance by using statsd which is a UDP listener which then forwards the data received onto carbon - this is very useful if you don't want the TCP connection to carbon interfering with the performance of your application writing the data - it literally fires and forgets!

Fortunately, there are plenty of tools out there which easily allow you to collect a whole raft of stats and send them to graphite - I'll mention two I use here but they are by no means the only ones out there:

Diamond is a python daemon that collects system metrics and publishes them to graphite. It works particularly well on CentOS systems. Its not limited to OS metrics either - it can be extended to pull in all sorts of stats including memcache, mysql etc.

I build diamond as an RPM to deploy - to build the RPM simply follow these steps:

1. On a system with build tools such as rpmbuild, check out the latest codebase using git, Use python setup.py bdist_rpm to create an initial SRPM.

2. Install that SRPM and update the SPEC file with the relevant sections and add an init script such as this one:

#!/bin/sh
#
# chkconfig: - 90 60
# pidfile: /var/run/diamond.pid
# config: /etc/diamond/diamond.conf
#
### BEGIN INIT INFO
# Provides: diamond
# Required-Start: $local_fs $remote_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Short-Description: run diamond daemon
# Description: Diamond collects system stats and passes them to graphite
### END INIT INFO
 

# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0
prog="diamond"
user="root"
exec="/usr/bin/diamond"
program="/usr/bin/diamond"
lockfile=/var/lock/subsys/diamond
config=/etc/diamond/diamond.conf

start() {
    echo -n $"Starting $prog: "
    daemon --user=$user $program
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch $lockfile
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $program
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f $lockfile
}

# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        stop
        start
        ;;
  *)
        echo $"Usage: $prog {start|stop|restart}"
        RETVAL=2
esac


3. At this point, you'll be able to build a new RPM and SRPM

Diamond provides quite a bit of info on configuration on their wiki - the main config file is /etc/diamond/diamond.conf - its pretty self explanatory

The other tool I frequently use is jmxtrans - this allows you to pull in stats from JMX on running java processes such as tomcat and forward them onto graphite. The build process to create an RPM is very similar to that as I have done for diamond.

There's a lot more you can do with Graphite including configuring relays so you can duplicate your data in two places at once, setup carbon clusters and so on. The first hurdle of course is getting graphite up and running and hopefully these posts will have helped towards that. Its worth reading up on whisper - the file based database used to store stats by carbon and of course read by graphite - you can tune (and retune) the data storage including retention time and compacting datapoints - whisper databases are similar to RRD but a bit more flexible.

Friday, 6 April 2012

Installing Graphite on CentOS - Part 2 - Setting up Graphite

The first post in this series detailed how to build rpms for graphite. This next post details how to get a working graphite installation setup on your production CentOS servers.

These instructions make a few assumptions - the system can use EPEL repositories and it has SELINUX disabled. This sets up a minimum graphite install - literally the carbon-cache daemon to receive the data and the graphite web front end to present the data via a web browser. You can extend the setup to carbon-cache clusters fronted by relay servers, duplicate your carbon data across datacentres, run mysql backends and much more. I'll touch on some of these scenarios in future posts.

1. Install the dependencies:

yum -y install Django django-tagging bitmap bitmap-fonts python-zope-interface python-memcached python-sqlite2 python-ldap python-twisted pycairo memcached

2. Install the graphite RPMs:

yum --nogpgcheck localinstall carbon-0.9.9-1.noarch.rpm graphite-web-0.9.9-1.noarch.rpm whisper-0.9.9-1.noarch.rpm

3. Setup the carbon and graphite-web configuration files:

cd /opt/graphite/conf/
cp graphite.wsgi.example graphite.wsgi
cp storage-schemas.conf.example storage-schemas.conf
cp carbon.conf.example carbon.conf
cd ../webapp/graphite
cp local_settings.py.example local_settings.py

4. Update local_settings.py above with correct Timezone, memcache location, ldap database for authentication. For a basic setup this is the only config file you need to modify.

5. Create the the django DB. Note that you can host this within mysql. You will need to install python-MySQL to do so and configure local_settings.py accordingly. Otherwise it will create a local file based database.

# python /opt/graphite/webapp/graphite/manage.py syncdb
Creating table account_profile
Creating table account_variable
Creating table account_view
Creating table account_window
Creating table account_mygraph
Creating table dashboard_dashboard_owners
Creating table dashboard_dashboard
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table auth_message
Creating table django_session
Creating table django_admin_log
Creating table django_content_type
Creating table tagging_tag
Creating table tagging_taggeditem
Creating table events_event

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): no
Installing index for account.Variable model
Installing index for account.View model
Installing index for account.Window model
Installing index for account.MyGraph model
Installing index for dashboard.Dashboard_owners model
Installing index for auth.Permission model
Installing index for auth.Group_permissions model
Installing index for auth.User_user_permissions model
Installing index for auth.User_groups model
Installing index for auth.Message model
Installing index for admin.LogEntry model
Installing index for tagging.TaggedItem model
No fixtures found.

6. Now Install and configure Apache - we use wsgi to create running web enabled graphite cgi processes. To enable this install mod_wsgi:

yum -y install httpd mod_wsgi

7. Create a virtualhost file for graphite /etc/httpd/conf.d/graphite.conf - your virtualhost config will probably differ slightly depending on how you want your virtualhost configuration:

<VirtualHost *:80>
  ServerName graphite.somehost.com
  DocumentRoot "/opt/graphite/webapp"
  ErrorLog logs/graphite_error_log
  TransferLog logs/graphite_access_log
  LogLevel warn
  WSGIDaemonProcess graphite processes=5 threads=5 display-name=" {GROUP}" inactivity-timeout=120
  WSGIProcessGroup graphite
  WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi
  Alias /content/ /opt/graphite/webapp/content/
  <Location "/content/">
   SetHandler None
  </Location>
  Alias /media/ "/usr/lib/python2.6/site-packages/django/contrib/admin/media/"
  <Location "/media/">
   SetHandler None
  </Location>
  <Directory /opt/graphite/conf/>
   Order deny,allow
   Allow from all
  </Directory>
</VirtualHost>

8. Update /etc/httpd/conf.d/wsgi.conf with a socket prefix:

LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix /var/run/wsgi

9. Allow apache access to the storage data:

chown -R apache:apache /opt/graphite/storage/

10. Create the init script /etc/init.d/carbon-cache. My script here also can start the carbon relay if you uncomment the relevant sections - carbon-relay is useful if you are running graphite in a high-availability setup but I'll talk about that in future posts....

#!/bin/bash
#
# This is used to start/stop the carbon-cache daemon

# chkconfig: - 99 01
# description: Starts the carbon-cache daemon

# Source function library.
. /etc/init.d/functions

RETVAL=0
prog="carbon-cache"

start_relay () {
    /usr/bin/python /opt/graphite/bin/carbon-relay.py start
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

start_cache () {
     /usr/bin/python /opt/graphite/bin/carbon-cache.py start
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

stop_relay () {
    /usr/bin/python /opt/graphite/bin/carbon-relay.py stop
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

stop_cache () {
          /usr/bin/python /opt/graphite/bin/carbon-cache.py stop
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

# See how we were called.
case "$1" in
  start)
    #start_relay
    start_cache
        ;;
  stop)
    #stop_relay
    stop_cache
        ;;
  restart)
    #stop_relay
    stop_cache
    #start_relay
    start_cache
    ;;

  *)
        echo $"Usage: $0 {start|stop}"
        exit 2
        ;;
esac


11. Start the services:

service memcached start
service carbon-cache start
service httpd start

You should now have a running carbon-cache process and be able to browse your graphite data through the web-interface. In the next post I'll discuss some techniques for injecting your data into graphite.

Tuesday, 3 April 2012

Installing Graphite on CentOS - Part 1 - Building the RPMs

Graphite is an on-demand graphing utility - fantastically useful for all sorts of reasons to the sysadmin. If you're reading this chances are you know what graphite is all about and you're after some instructions on how to deploy onto CentOS/RHEL using RPMs rather than following the default instructions which are more distro agnostic. More information on graphite can be found here but essentially it allows you to automatically generate pretty graphs like these:



This post will detail how to create the graphite RPMs which can then be deployed onto production servers through whatever mechanism you use to distribute package.

You'll need to run up a non-production machine (I use my desktop) with CentOS (32 or 64 bit - note though that memcache is only available for 64-bit so if you want to use memcache on the same host as your graphite installation you'll be stuck with x86_64).

On this machine, install the dependencies:

yum install -y gcc zlib-devel curl curl-devel openssl rpm-build gcc-c++ rpm-build python python-ldap python-memcached python-sqlite2 pycairo python-twisted Django django-tagging bitmap bitmap-fonts python-devel glibc-devel gcc-c++ openssl-devel python-zope-interface httpd memcached mod_wsgi

Switch to your rpmbuild area (e.g. ~/rpmbuild), download the sources and create backups of them:

cd ~/rpmbuild/SOURCES
wget http://launchpad.net/graphite/0.9/0.9.9/+download/whisper-0.9.9.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.9/+download/carbon-0.9.9.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz
cp carbon-0.9.9.tar.gz carbon-0.9.9.tar.gz.orig
cp graphite-web-0.9.9.tar.gz graphite-web-0.9.9.tar.gz.orig
cp whisper-0.9.9.tar.gz whisper-0.9.9.tar.gz.orig


Build and install Whisper SRPM, then update the SPEC file, restore the original source and recreate the RPMs (if you want you can update the spec files to add a suffix to the end of the release information and add a description):

cd ~/rpmbuild/SOURCES
tar -zxvf whisper-0.9.9.tar.gz
cd whisper-0.9.9
python setup.py bdist_rpm
rpm -ivh dist/whisper-0.9.9-1.src.rpm
cd ..
cp whisper-0.9.9.tar.gz.orig whisper-0.9.9.tar.gz
cd ../SPECS/
vi whisper.spec
rpmbuild -ba whisper.spec
yum --nogpgcheck localinstall ../RPMS/noarch/whisper-0.9.9-1.noarch.rpm

Build and install Graphite web SRPM, then update the SPEC file, restore the original source and recreate the RPMs:

cd ~/rpmbuild/SOURCES/
tar -zxvf graphite-web-0.9.9.tar.gz
cd graphite-web-0.9.9
python setup.py bdist_rpm
rpm -ivh dist/graphite-web-0.9.9-1.src.rpm
cd ../
cp graphite-web-0.9.9.tar.gz.orig graphite-web-0.9.9.tar.gz
cd ../SPECS/
vi graphite-web.spec
rpmbuild -ba graphite-web.spec
yum --nogpgcheck localinstall ../RPMS/noarch/graphite-web-0.9.9-1.noarch.rpm

Build and install Carbon SRPM, then update the SPEC file, restore the original source and recreate the RPMs:

cd ~/rpmbuild/SOURCES/
tar -zxvf carbon-0.9.9.tar.gz
cd carbon-0.9.9
python setup.py bdist_rpm
rpm -ivh dist/carbon-0.9.9-1.src.rpm
cd ..
cp carbon-0.9.9.tar.gz.orig carbon-0.9.9.tar.gz
cd ../SPECS/
vi carbon.spec
rpmbuild -ba carbon.spec

The RPMS and SRPMS are now under ~/rpmbuild/RPMS/noarch and ~/rpmbuild/SRPMS. You can now copy these up to your RPM hosting area/production system on which you are installing graphite.

Next post will detail more information on the actual installation of graphite on a production system, including setting up Apache, configuration of graphite itself. Finally, I will put a post on how to get data into graphite, including some of the utilities I've found to help with this.