Artur Krzywanski

Puppet dashboard is a useful tool in terms of reporting and actually seeing what’s going on with your puppets. The things get a bit tricky when you have hundreds of nodes reporting every 5 minutes or so and running dashboard on a single host.

I assume you are using dashboard already and you looking into scaling it, although I’ll describe how to install it from scratch. This post will help you set up 2+ dashboard nodes with one node running MySQL db and shared spool/ directory.

From puppet dashboard documentation:

Run exactly one delayed_job worker per CPU core.

This basically means that in order to process a lot of reports quickly you need enough CPU power.

MySQL

Puppet reports can clogged up your DB fast and InnoDB make things even worse (in terms of storage) so just to get good performance set max_allowed_packet to 32M and add following innodb settings:

innodb_file_per_table = 1
innodb_data_file_path = ibdata:10M:autoextend:max:30G
innodb_buffer_pool_size = 64MB

Next create a dashboard user accessible from any host (remember to set MySQL to listen on public NIC and add firewall rules to be secure).

CREATE USER 'dashboard'@'%' IDENTIFIED BY 'PASS';
GRANT ALL ON puppet_dashboard.* TO 'dashboard'@'%';

Dashboard

You need to pull same code to all nodes you want to use (and install rubygems 1.3.7):

Edit database.yml and set proper details in production section.

Create DB on master node and apply migrations:

Apache

For security add password protection for everyone apart of puppet master

<VirtualHost *:80>

	ServerName dashboard.dns.com
	ServerAdmin support@dns.com

	Order allow,deny
	Allow from 10.5.3.7 # Puppet master's IP
	Satisfy any
	AuthType Basic
	AuthName "Restricted Access"
	AuthUserFile /usr/local/puppet-dashboard/htaccess
	Require user admin

	ProxyPass / http://localhost:3000/
	ProxyPassReverse / http://localhost:3000/
	ProxyPreserveHost On

</VirtualHost>

Enable Proxy mod and create password file:

SSHFS

Sharing a DB is not enough to process reports on all nodes, the delayed job expects report files in spool/ of puppet-dashboard, since jobs are not assigned to nodes I found the only way to process reports is to share this directory across all nodes. The simplest way was SSHFS, also more secure that NFS.

On slave nodes:

Generate ssh keys for all slave nodes and add public part to /home/puppet-dashboard/.ssh/authorized_keys.

test ssh connection just to make sure you are able to connect.

Now install SSHFS (more info here)

add user puppet-dashboard to fuse system group.

To mount spool/ on system boot add following line to /etc/fstab:

sshfs#puppet-dashboard@dashboard-1.dns.com:/usr/local/puppet-dashboard/spool /usr/local/puppet-dashboard/spool fuse defaults,idmap=user,user 0 0

mount it as puppet-dashboard user:

Make sure /usr/local/puppet-dashboard/spool is mounted and can be accessed

Finally start all necessary processes:

On all nodes:

delayed jobs should be 1 per core

Final touches

To make master report to any of your dashboard nodes just use basic DNS round robin and point your master to it.

Finally set up cleaning cron jobs on the node running MySQL DB:

# Puppet dashboard cleanup
0 * * * * cd /usr/local/puppet-dashboard && /usr/bin/rake RAILS_ENV=production reports:prune upto=7 unit=day > /usr/local/puppet-dashboard/log/daily_cleanup_jobs.log 2>&1
13 0 * * * cd /usr/local/puppet-dashboard && /usr/bin/rake RAILS_ENV=production db:raw:optimize > /usr/local/puppet-dashboard/log/daily_optimize_jobs.log 2>&1

If i’m talking rubbish, or you have any ideas to improve this guide. Please leave a comment.