By Luke Tymowski, Systems Administrator, Calgary
One tool we've adopted to help us is a configuration change management tool called Puppet. The Universite de Sherbrooke's HPC team, one of our partners in the DAIR project, uses Puppet to help administer their HPC installations. Normally if you want to install a new tool (say NTP) or change a configuration file (say /etc/ntp.conf), you log into your server, do the work, then log out. If you've got dozens, hundreds, or thousands of servers, it can take a very long time, and the process can be error-prone. You might forget to restart the NTP service on one server, or you might forget about a server or three.
Puppet allows you to configure those upgrades on a central server. Then every 30 minutes the Puppet client (installed on each node), asks the server for updates, then installs them. Work that might otherwise have taken hours, days, or weeks can be done in a few seconds. As Jean-Francois Landry, a Universite de Sherbrooke System Administrator, said after enabling a new service through Puppet: "Now watch the magic."
CFengine started this movement. Puppet tried to improve upon CFengine. And Chef tries to improve upon Puppet. But in the last year or so Puppet have resolved some of the issues that inspired the creation of Chef.
There is, to me, no simple argument that proves one or the other as the superior tool, though a lot of people would argue that. An overly simplified explanation of their differences would be: with Puppet you use Puppet's own DSL to write recipes, but with Chef you use straight Ruby (both Puppet and Chef are written in Ruby). Andrew Clay Shafer describes in more detail, but not too much, the differences between the two.
Puppet also has MCollective, which I see as a very powerful tool that is especially useful for cloud-based work.
So why did we choose to use Puppet and not Chef? The Universite de Sherbrooke already use Puppet to manage their HPC servers, and they used Puppet, in part, to roll out their DAIR stack. They already have expertise with a tool that none of us have. So Puppet it is.
Puppet Labs, the startup behind Puppet, are making an effort to improve their HCI score. Would that all software projects would do something similar! If you aren't already using a tool like CFengine, Puppet, or Chef, pick one, and get started.