Setting up puppet for the first time

Working as a Sysadmin in a change intensive environment one - sooner or later - has to look into some sort of configuration management toolset. At work we have chosen to setup a system configuration backend based on the highly trendy puppet from puppetlabs. Puppet seems like a robust choice for a couple of different reasons.

It has a well matured community, a lot of documentation online and have been heavily tested by others.

All strong points when choosing to invest in a new tool. Reinventing the wheel can be fun some times, but keep falling into pitfalls because of a tool isn’t mature enough has high costs in a large organization. Puppetlabs seem to have ironed out the worst kinks and we felt confident in using their product and give it a thorough test run.

Treating configs as code

Using puppet’s manifests and modules result in treating system configuration as code and puts some pressure on the culture around storing those files. After some consideration we turned to git and are trying to implement a successful branching model for easily maintainable code. Using branches for module development and splitting up puppet in different environments are essential when investing so much in one tool that could potentially bring down critical applications and systems. Keeping a production, testing and development environment for different stages of modules are nothing new when releasing ordinary software and the concepts are already ingrained in most Sysadmins from customers with a focus on developing new code and releasing it.

I believe that this shift from using a scaffolding of different scripts by solo administrators to setting up a unifying configuration management tool with its configs in a software repository is the single most influential change for system administrators and their workflows. Cooperating and developing shared code as a team is the natural way forward. The whole “devops” movement and treating infrastructure as code leads to leaner system setups and could potentially save a lot of money, time and stress. Moving from the grumpy old system operators from hell to a leaner crowd of system operators who understands the hidden costs of manual work and quick fixes, are necessary for organizations where time and/or money becomes scarce. Automation, reusability and documentation should be essentially be parts of the culture we belong to and should be strived for in all the tools we choose to use as system administrators.

But enough ranting about “devops” and back to my first Puppet install.

Installing puppet on RedHat Enterprise Linux

Installing Puppet is as straightforward as it can get on RedHat Enterprise Linux. You can use the prepackaged rpm’s from EPEL for the core puppet install and rubygems for specific dependencies. Rubygems comes packaged for the most architectures and can also be installed with yum from EPEL.

At first I tried the rubygem-passenger from an old EPEL repo but that version seemed to come prepackaged with a nasty gcc-c++ bug that spit out compilation errors from the passenger-install-apache2-module command. After a couple of hours googling answers and talking to colleagues, I made the choice to just forget reading the existing documentation and just make it work. Keeping it as simple as possible and with the latest possible software that works.

So I tried installing the latest version of passenger, rack and dependencies with rubygems. I had to fetch them locally and install them from gem files on disk because I didn’t have internet access to rubygem’s repository from my firewalled server.

     [danieln@macbook ~]$ gem fetch daemon_controller
     [danieln@macbook ~]$ gem fetch passenger
     [danieln@macbook ~]$ gem fetch rack

     [root@puppetmaster01 ~]# gem install --local rack-1.4.1.gem 
     [root@puppetmaster01 ~]# gem install --local passenger-3.0.11.gem 
     [root@puppetmaster01 ~]# yum install -y curl-devel openssl-devel zlib-devel
     [root@puppetmaster01 ~]# /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11/bin/passenger-install-apache2-module 

After reinstalling rack and passenger and putting some standard configurations for rack in the right place everything worked as intended for the first time. Going the route of choosing a specific version of passenger and rack cost me half a day and some minor headache. I don’t know if I was unlucky or if the puppet wiki hasn’t been updated in a while, but the “Using Passenger” documentation wasn’t that good in my honest opinion. But at least it got me to document my steps and updating this sleeping blog!

To keep things as simple as possible I then removed all ssl certs, on both the puppet server and the agents, and rerun the puppet master process on the server to recreate new ssl certs. After creating new certs I had to put the new passenger app in apache’s conf and load the newly compiled module. A basic file structure for rack and a file also had to be copied from the install dir.

     [root@puppetmaster01 ~]# mkdir -p /etc/puppet/rack/public
     [root@puppetmaster01 ~]# mkdir -p /etc/puppet/rack/tmp
     [root@puppetmaster01 ~]# cp /usr/share/puppet/ext/rack/files/ /etc/puppet/rack

     [root@puppetmaster01 ~]# cat /etc/puppet/rack/ 
     # a, for use with every rack-compatible webserver.
     # SSL needs to be handled outside this, though.

     # if puppet is not in your RUBYLIB:
     # $:.unshift('/opt/puppet/lib')

     $0 = "master"

     # if you want debugging:
     # ARGV << "--debug"

     ARGV << "--rack"
     require 'puppet/application/master'
     # we're usually running inside a {} block,
     # therefore we need to call run *here*.
     run Puppet::Application[:master].run

     [root@puppetmaster01 ~]# cat /etc/httpd/conf.d/rack.conf 
     LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-                  3.0.11/ext/apache2/
     PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11
     PassengerRuby /usr/bin/ruby

     # you probably want to tune these settings
     PassengerHighPerformance on
     PassengerMaxPoolSize 12
     PassengerPoolIdleTime 1500
     # PassengerMaxRequests 1000
     PassengerStatThrottleRate 120
     RackAutoDetect Off
     RailsAutoDetect Off

     Listen 8140

     <VirtualHost *:8140>
             SSLEngine on
             SSLProtocol -ALL +SSLv3 +TLSv1
             SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

             SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
             SSLCertificateFile /var/lib/puppet/ssl/certs/
             SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
             SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
             SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem

             SSLVerifyClient optional
             SSLVerifyDepth  1
             SSLOptions +StdEnvVars

             DocumentRoot /etc/puppet/rack/public/
             RackBaseURI /
             <Directory /etc/puppet/rack/>
                     Options None
                     AllowOverride None
                     Order allow,deny
                     allow from all

After I installed Puppet and rubygems with yum from EPEL and got passenger and rack from rubygems repository everything finally worked as intended. All in all it took a couple of weeks to get puppet running for a complete beginner with some spare hours here and there. But I spend a lot of that time reading documentation and the excellent “Pro Pupet” book by James Turnbull and Jeffrey McCune. When everything was in place I started to write my own module for configuring the puppet server and started to configure the different environments.

I ended up with a simple layout with three environments, but I guess it will grow when the server gets more users.

         modulepath = $confdir/environments/$environment/modules
         manifestdir = $confdir/environments/production/manifests
         manifest = $manifestdir/site.pp

         manifest    = $confdir/environments/development/manifests/site.pp

         manifest    = $confdir/environments/testing/manifests/site.pp

So far I’m really impressed by puppet and it’s a fun product to work with. There’s a ton of great resources out there and the community is active and helpful.