From Fedora Project Wiki

No edit summary
 
(3 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{header|infra}}
{{header|infra}}
{{shortcut|ISOP:PUPPET}}


This page is intended for people who are already familiar with puppet.  It is specific to our Fedora Infrastructure though the techniques are general enough to apply to other environments.
{{admon/important|All SOPs have been moved to the Fedora Infrastructure [https://pagure.io/infra-docs/ SOP git repository]. Please consult the [https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/index.html online documentation] for the current version of this document.}}
 
== VCS/GIT ==
All of our configs and manifests are stored in GIT.  We also have a private GIT that stores keys, certificates and other files (like sudoers) that shouldn't be messed with or altered but by a select few people.
 
To make the GIT logs and commit notifications a bit more informative, it is a good idea to set your name and email in your <code>~/.gitconfig</code> file. The easiest way to do so is:
 
<pre>
git config --global user.name "Your Name Here"
git config --global user.email address@example.com
</pre>
 
The following commands will check out all GIT configs from a local shell on puppet01.phx2.fedoraproject.org:
 
<pre>
git clone /git/puppet
</pre>
 
Note that you should not copy or manipulate this repo except on the puppet01 machine.
 
There is a different procedure for cloning the private repo, as it
contains private data and should not be world-readable. This repo is restricted to a small group of people.
<pre>
RESTORE_UMASK=$(umask -p)
umask 0077
git clone /git/private
$RESTORE_UMASK
</pre>
 
or if you've got a checkout already:
<pre>
cd puppet
git pull
</pre>
 
You should pull changes before starting in on any changes yourself.
 
== Altering config files ==
The config files need to be edited, saved and installed.  The following example assumes everything is checked out already and is to alter a proxy file on the webserver.
 
<pre>
cd configs
vi web/fedoraproject.org/wiki.conf
git commit -a
</pre>
 
Once you have all of your changes in and ready run:
<pre>
git push
</pre>
 
Each puppet client pulls from the puppetmaster from a cron job that runs at 23 and 53 minutes past the hour +-10min. Output from puppet for each client is logged. When it changes it's mailed out, and once a month all outstanding errors from puppet runs are mailed out. There is also a job that checks to make sure all clients have checked in in the last hour, and emails an alert if they have not.
 
== Adding new files ==
Adding new files consists of two things, adding the file and adding the manifest.  Adding the config file is similar to the alteration section above.  Just add the file to the appropriate directory, git add filename.  Then commit and run 'git push'
 
Once the config file is in place move to your manifests directory and add the file to whichever manifest seems appropriate.  If none do, add a new manifest.  Most of our non-Fedora specific manifests exist in various puppet modules.  For manifests specific to our setup, there is a manifests directory with a servergroups directory grouping the services into logical machines.  The node file (in manifests/nodes) then includes those servergroups.
 
== Adding new node ==
Adding new hosts of an already known type is easy.  The following example is for a builder.  First change into the manifests/nodes directory.  Then add a new node file (see other node files for an example):
 
For example
 
<pre>
node "xenbuilder3.fedora.phx.redhat.com" {
  include build
}
</pre>
 
The "include build" line refers to a 'build' group in the servergroups directory.
 
=== Node Access ===
 
If this is a new node coming from a previously un-used IP address. We need to give puppet, infrastructure and the accounts system access to the node:
 
<pre>
git clone /git/puppet
cd puppet
vi modules/puppet/files/fileserver.conf
vi modules/infra-repo/files/allows
git commit -a
git push
</pre>
 
=== On the node ===
 
Do a manual run of puppet:
<pre>
puppetd -t --pluginsync --server=puppet
</pre>
 
This initial run will send a certificate request over to the puppet master.  once you have signed the new request (below) go back to the node and re-run the command.  It may be necessary to run puppet several times on the first run to make sure that all of the dependencies get worked out.
 
Note the additional pluginsync option, which sync custom facts that our
puppet configuration relies on.  This option not enabled by default on a
fresh puppet install which is why it is necesary on an initial run.
Once puppet installs our puppet configuration, it should not be
necessary to specify this on future runs.
 
=== Signing new ca request ===
If using one of our official kickstart scripts a server will automatically request a cert from puppet1.  To view a list just run (as root)
<pre>
puppetca --list
</pre>
 
followed by a sign of whatever hostname needs to be signed:
 
<pre>
puppetca --sign xenbuilder3.fedora.phx.redhat.com
</pre>
 
to remove an existing key from the puppet CA (for example when rebuilding a machine), run
<pre>
puppetca --revoke --clean xenbuilder3.fedora.phx.redhat.com
</pre>
 
first. Note that cleaning the key (puppetca --clean) and revoking the key (puppetca --revoke) are very different concepts.  A cleaned key can still be used to access puppet, while a revoked key cannot.
 
NOTE: --revoke --clean MUST BE PASSED IN THAT ORDER OR PUPPET IGNORES the --clean.
 
=== Finalizing ===
Sometimes a new node needs to have puppet restarted in order to connect correctly after its certificate has been approved.  Simply log in and run 'service puppet restart' as root.
 
=== Regenerate cert request ===
To regenerate the cert request (for example when re-kicking a box) first remove the old cert
<pre>
puppetca -c xenbuilderx.fedora.phx.redhat.com
</pre>
then restart puppet on the remote node.  Then sign the certificate as mentioned above with --sign
 
== Debugging ==
To debug puppet on a node (have it print verbosely) try this:
<pre>
sudo run-puppet nowait
</pre>
 
This command will generate a lot of output, so use carefully.
 
If you want to stop puppet from running while you mess with something then do:
 
<pre>
touch /var/lib/puppet/state/puppetdlock
</pre>
 
as root
 
remember to remove this file when you are done.
 
== Renaming a node ==
 
Sometimes you will need to rename a node due to Smooge deciding that a new naming convention is needed. The steps for doing so are the following :
 
# Log onto puppet01 and oldhostname in seperate windows.
# On puppet
## Edit associated DNS names in any places (.pp files etc) referring to the old host name in puppet
## <pre> sudo /usr/sbin/puppetca --revoke --clean ${OLDHOSTNAME}</pre>
# On oldhostname
## edit <b>/etc/sysconfig/network</b>,<b>/etc/hosts</b> and any other files not controlled by puppet.
## remove old cert files. <pre> find /var/lib/puppet/ssl -type f -print | xargs rm -v</pre>
## create new cert files. <pre> /usr/sbin/puppetd --enable ; /usr/sbin/puppetd -t</pre>
# On puppet
## sign new cert files <pre>sudo /usr/sbin/puppetd --list; sudo /usr/sbin/puppetd --sign ${NEWHOSTNAME}</pre>
# On oldhostname
## <pre>/usr/sbin/puppetd -t</pre> and begin looking for any problems.


[[Category:Infrastructure SOPs]]
[[Category:Infrastructure SOPs]]

Latest revision as of 12:00, 16 February 2017


Important.png
All SOPs have been moved to the Fedora Infrastructure SOP git repository. Please consult the online documentation for the current version of this document.