Configuration Mgmt using Puppet (2008-11-09 classroom)

Fedora Classroom - Configuration Management using Puppet - Jeroen van Meeuwen - Sunday, November 9, 2008

IRC Log of the Class

kanarip	so this session is about configuration management, which does *not* include the soft side of configuration management for a change	23:04
*Bugz_ is		23:04
kanarip	we're *not* going to talk about procedural issues, but keep it on the technical side	23:04
kanarip	i'd like to invite you all to take a look at http://www.kanarip.com/courses/puppet/puppet.pdf	23:04
kanarip	which is a nice reader i wrote for a one-day workshop i'm giving on this topic, and that is commercially available	23:04
kanarip	community people like us however *do not have to pay*	23:04
kanarip	sounds great, huh? ;-)	23:04
domg472_	yes	23:04
erinlea80	:)	23:04
delhage	from Delft eh?	23:04
kanarip	so, a little something on configuration management first, then a sneak peak at the installation of puppet, and a quick howto on writing your "manifest" as they call the puppet configuration syntax	23:04
kanarip	delhage, Utrecht	23:04
delhage	ok	23:04
kanarip	so, what is it we would want to do with configuration management in the technical sense of the words	23:04
kanarip	we all know that the more systems we need to maintain, the harder it becomes to keep some extend of consistency between different systems	23:04
kanarip	some organizations use scripts from an NFS share, or use an SVN tree with scripts periodically updated and execute scripts	23:05
kanarip	does that sound familiar to anyone?	23:05
*erinlea80 nods		23:05
kanarip	so what happens in these cases is:	23:05
kanarip	1) your scripts tend to get very large	23:05
kanarip	describing every unique situation, every exemption, every update, for every operating system, distribution, distribution version, makes the script less maintainable	23:05
kanarip	2) your scripts are executed at arbitrary intervals	23:06
kanarip	either a cronjob, on boot, may or may not fail, and you cannot make the changes you want pronto	23:06
kanarip	3) your scripts replace configuration and reconfigure stuff without actual changes being applied	23:06
kanarip	either of these 3 things cries out for a configuration management like...	23:07
kanarip	Puppet (or CFEngine)	23:07
kanarip	anyone heard of CFEngine before?	23:07
delhage	yup	23:07
jds2001	however cfengine is fail	23:07
jds2001	we use it :)	23:07
kanarip	jds2001, you're in the right spot ;-)	23:07
kanarip	what CFEngine does is very accurately describe the configuration state a system has to be in	23:08
kanarip	it does it so accurately, in fact, that it ends up being a very low-level tool	23:08
kanarip	to give you an example;	23:08
kanarip	in CFEngine, to install a package called foo, you will have to manually assign a package provider to each operating system, distribution or distribution version, and then use that provider to install the package	23:09
kanarip	in addition, you would need to change the name for each OS, distro and distribution version	23:09
kanarip	lots to type, lots to care about, while in fact all you care about is that the package is installed	23:10
*kanarip introduces... the high-level abstraction of Puppet		23:10
kanarip	no matter what the operating system, distribution or distribution may be, Puppet speaks the local language	23:10
kanarip	it'll figure out on it's own whether to use yum, up2date, dpkg, apt, PackageKit, alien, smart, rpm or what-package-manager-have-you	23:11
kanarip	this way, ensuring that package foo is installed on a system becomes:	23:11
kanarip	package { "foo":	23:11
kanarip	ensure => installed	23:11
kanarip	}	23:11
kanarip	and you're done	23:11
kanarip	jds2001, can you confirm that is easier then CFEngine please? ;-)	23:11
jds2001	yes :)	23:12
kanarip	thanks ;-)	23:12
kanarip	so, how does puppet work?	23:12
kanarip	given these abstracted "system resources" such as packages, services, cronjobs, users, ..., this and that and more, puppet uses so-called "types" you can use in a configuration "manifest" to describe what state a managed system should be in	23:13
kanarip	here's a list of types: http://reductivelabs.com/trac/puppet/wiki/TypeReference	23:13
kanarip	now let me describe to you what a working setup looks like;	23:14
kanarip	there is a single server, called the puppetmaster	23:15
kanarip	personally i think the puppeteer would have been a more appropriate name, but puppetmaster it is	23:15
kanarip	the managed system is called a "puppet" - whos strings you can pull using the puppetmaster	23:15
kanarip	the puppet, or managed system, polls the puppetmaster at a default interval of every 30 minutes	23:16
kanarip	the puppetmaster, having *all* the configuration for *all* nodes in an environment, will parse *all* it's manifests, and send to the puppet a set of resources to be managed on that specific node	23:17
nirik	Q: do they all pull at once? or is it staggered?	23:17
kanarip	nirik, it depends on the time at which the puppet starts counting	23:17
kanarip	it has a default 1 minute randomization i believe	23:17
kanarip	but it'll poll at least once every 30.9 minutes, if that makes sense ;-)	23:18
domg472_	why does it do that?	23:18
kanarip	to check if there's any changes to the configuration for the node	23:19
domg472_	if yo have changes on the master can you just tell it to push them?	23:19
domg472_	ok thanks	23:19
kanarip	if you change anything, you let the master pick it up and at most 59 later the puppet will have picked up on the changes	23:20
kanarip	that is, if you are using the default interval of 30 minutes	23:20
kanarip	sorry, should have said: at most 30.9 minutes after you apply changes, puppet will have picked up the changes	23:21
kanarip	so here's how the puppetmaster distinguishes configuration to be applied to one node: http://git.puppetmanaged.org/?p=domain-kanarip.com;a=blob;f=puppet/manifests/nodes/anakin.kanarip.com.pp	23:21
daMaestro	is there any way to push changes? or is it all pull operations?	23:21
daMaestro	(in the case of needing an isolated DMZ/VLAN)	23:21
kanarip	the puppetmaster will load that file and know that anakin.kanarip.com which is an actual system is to be configured accordingly	23:22
neverho0d	is there way to stop propagating changes?	23:22
domg472_	evry 30 minutes sounds like overhead	23:22
kanarip	daMaestro, puppet provides a utility called puppetrun which tells the client or clients to do a puppet run	23:22
daMaestro	kanarip, yet this is still a pull?	23:22
kanarip	neverho0d, yes;	23:23
kanarip	1) stage your changes in environments	23:23
neverho0d	kanarip: ok	23:23
kanarip	2) stop the puppetmaster	23:23
kanarip	3) stop the puppet client daemon	23:23
*daMaestro attempts to not interrupt anymore		23:23
kanarip	4) let the puppetmaster not pull from a SCM to load new configuration	23:23
kanarip	daMaestro, the puppet client will perform a pull from the puppetmaster, yes	23:23
daMaestro	k	23:24
kanarip	any other questions so far?	23:24
kanarip	i'd like to dive into modules, if that's ok?	23:24
erinlea80	any utilities to automate data/config collections on pre-existing servers?	23:24
erinlea80	or is it all by hand?	23:24
kanarip	erinlea80, it's very complex to harvest data and then roll it out but i guess a certain scale justifies putting in such effort	23:25
kanarip	the harvesting would only happen once is what i'm concerned with	23:25
erinlea80	agreed -- was wondering what the options are.	23:26
kanarip	so here's a sample module for puppet: http://git.puppetmanaged.org/?p=sudo;a=tree	23:26
kanarip	a module is a collection of the manifest: manifests/init.pp, the files used by that module (in files/), any templates used by the module (in templates/), and more	23:26
kanarip	http://reductivelabs.com/trac/puppet/wiki/ModuleOrganisation for a reference	23:27
kanarip	and of course there's something mentioned in http://www.kanarip.com/courses/puppet/puppet.pdf as well	23:27
kanarip	what makes modules so great is a couple of things;	23:27
kanarip	1) i can share it with you and you can just start using it	23:28
kanarip	2) you can customize it and still pull changes from an upstream SCM like the GIT repository I just pointed you to and send me/upstream patches	23:28
kanarip	3) modules allow "staging"	23:28
kanarip	i guess the next topic is "staging" ;-)	23:29
kanarip	4) modules keep your files, manifests, templates and plugins organized	23:29
kanarip	believe me when I say puppet configuration trees can become a mess if you start managing more and more with puppet, and not use modules	23:29
kanarip	== Staging ==	23:29
kanarip	What I mean by staging is simply this; stage your changes from a development environment, possibly via a testing environment onto your production environment	23:30
kanarip	you can develop all you want in your development environment, make syntax errors, rm -rf / sorta speak, and never nuke that business critical application server you have running	23:31
kanarip	then once you're satisfied, you may move the changes into the testing or production environment	23:32
kanarip	does that make sense?	23:32
nuonguy	oh yeah	23:32
neverho0d	sure	23:32
jds2001	yes, but how does one do this in practice?	23:32
kanarip	awesome	23:32
erinlea80	mmmm yes	23:32
erinlea80	and does it require a unique environment for each corresponding production environment?	23:32
kanarip	i can recommend anyone getting his feet wet in using environments	23:32
*nirik has some questions, but can save them for the end as well.		23:32
erinlea80	unique test environ.	23:32
kanarip	jds2001, here's a practical example;	23:32
kanarip	all the repositories you see on http://git.puppetmanaged.org/ have a development, a testing, and a production branch	23:33
kanarip	what i do when i'm satisfied with my developments, is I change the working branch to testing, and git pull development	23:34
kanarip	then when i push, the puppetmaster runs a puppet that is configured so that it will pull the changes from the GIT repo	23:34
jds2001	then you have post-receive hooks to put them someplace?	23:34
kanarip	http://git.puppetmanaged.org/?p=domain-puppetmanaged.org;a=blob;f=puppet/manifests/nodes/master.puppetmanaged.org.pp as a reference	23:35
kanarip	jds2001, no, the puppetmaster in my case runs a puppet that has ^^ configuration applied to itself	23:35
kanarip	a post-hook making the puppetmaster pull in the changes is a viable solution too; the Fedora Project does such	23:36
kanarip	however, in my situation, the puppetmaster and the SCM are not on the same machine	23:36
jds2001	ok, makes sense	23:37
erinlea80	kk	23:37
kanarip	http://www.kanarip.com/courses/puppet/puppet.odp is a presentation slide-deck BTW, that goes along with the .pdf i've already linked	23:37
kanarip	note that what you see on puppetmanaged.org is heavily dependent on the other modules available from puppetmanaged.org	23:37
kanarip	so... what do you think about setting it up?	23:38
erinlea80	lets do it?	23:39
kanarip	there you go ;-)	23:39
kanarip	let me refer to the documentation i shared earlier because it's a lot to type	23:39
kanarip	http://www.kanarip.com/courses/PuppetWorkshop-SettingUpPuppet.html	23:39
kanarip	it's a yum install puppet-server to install it on any Fedora system	23:40
kanarip	for Red Hat and CentOS, you'll need EPEL configured	23:40
kanarip	http://fedoraproject.org/wiki/EPEL	23:40
*erinlea80 nods		23:40
kanarip	for anyone who's wondering about what EPEL is... I maintain or co-maintain the entire Ruby stack to puppet in that repository ;-)	23:41
kanarip	and that brings me to one more nifty feature of puppet;	23:41
kanarip	but first... facts!	23:42
kanarip	facts are... well... facts...	23:42
kanarip	little information about the current state of a client	23:42
kanarip	it includes the kernel version, number of network interfaces, the architecture, operating system name, operating system version, and lots more	23:43
kanarip	yum install facter and run "facter" on your machine to check it out	23:43
kanarip	these are all "variables" you can use in your puppet manifests	23:43
kanarip	given a small amount of RAM for example, you may want to apply a snippet similar to: package { "firefox": ensure => absent }	23:44
kanarip	anyway, what i really wanted to say rather then to give a rant on firefox's memory consumption is this;	23:44
kanarip	facts, types and providers, these things that make puppet do smart things...	23:44
kanarip	facts for information about the client...	23:45
erinlea80	do we have a sample facter output?	23:45
kanarip	i could fpaste you some	23:45
erinlea80	that would be great :)	23:45
zcat	kanarip, is facter supposed to spit out "virtual => vmware_server" if detected?	23:45
domg472_	creating new ones doesnt look that easy	23:45
kanarip	http://fpaste.org/paste/15	23:46
domg472_	but theres good documentation on create types fact etc	23:46
kanarip	types to define resources to be managed on the puppet...	23:46
erinlea80	thanks	23:47
kanarip	and providers to make puppet speak the local language...	23:47
kanarip	the great thing is... you can write your own for all of these	23:47
kanarip	it does require ruby knowledge, but then again... there's sample recipes on reductivelabs.com for...for example zcat, a vmware custom fact ;-)	23:47
kanarip	see, i was getting there ;-)	23:47
kanarip	actually i think the custom fact is called "this is a vmware machine" and "whether it has vm-tools installed"	23:48
kanarip	one more thing i really, really need to stress out to you	23:48
kanarip	all communication between the puppet and it's puppetmaster is *secure*	23:49
*nirik has one of his questions answered. ;)		23:49
kanarip	that being said, the security is based on the puppetmaster running the Certificate Authority, the client generating a certificate and putting in a request with the puppetmaster to get a signed copy of that certificate	23:50
erinlea80	can communication be tunneled over ssh if necessary? (firewall restrictions, etc.)	23:50
domg472_	but the puppets have root access obviously?	23:50
domg472_	and the master	23:50
kanarip	erinlea80, you can make the puppetmaster run on any port you want	23:50
kanarip	erinlea80, may i point out "autossh" with which you can create tunnels to communicate over if you wish	23:50
erinlea80	ty :)	23:50
kanarip	domg472_, the puppet client daemon runs with root privileges, yes	23:51
domg472_	has anyone tested this with selinux?	23:51
kanarip	domg472_, the master runs with under the puppet user	23:51
domg472_	ok thanks	23:51
kanarip	domg472_, i have, and i can tell you it requires a custom selinux policy in which you basically describe the puppet daemon can do this and that and more	23:52
kanarip	in my case, the puppet daemon can do anything it likes	23:52
domg472_	youd need custom modules i bet	23:52
domg472_	interesting	23:52
sdodson	There are also a few minor bugs which I think have been fixed recently where puppet creates files without the proper contexts.	23:53
kanarip	selinux is being integrated better into puppet as we speak	23:53
domg472_	great	23:53
kanarip	it's a big issue, i concur	23:53
nuonguy	Q: if I enable, for example httpd in a manifest, and 2 hours later comment it out, does the puppet uninstall/disable httpd or ignore it from then on?	23:53
kanarip	nuonguy, puppet only applies changes	23:54
kanarip	nuonguy, so it only enforces what you describe needs to happen	23:54
kanarip	if you say package httpd should be installed at one point	23:54
nuonguy	so I'd have to explicity turn it off, if that's what I meant?	23:54
kanarip	and then comment that out... no other machine is going to care whether the package should be installed, but it's not going to be removed on puppet's behalf	23:55
kanarip	nuonguy, yes	23:55
nuonguy	I dig, thanks	23:55
kanarip	nuonguy, same goes for the service type; ensure => running, enable => true would need to become ensure => stopped, enable => false	23:56
nuonguy	precisely	23:56
nuonguy	I can just see myself not remembering that...	23:56
kanarip	so, another thing on the topic; what do you think happens if the puppet daemon cannot contact the puppetmaster?	23:56
nuonguy	and commenting it out instead	23:56
nirik	does it do them in order in the manifest? ie, make sure to stop before ensure	23:56
kanarip	nuonguy, there's some discussion going on about supporting an "exclude" statement like we do "include webserver" now	23:57
nuonguy	nice	23:57
kanarip	nuonguy, and then flipping all the false's into true's and so forth	23:57
kanarip	nirik, good point!	23:57
sdodson	Could create a baseline that ensures none of that is installed/running then include webserver for only your webservers which would override those baselines?	23:57
kanarip	puppet allows ordering the resources;	23:57
kanarip	let me look you up an example	23:57
domg472_	why was ruby chosen as the language for writing types etc?	23:58
domg472_	why not bash?	23:59
kanarip	here's a yp manifest: http://fpaste.org/paste/16	23:59
kanarip	domg472_, i'm not sure i'm not one of the core developers, but i can try and answer it if you promise not to quote me on it ;-)	00:00
domg472_	hah ok	00:00
domg472_	i think most sysadmins know bash	00:00
kanarip	nirik, you can see how "require => (...)" in that yp manifest causes one resource to be dependent on another	00:00
domg472_	and they will need to write custom types probably	00:00
kanarip	domg472_, i think bash is what you use to do a quick thing, clean and simple. I think ruby allows a framework to be built, reusing snippets of code you've already written in a different place. I think ruby is more modular then is bash	00:01
domg472_	ok thanks	00:02
kanarip	you could write a website in bash but you won't, right?	00:02
domg472_	i guess ill be learning ruby then	00:02
domg472_	true	00:02
domg472_	but these are simplate admin tasks	00:02
domg472_	simple	00:02
kanarip	not saying you should choose php to do it BTW ;-)	00:02
*nirik notes we are over time, but this is also the last class today.		00:02
kanarip	in puppet terms; distribute the bash scripts amongst the clients that need it using the file type, and exec that script	00:03
nirik	I had one last question: how do you decide what should be in a module? whats the logical boundry there?	00:03
kanarip	anything you;	00:03
domg472_	ah right, thanks kanarip	00:03
kanarip	1) pull from an upstream source	00:03
kanarip	2) need to share (with the general public)	00:04
kanarip	3) want to stage	00:04
nirik	so it doesn't need to be a specific service or anything? just whatever you want to put in one?	00:04
kanarip	4) that is a concise set of files, manifests, etc.	00:04
sdodson	They're advising people to get away from writing classes and move towards modules.	00:04
kanarip	nirik, you're almost right, a module called everything-else doesn't make sense ;-)	00:05
kanarip	sdodson, yes, very much so	00:05
kanarip	nirik, a module webserver makes sense; keeping it close and concise within a module is way easier then dispersing it all over the place	00:05
nirik	ok, makes sense.	00:06
tmz	domg472_: regarding why ruby was used to write puppet, here's a short interview with Luke Kanies (primary puppet author), where he touches on that choice:	00:06
tmz	http://on-ruby.blogspot.com/2008/02/puppet-interview-with-luke-kanies.html	00:06
domg472_	thanks	00:06
kanarip	what i tend to recommend to customers is also, to keep the systems they manage as consistent as possible	00:06
kanarip	being able to determine the exceptions to the rule you've layed down saves you a lot of time in determining what the cause might be for a specific issue	00:07
nirik	shall we wrap up and let kanarip get some sleep?	00:08
kanarip	;-)	00:08
daMaestro	no, no sleep for kanarip !	00:08
kanarip	thank you all for your interest, and thank you for attending	00:08
*daMaestro /dcc kanarip latte		00:08
jds2001	he must stay up all night!	00:08
nirik	thanks so much for the session kanarip !	00:08
erinlea80	thanks kanarip!	00:08
domg472_	thanks again	00:08
erinlea80	:)	00:08
iondrip	thanks	00:08
kanarip	i hope you find a lot of use in the pdf/html/odp i've given you ;-)	00:08
neverho0d	kanarip: thanks	00:08
nirik	thanks everyone for coming...	00:09
Bugz_	kanarip: Thanks	00:09
erinlea80	:)	00:09
domg472_	thanks for hosting	00:09
SSlater	Thanks	00:09

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!