From Fedora Project Wiki

Revision as of 01:20, 10 November 2008 by Dp67 (talk | contribs) (→‎IRC Log of the Class: added session log)

Fedora Classroom - Configuration Management using Puppet - Jeroen van Meeuwen - Sunday, November 9, 2008

IRC Log of the Class

 Nov 09 18:04:12 <kanarip>	so this session is about configuration management, which does *not*
 include the soft side of configuration management for a change
 Nov 09 18:04:12 *	Bugz_ is
 Nov 09 18:04:12 <kanarip>	we're *not* going to talk about procedural issues, but keep it on the
 technical side
 Nov 09 18:04:12 <kanarip>	i'd like to invite you all to take a look at
 Nov 09 18:04:12 <kanarip>	which is a nice reader i wrote for a one-day workshop i'm giving on this topic,
  and that is commercially available
 Nov 09 18:04:12 <kanarip>	community people like us however *do not have to pay*
 Nov 09 18:04:12 <kanarip>	sounds great, huh? ;-)
 Nov 09 18:04:12 <domg472_>	yes
 Nov 09 18:04:12 <erinlea80>	:)
 Nov 09 18:04:12 <delhage>	from Delft eh?
 Nov 09 18:04:12 <kanarip>	so, a little something on configuration management first, then a sneak peak
 at the installation  of puppet, and a quick howto on writing your "manifest" as they call the puppet
 configuration syntax
 Nov 09 18:04:12 <kanarip>	delhage, Utrecht
 Nov 09 18:04:12 <delhage>	ok
 Nov 09 18:04:12 <kanarip>	so, what is it we would want to do with configuration management in the technical
 sense of the words
 Nov 09 18:04:12 <kanarip>	we all know that the more systems we need to maintain, the harder it becomes to
 keep some extend of consistency between different systems
 Nov 09 18:05:14 <kanarip>	some organizations use scripts from an NFS share, or use an SVN tree with scripts
 that is periodically updated and execute scripts
 Nov 09 18:05:14 <kanarip>	does that sound familiar to anyone?
 Nov 09 18:05:14 *	erinlea80 nods
 Nov 09 18:05:14 <kanarip>	so what happens in these cases is:
 Nov 09 18:05:14 <kanarip>	1) your scripts tend to get very large
 Nov 09 18:05:50 <kanarip>	describing every unique situation, every exemption, every update, for every
 operating system, distribution, distribution version, makes the script less maintainable
 Nov 09 18:06:10 <kanarip>	2) your scripts are executed at arbitrary intervals
 Nov 09 18:06:34 <kanarip>	either a cronjob, on boot, may or may not fail, and you cannot make the changes
 you want pronto
 Nov 09 18:06:58 <kanarip>	3) your scripts replace configuration and reconfigure stuff without actual changes
 being applied
 Nov 09 18:07:18 <kanarip>	either of these 3 things cries out for a configuration management like...
 Nov 09 18:07:26 <kanarip>	Puppet (or CFEngine)
 Nov 09 18:07:32 <kanarip>	anyone heard of CFEngine before?
 Nov 09 18:07:37 <delhage>	yup
 Nov 09 18:07:39 <jds2001>	however cfengine is fail
 Nov 09 18:07:43 <jds2001>	we use it :)
 Nov 09 18:07:51 <kanarip>	jds2001, you're in the right spot ;-)
 Nov 09 18:08:09 <kanarip>	what CFEngine does is very accurately describe the configuration state a system
 has to be in
 Nov 09 18:08:32 <kanarip>	it does it so accurately, in fact, that it ends up being a very low-level tool
 Nov 09 18:08:36 <kanarip>	to give you an example;
 Nov 09 18:09:22 <kanarip>	in CFEngine, to install a package called foo, you will have to manually assign a
 package provider to each operating system, distribution or distribution version, and then use that provider to
 install the package
 Nov 09 18:09:43 <kanarip>	in addition, you would need to change the name for each OS, distro and
 distribution version
 Nov 09 18:10:01 <kanarip>	lots to type, lots to care about, while in fact all you care about is that the
 package is installed
 Nov 09 18:10:18 *	kanarip introduces... the high-level abstraction of Puppet
 Nov 09 18:10:36 <kanarip>	no matter what the operating system, distribution or distribution may be, Puppet
 speaks the local language
 Nov 09 18:11:13 <kanarip>	it'll figure out on it's own whether to use yum, up2date, dpkg, apt, PackageKit,
 alien, smart, rpm or what-package-manager-have-you
 Nov 09 18:11:30 <kanarip>	this way, ensuring that package foo is installed on a system becomes:
 Nov 09 18:11:35 <kanarip>	package { "foo":
 Nov 09 18:11:40 <kanarip>	    ensure => installed
 Nov 09 18:11:41 <kanarip>	}
 Nov 09 18:11:44 <kanarip>	and you're done
 Nov 09 18:11:58 <kanarip>	jds2001, can you confirm that is easier then CFEngine please? ;-)
 Nov 09 18:12:06 <jds2001>	yes :)
 Nov 09 18:12:11 <kanarip>	thanks ;-)
 Nov 09 18:12:19 <kanarip>	so, how does puppet work?
 Nov 09 18:13:15 <kanarip>	given these abstracted "system resources" such as packages, services, cronjobs,
 users, ..., this and that and more, puppet uses so-called "types" you can use in a configuration "manifest"
 to describe what state a managed system should be in 
 Nov 09 18:13:29 <kanarip>	here's a list of types:
 Nov 09 18:14:56 <kanarip>	now let me describe to you what a working setup looks like;
 Nov 09 18:15:15 <kanarip>	there is a single server, called the puppetmaster
 Nov 09 18:15:34 <kanarip>	personally i think the puppeteer would have been a more appropriate name, but
 puppetmaster it is
 Nov 09 18:15:56 <kanarip>	the managed system is called a "puppet" - whos strings you can pull using the
 Nov 09 18:16:20 <kanarip>	the puppet, or managed system, polls the puppetmaster at a default interval of
 every 30 minutes
 Nov 09 18:17:08 <kanarip>	the puppetmaster, having *all* the configuration for *all* nodes in an environment,
 will parse
 *all* it's manifests, and send to the puppet a set of resources to be managed on that specific node
 Nov 09 18:17:09 <nirik>	Q: do they all pull at once? or is it staggered?
 Nov 09 18:17:34 <kanarip>	nirik, it depends on the time at which the puppet starts counting
 Nov 09 18:17:56 <kanarip>	it has a default 1 minute randomization i believe
 Nov 09 18:18:34 <kanarip>	but it'll poll at least once every 30.9 minutes, if that makes sense ;-)
 Nov 09 18:18:52 <domg472_>	why does it do that?
 Nov 09 18:19:21 <kanarip>	to check if there's any changes to the configuration for the node
 Nov 09 18:19:21 <domg472_>	if yo have changes on the master can you just tell it to push them?
 Nov 09 18:19:40 <domg472_>	ok thanks
 Nov 09 18:20:38 <kanarip>	if you change anything, you let the master pick it up and at most 59
 minutes later the puppet will have picked up on the changes
 Nov 09 18:20:51 <kanarip>	that is, if you are using the default interval of 30 minutes
 Nov 09 18:21:10 <kanarip>	sorry, should have said: at most 30.9 minutes after you apply changes,
 the puppet will have picked up the changes
 Nov 09 18:21:33 <kanarip>	so here's how the puppetmaster distinguishes configuration to be applied to one
 Nov 09 18:21:46 <daMaestro>	is there any way to push changes? or is it all pull operations?
 Nov 09 18:21:56 <daMaestro>	(in the case of needing an isolated DMZ/VLAN)
 Nov 09 18:22:09 <kanarip>	the puppetmaster will load that file and know that which is an
 actual system is to be configured accordingly
 Nov 09 18:22:32 <neverho0d>	is there way to stop propagating changes?
 Nov 09 18:22:40 <domg472_>	evry 30 minutes sounds like overhead
 Nov 09 18:22:41 <kanarip>	daMaestro, puppet provides a utility called puppetrun which tells the client or
 clients to do a puppet run
 Nov 09 18:22:59 <daMaestro>	kanarip, yet this is still a pull?
 Nov 09 18:23:02 <kanarip>	neverho0d, yes;
 Nov 09 18:23:13 <kanarip>	1) stage your changes in environments
 Nov 09 18:23:14 <neverho0d>	kanarip: ok
 Nov 09 18:23:17 <kanarip>	2) stop the puppetmaster
 Nov 09 18:23:23 <kanarip>	3) stop the puppet client daemon
 Nov 09 18:23:30 *	daMaestro attempts to not interrupt anymore
 Nov 09 18:23:37 <kanarip>	4) let the puppetmaster not pull from a SCM to load new configuration
 Nov 09 18:23:52 <kanarip>	daMaestro, the puppet client will perform a pull from the puppetmaster, yes
 Nov 09 18:24:03 <daMaestro>	k
 Nov 09 18:24:28 <kanarip>	any other questions so far?
 Nov 09 18:24:38 <kanarip>	i'd like to dive into modules, if that's ok?
 Nov 09 18:24:45 <erinlea80>	any utilities to automate data/config collections on pre-existing servers?
 Nov 09 18:24:53 <erinlea80>	or is it all by hand?
 Nov 09 18:25:38 <kanarip>	erinlea80, it's very complex to harvest data and then roll it out but i guess a
 certain scale justifies putting in such effort
 Nov 09 18:25:56 <kanarip>	the harvesting would only happen once is what i'm concerned with
 Nov 09 18:26:00 <erinlea80>	agreed -- was wondering what the options are.
 Nov 09 18:26:13 <kanarip>	so here's a sample module for puppet:;a=tree
 Nov 09 18:26:52 <kanarip>	a module is a collection of the manifest: manifests/init.pp, the files used by
 that module (in files/), any templates used by the module (in templates/), and more
 Nov 09 18:27:06 <kanarip> for a reference
 Nov 09 18:27:25 <kanarip>	and of course there's something mentioned in as well
 Nov 09 18:27:55 <kanarip>	what makes modules so great is a couple of things;
 Nov 09 18:28:13 <kanarip>	1) i can share it with you and you can just start using it
 Nov 09 18:28:45 <kanarip>	2) you can customize it and still pull changes from an upstream SCM like the GIT
 repository I just pointed you to and send me/upstream patches
 Nov 09 18:28:55 <kanarip>	3) modules allow "staging"
 Nov 09 18:29:03 <kanarip>	i guess the next topic is "staging" ;-)
 Nov 09 18:29:25 <kanarip>	4) modules keep your files, manifests, templates and plugins organized
 Nov 09 18:29:52 <kanarip>	believe me when I say puppet configuration trees can become a mess if you start
 managing more and more with puppet, and not use modules
 Nov 09 18:29:58 <kanarip>	== Staging ==
 Nov 09 18:30:38 <kanarip>	What I mean by staging is simply this; stage your changes from a development
 environment, possibly via a testing environment onto your production environment
 Nov 09 18:31:26 <kanarip>	you can develop all you want in your development environment, make syntax errors,
 rm -rf / sorta speak, and never nuke that business critical application server you have running
 Nov 09 18:32:08 <kanarip>	then once you're satisfied, you may move the changes into the testing or
 production environment
 Nov 09 18:32:12 <kanarip>	does that make sense?
 Nov 09 18:32:21 <nuonguy>	oh yeah
 Nov 09 18:32:24 <neverho0d>	sure
 Nov 09 18:32:24 <jds2001>	yes, but how does one do this in practice?
 Nov 09 18:32:26 <kanarip>	awesome
 Nov 09 18:32:26 <erinlea80>	mmmm yes
 Nov 09 18:32:42 <erinlea80>	and does it require a unique environment for each corresponding production
 Nov 09 18:32:43 <kanarip>	i can recommend anyone getting his feet wet in using environments
 Nov 09 18:32:48 *	nirik has some questions, but can save them for the end as well. 
 Nov 09 18:32:49 <erinlea80>	unique test environ.
 Nov 09 18:32:52 <kanarip>	jds2001, here's a practical example;
 Nov 09 18:33:27 <kanarip>	all the repositories you see on have a development,
 a testing, and a production branch
 Nov 09 18:34:10 <kanarip>	what i do when i'm satisfied with my developments, is I change the working branch
 to testing, and git pull development
 Nov 09 18:34:40 <kanarip>	then when i push, the puppetmaster runs a puppet that is configured so that it
 will pull the changes from the GIT repo
 Nov 09 18:34:57 <jds2001>	then you have post-receive hooks to put them someplace?
 Nov 09 18:35:05 <kanarip>;a=blob;f=puppet/manifests/nodes/
 as a reference
 Nov 09 18:35:28 <kanarip>	jds2001, no, the puppetmaster in my case runs a puppet that has ^^ configuration
 applied to itself
 Nov 09 18:36:10 <kanarip>	a post-hook making the puppetmaster pull in the changes is a viable solution too;
 the Fedora Project does such
 Nov 09 18:36:25 <kanarip>	however, in my situation, the puppetmaster and the SCM are not on the same machine
 Nov 09 18:37:08 <jds2001>	ok, makes sense
 Nov 09 18:37:12 <erinlea80>	kk
 Nov 09 18:37:14 <kanarip> is a presentation slide-deck BTW,
 that goes along with the .pdf i've already linked
 Nov 09 18:37:38 <kanarip>	note that what you see on is heavily dependent on the other
 modules available from
 Nov 09 18:38:43 <kanarip>	so... what do you think about setting it up?
 Nov 09 18:39:15 <erinlea80>	lets do it?
 Nov 09 18:39:24 <kanarip>	there you go ;-)
 Nov 09 18:39:37 <kanarip>	let me refer to the documentation i shared earlier because it's a lot to type
 Nov 09 18:39:49 <kanarip>
 Nov 09 18:40:21 <kanarip>	it's a yum install puppet-server to install it on any Fedora system
 Nov 09 18:40:36 <kanarip>	for Red Hat and CentOS, you'll need EPEL configured
 Nov 09 18:40:43 <kanarip>
 Nov 09 18:40:56 *	erinlea80 nods
 Nov 09 18:41:26 <kanarip>	for anyone who's wondering about what EPEL is... I maintain or co-maintain the
 entire Ruby stack to puppet in that repository ;-)
 Nov 09 18:41:49 <kanarip>	and that brings me to one more nifty feature of puppet;
 Nov 09 18:42:08 <kanarip>	but first... facts!
 Nov 09 18:42:17 <kanarip>	facts are... well... facts...
 Nov 09 18:42:33 <kanarip>	little information about the current state of a client
 Nov 09 18:43:02 <kanarip>	it includes the kernel version, number of network interfaces, the architecture,
 operating system name, operating system version, and lots more
 Nov 09 18:43:16 <kanarip>	yum install facter and run "facter" on your machine to check it out
 Nov 09 18:43:38 <kanarip>	these are all "variables" you can use in your puppet manifests
 Nov 09 18:44:09 <kanarip>	given a small amount of RAM for example, you may want to apply a snippet similar
 to: package { "firefox": ensure => absent }
 Nov 09 18:44:35 <kanarip>	anyway, what i really wanted to say rather then to give a rant on firefox's memory
 consumption is this;
 Nov 09 18:44:52 <kanarip>	facts, types and providers, these things that make puppet do smart things...
 Nov 09 18:45:01 <kanarip>	facts for information about the client...
 Nov 09 18:45:09 <erinlea80>	do we have a sample facter output?
 Nov 09 18:45:21 <kanarip>	i could fpaste you some
 Nov 09 18:45:35 <erinlea80>	that would be great :)
 Nov 09 18:45:53 <zcat>	kanarip, is facter supposed to spit out "virtual => vmware_server" if detected?
 Nov 09 18:45:56 <domg472_>	creating new ones doesnt look that easy
 Nov 09 18:46:10 <kanarip>
 Nov 09 18:46:41 <domg472_>	but theres good documentation on create types fact etc
 Nov 09 18:46:45 <kanarip>	types to define resources to be managed on the puppet...
 Nov 09 18:47:00 <erinlea80>	thanks
 Nov 09 18:47:03 <kanarip>	and providers to make puppet speak the local language...
 Nov 09 18:47:14 <kanarip>	the great thing is... you can write your own for all of these
 Nov 09 18:47:53 <kanarip>	it does require ruby knowledge, but then again... there's sample recipes on for...for example zcat, a vmware custom fact ;-)
 Nov 09 18:47:59 <kanarip>	see, i was getting there ;-)
 Nov 09 18:48:22 <kanarip>	actually i think the custom fact is called "this is a vmware machine" and
 "whether it has vm-tools installed"
 Nov 09 18:48:59 <kanarip>	one more thing i really, really need to stress out to you
 Nov 09 18:49:17 <kanarip>	all communication between the puppet and it's puppetmaster is *secure*
 Nov 09 18:49:45 *	nirik has one of his questions answered. ;) 
 Nov 09 18:50:02 <kanarip>	that being said, the security is based on the puppetmaster running the
 Certificate Authority, the client generating a certificate and putting in a request with the puppetmaster
 to get a signed copy of that certificate
 Nov 09 18:50:05 <erinlea80>	can communication be tunneled over ssh if necessary? (firewall restrictions, etc.)
 Nov 09 18:50:11 <domg472_>	but the puppets have root access obviously?
 Nov 09 18:50:18 <domg472_>	and the master
 Nov 09 18:50:22 <kanarip>	erinlea80, you can make the puppetmaster run on any port you want
 Nov 09 18:50:43 <kanarip>	erinlea80, may i point out "autossh" with which you can create tunnels to
 communicate over if you wish
 Nov 09 18:50:49 <erinlea80>	ty :)
 Nov 09 18:51:01 <kanarip>	domg472_, the puppet client daemon runs with root privileges, yes
 Nov 09 18:51:16 <domg472_>	has anyone tested this with selinux?
 Nov 09 18:51:16 <kanarip>	domg472_, the master runs with under the puppet user
 Nov 09 18:51:51 <domg472_>	ok thanks
 Nov 09 18:52:20 <kanarip>	domg472_, i have, and i can tell you it requires a custom selinux policy in
 which you basically describe the puppet daemon can do this and that and more
 Nov 09 18:52:31 <kanarip>	in my case, the puppet daemon can do anything it likes
 Nov 09 18:52:45 <domg472_>	youd need custom modules i bet
 Nov 09 18:52:58 <domg472_>	interesting
 Nov 09 18:53:21 <sdodson>	There are also a few minor bugs which I think have been fixed recently where
 puppet creates files without the proper contexts.
 Nov 09 18:53:34 <kanarip>	selinux is being integrated better into puppet as we speak
 Nov 09 18:53:39 <domg472_>	great
 Nov 09 18:53:49 <kanarip>	it's a big issue, i concur
 Nov 09 18:53:54 <nuonguy>	Q: if I enable, for example httpd in a manifest, and 2 hours later comment it out,
 does the puppet uninstall/disable httpd or ignore it from then on?
 Nov 09 18:54:17 <kanarip>	nuonguy, puppet only applies changes
 Nov 09 18:54:34 <kanarip>	nuonguy, so it only enforces what you describe needs to happen
 Nov 09 18:54:46 <kanarip>	if you say package httpd should be installed at one point
 Nov 09 18:54:46 <nuonguy>	so I'd have to explicity turn it off, if that's what I meant?
 Nov 09 18:55:20 <kanarip>	and then comment that out... no other machine is going to care whether the package
 should be installed, but it's not going to be removed on puppet's behalf
 Nov 09 18:55:26 <kanarip>	nuonguy, yes
 Nov 09 18:55:37 <nuonguy>	I dig, thanks
 Nov 09 18:56:00 <kanarip>	nuonguy, same goes for the service type; ensure => running, enable => true would
 need to become ensure => stopped, enable => false
 Nov 09 18:56:10 <nuonguy>	precisely
 Nov 09 18:56:22 <nuonguy>	I can just see myself not remembering that...
 Nov 09 18:56:33 <kanarip>	so, another thing on the topic; what do you think happens if the puppet daemon
 cannot contact the puppetmaster?
 Nov 09 18:56:34 <nuonguy>	and commenting it out instead
 Nov 09 18:56:56 <nirik>	does it do them in order in the manifest? ie, make sure to stop before ensure
 Nov 09 18:57:08 <kanarip>	nuonguy, there's some discussion going on about supporting an "exclude" statement
 like we do "include webserver" now
 Nov 09 18:57:18 <nuonguy>	nice
 Nov 09 18:57:24 <kanarip>	nuonguy, and then flipping all the false's into true's and so forth
 Nov 09 18:57:37 <kanarip>	nirik, good point!
 Nov 09 18:57:41 <sdodson>	Could create a baseline that ensures none of that is installed/running then
 include webserver for only your webservers which would override those baselines?
 Nov 09 18:57:46 <kanarip>	puppet allows ordering the resources;
 Nov 09 18:57:52 <kanarip>	let me look you up an example
 Nov 09 18:58:55 <domg472_>	why was ruby chosen as the language for writing types etc?
 Nov 09 18:59:05 <domg472_>	why not bash?
 Nov 09 18:59:42 <kanarip>	here's a yp manifest:
 Nov 09 19:00:21 <kanarip>	domg472_, i'm not sure i'm not one of the core developers, but i can try and
 answer it if you promise not to quote me on it ;-)
 Nov 09 19:00:32 <domg472_>	hah ok
 Nov 09 19:00:40 <domg472_>	i think most sysadmins know bash
 Nov 09 19:00:49 <kanarip>	nirik, you can see how "require => (...)" in that yp manifest causes one resource
 to be dependent on another
 Nov 09 19:00:59 <domg472_>	and they will need to write custom types probably
 Nov 09 19:01:51 <kanarip>	domg472_, i think bash is what you use to do a quick thing, clean and simple.
 I think ruby allows a framework to be built, reusing snippets of code you've already written in a different
 place. I think ruby is more modular then is bash
 Nov 09 19:02:06 <domg472_>	ok thanks
 Nov 09 19:02:16 <kanarip>	you could write a website in bash but you won't, right?
 Nov 09 19:02:21 <domg472_>	i guess ill be learning ruby then
 Nov 09 19:02:23 <domg472_>	true
 Nov 09 19:02:33 <domg472_>	but these are simplate admin tasks
 Nov 09 19:02:40 <domg472_>	simple
 Nov 09 19:02:42 <kanarip>	not saying you should choose php to do it BTW ;-)
 Nov 09 19:02:56 *	nirik notes we are over time, but this is also the last class today. 
 Nov 09 19:03:18 <kanarip>	in puppet terms; distribute the bash scripts amongst the clients that need it
 using the file type, and exec that script
 Nov 09 19:03:24 <nirik>	I had one last question: how do you decide what should be in a module? whats the
 logical boundry there?
 Nov 09 19:03:36 <kanarip>	anything you;
 Nov 09 19:03:42 <domg472_>	ah right, thanks kanarip
 Nov 09 19:03:47 <kanarip>	1) pull from an upstream source
 Nov 09 19:04:00 <kanarip>	2) need to share (with the general public)
 Nov 09 19:04:11 <kanarip>	3) want to stage
 Nov 09 19:04:33 <nirik>	so it doesn't need to be a specific service or anything? just whatever you want
 to put in one?
 Nov 09 19:04:38 <kanarip>	4) that is a concise set of files, manifests, etc.
 Nov 09 19:04:57 <sdodson>	They're advising people to get away from writing classes and move towards modules.
 Nov 09 19:05:04 <kanarip>	nirik, you're almost right, a module called everything-else doesn't make sense ;-)
 Nov 09 19:05:12 <kanarip>	sdodson, yes, very much so
 Nov 09 19:05:51 <kanarip>	nirik, a module webserver makes sense; keeping it close and concise within a
 module is way easier then dispersing it all over the place
 Nov 09 19:06:12 <nirik>	ok, makes sense.
 Nov 09 19:06:27 <tmz>	domg472_: regarding why ruby was used to write puppet, here's a short interview with
 Luke Kanies (primary puppet author),  where he touches on that choice:
 Nov 09 19:06:31 <tmz>
 Nov 09 19:06:40 <domg472_>	thanks
 Nov 09 19:06:46 <kanarip>	what i tend to recommend to customers is also, to keep the systems they manage as
 consistent as possible
 Nov 09 19:07:20 <kanarip>	being able to determine the exceptions to the rule you've layed down saves you a
 lot of time in determining what the cause might be for a specific issue
 Nov 09 19:08:11 <nirik>	shall we wrap up and let kanarip get some sleep?
 Nov 09 19:08:17 <kanarip>	;-)
 Nov 09 19:08:23 <daMaestro>	no, no sleep for kanarip !
 Nov 09 19:08:34 <kanarip>	thank you all for your interest, and thank you for attending
 Nov 09 19:08:37 *	daMaestro /dcc kanarip latte
 Nov 09 19:08:38 <jds2001>	he must stay up all night!
 Nov 09 19:08:41 <nirik>	thanks so much for the session kanarip !
 Nov 09 19:08:42 <erinlea80>	thanks kanarip!
 Nov 09 19:08:46 <domg472_>	thanks again
 Nov 09 19:08:47 <erinlea80>	:)
 Nov 09 19:08:49 <iondrip>	thanks
 Nov 09 19:08:53 <kanarip>	i hope you find a lot of use in the pdf/html/odp i've given you ;-)
 Nov 09 19:08:57 <neverho0d>	kanarip: thanks
 Nov 09 19:09:04 <nirik>	thanks everyone for coming...
 Nov 09 19:09:08 <Bugz_>	kanarip: Thanks
 Nov 09 19:09:15 <erinlea80>	:)
 Nov 09 19:09:31 <domg472_>	thanks for hosting
 Nov 09 19:09:57 <SSlater>	Thanks