The 4 key causes of SELinux errors (20090503 Classroom)

Given by Dan Walsh (irc: dwalsh)

Slides

OpenOffice Slides

IRC Log

dwalsh	This is going to be the session on "What is SELinux trying to tell me?"	12:59
dwalsh	My name is Dan Walsh. I am in the Boston area.	12:59
dwalsh	I put together some slides for this presentation	13:00
dwalsh	http://people.fedoraproject.org/~dwalsh/SELinux/selinux4things.odp	13:00
dwalsh	Where is everyone else? It is 9 AM here.	13:01
* ianweller waves		13:01
dgrift_	Rotterdam Netherlands 15:00	13:01
maximus007	florida	13:01
ianweller	kansas, 8am	13:01
nihed	Tunisia	13:01
pluvo	Germany, 15:00	13:01
nihed	Tunisia, 14:00	13:01
linuxguru	india , 18:30	13:02
dwalsh	Cool.	13:02
dwalsh	I have been working on and off with the Fedora Infrastructure team on implementing SELinux and learned quite a bit.	13:03
dwalsh	A few months ago I wrote a blog on it called	13:03
dwalsh	Top three things to understand in fixing SELinux problems.	13:03
dwalsh	http://danwalsh.livejournal.com/22347.html	13:03
dwalsh	What I found is just about every machine we went on required a little bit of massaging	13:04
dwalsh	So I came up with this list.	13:04
dwalsh	I plan on writing a paper on Securing Fedora Infrastructure and explaining everything we did for Red Hat Magazine. And presenting on it at Red Hat Summit in September.	13:05
dwalsh	Anyways lets start.	13:05
dwalsh	The #1 thing I try to explain to people about SELinux is LABELING.	13:06
dwalsh	SELinux is all about labeling. Every process has a label, every file has a label.	13:06
dwalsh	If the labels are wrong, SELinux breaks down.	13:06
dwalsh	So when a problem happens in SELinux the first thing to check is the labeling.	13:06
dwalsh	One example of labeling getting screwed up is in the vmware package.	13:07
dwalsh	In the RPM spec, they decided to edit the /etc/services file.	13:07
dwalsh	They did this by copying it to /tmp and adding a port number and then moving it back to /etc	13:08
dwalsh	Well in SELinux if you cp /etc/services to /tmp; the tmp file gets labeled rpm_script_tmp_t	13:08
dwalsh	But the mv command maintains context, so mv moves the file back to /etc and the file is still labeled rpm_script_tmp_t.	13:09
dwalsh	Now every confined domain that tries to read /etc/services is going to blow up because the label is wrong.	13:09
dwalsh	Fixing the problem is as simple as restorecon /etc/services.	13:09
dwalsh	But SELinux generates nasty avc's saying apache process tried to read rpm_script_tmp_t.	13:10
dwalsh	BTW if anyone has questions, feel free to ask at any time.	13:10
dwalsh	When we did the Fedora infrastructure, the admins decided they wanted apache content served out of /srv/web.	13:11
icarus-c	dwalsh: are you meaning that sometimes avc throws "wrong" message?	13:11
dwalsh	No it throws the correct message, but the message is caused by bad labeling.	13:11
dwalsh	The /srv/web context on a RHEL5 box was var_t	13:12
dwalsh	But apache policy does not allow httpd_t to read/write var_t. so we needed to change the labeling.	13:13
dwalsh	We could use chcon to change the labels.	13:13
dwalsh	chcon -R -t http_sys_content_t /srv/web	13:14
dwalsh	But this does not survive a relabel.	13:14
dwalsh	You need to TELL selinux about the change from default labeling.	13:14
dwalsh	So you use the semanage command	13:14
dwalsh	semanage fcontext -a -t http_sys_content_t '/srv/web(/.*)?'	13:15
dwalsh	This tells SELinux to label all files/directories under /srv/web as apache content.	13:15
dwalsh	semanage fcontext -a -t httpd_sys_content_t '/srv/web(/.*)?'	13:15
dwalsh	Should have been httpd_sys_content_t	13:15
dwalsh	In SELinux we use regular expressions to figure out the labeling. So that is why we have the weird file expression.	13:16
dwalsh	After you tell SELinux about the labeling, you need to change the actual files on disk	13:16
dwalsh	restorecon -R -v /srv/web	13:16
dwalsh	Now your httpd daemon is able to read the files.	13:17
dwalsh	If you wanted to be able to write the files you could have set the labels to	13:17
dwalsh	httpd_sys_content_rw_t.	13:17
dwalsh	Questions?	13:17
moixs	One :)	13:18
BounceCat	dwalsh: if you want us to follow along in your slides, please say when to change slides. thanks.	13:18
dwalsh	Ok, This is slide 3	13:18
dwalsh	SELinux == Labeling.	13:18
linuxguru	Sometimes it still doesn't work i mean setting up context to httpd_sys_content_rw_t in case of a typical dokuwiki installation in /var/www/html/ i faced this situation.	13:18
moixs	When you make these changes, can you save them and replicate them easily on another system?	13:18
BounceCat	ok	13:18
dwalsh	The notes section has decent content also	13:18
dwalsh	moixs, In Fedora infrastructure we are using puppet to basically execute the semanage commands on a large group of machines.	13:19
dwalsh	We plan on using IPA also for this in RHEL6 time frame.	13:20
dwalsh	linuxguru, Please explain?	13:20
moixs	Ok, seems reasonable	13:20
dgrift_	one can also use a policy package to distribute file contexts across systems	13:20
dwalsh	The commands are pemanent.	13:20
linuxguru	dwalsh, it said me to enable a boolean , httpd_unified	13:20
dwalsh	linuxguru, Yes that is a bug in httpd_unified, in F10.	13:20
linuxguru	okay	13:21
dwalsh	linuxguru, I will cover the boolean in the next section.	13:21
linuxguru	dwalsh, okay sir	13:21
dwalsh	In this case it is more secure to leave httpd_unified off and set the context to httpd_sys_content_rw_t.	13:21
dwalsh	A couple more comments on labeling.	13:21
linuxguru	yeah. i had to do that for every file it complained off.	13:22
linuxguru	in random directories sometimes. so not sure if should apply on directory or not	13:22
dwalsh	SELinux handles labels at the directory level better then individual files.	13:22
dwalsh	So if you can label the directory and all its contents with a single label, SELinux becomes more managable.	13:23
dwalsh	For example, If I have a confined domain that wants to manage files in /etc.	13:23
dwalsh	It would be better to have the app manage /etc/myapp/*	13:24
dwalsh	And label /etc/myapp as myapp_etc_rw_t	13:24
dwalsh	That way by default all files created in /etc/myapp will be created as myapp_etc_rw_t	13:24
dwalsh	If I just put the files in /etc. they would default to etc_t and then I would have to rely on the administrator maintaining the label.	13:25
dwalsh	If an administrator decides to create his own top level directory under /	13:25
dwalsh	The label will be labeled default_t.	13:25
dwalsh	NO confined apps are allowed to use default_t for that reason.	13:26
dwalsh	We expect the admin might be putting the secrets to the "Lost Arch" there so we don't want the untrusted domains to be able to read it.	13:26
dwalsh	So if you create a directory for use by a service, you need to change the label to the correct context.	13:27
dwalsh	A good command to look for context that a domain can write, is sesearch	13:27
dwalsh	sesearch --allow -s httpd_t -c file -p write	13:27
dwalsh	This should show you all the context that the httpd_t label can write	13:28
dwalsh	That is one method to find the correct context.	13:28
dwalsh	We also have man httpd_selinux and a few other man pages.	13:29
dwalsh	Red Hat is developing a guide in Fedora for managing confined services.	13:29
dwalsh	Which I can not find the link for now. But it is out there.	13:30
dwalsh	One special context type you will see out there is "file_t"	13:30
dgrift_	http://sradvan.fedorapeople.org/SELinux_Managing_Co	13:30
dwalsh	THis type indicates the file has no label at all on it.	13:31
dwalsh	It should have been called unlabeled_t if I could use a wayback machine I would go back to 2002 and change it.	13:31
dgrift_	proper url: http://sradvan.fedorapeople.org/SELinux_Managing_Confined_Services/en-US/html-single/	13:31
dwalsh	If you see this label is usually means you have a very badly mislabeled machine, and you need to trigger a full relabel	13:31
dwalsh	touch /.autorelabel; reboot	13:32
dwalsh	Or Just run fixfiles restore.	13:32
dwalsh	But you usually do not need to do this...	13:32
dwalsh	One other time I see file_t showing up, might be related to usb sticks.	13:32
dwalsh	Someone moving a file off of usb stick. Although that is only a theory of mine, I have not been able to prove it.	13:33
dwalsh	File labeling also comes into play, when selecting the label to run a process as	13:34
dwalsh	What I mean by this is SELinux defines "Transitions"	13:34
dwalsh	When an SELinux system boots, it starts out running the kernel with no context, then the initrd loads the policy.	13:35
dwalsh	The policy says to run the system as kernel_t	13:35
dwalsh	It also defines a transition from kernel_t.	13:35
dwalsh	When kernel_t runs an executable labeled init_exec_t it will transition to init_t	13:36
dwalsh	So this is how upstart or init gets labeled init_t.	13:36
dwalsh	Other transtions also happen	13:36
dwalsh	init_t->initrc_exec_t->initrc_t	13:36
dwalsh	So all init scripts are labeled initrc_exec_t by default.	13:37
dwalsh	Or some varient,	13:37
dwalsh	Then a transition rule is written that says	13:37
dwalsh	initrc_t->httpd_exec_t->httpd_t	13:37
dwalsh	And finally	13:37
dwalsh	httpd_t->httpd_sys_script_exec_t->httpd_sys_script_t	13:37
dwalsh	So if ANY labeling is wrong the transitions will fail and the app will end up running under the wrong context.	13:38
dwalsh	So Always check you labeling.	13:38
dwalsh	Slide 4	13:38
BounceCat	If I want to see what domain a particular executable is labelled as, can you give a cli example how to see that? (apart from reading selinux errror message)	13:38
dwalsh	Well you can check the label on disk	13:39
dwalsh	ls -lZ /usr/sbin/httpd	13:39
dwalsh	This will show you the current label	13:39
dwalsh	You can ask the machine what the label should be	13:39
dwalsh	matchpathcon /usr/sbin/httpd	13:39
dwalsh	If they are different you can make it write	13:40
dwalsh	restorecon /usr/sbin/httpd	13:40
BounceCat	great thanks.	13:40
dwalsh	But say you have another app that works just like httpd. Has the same security domain.	13:40
dwalsh	/usr/sbin/myhttpd	13:40
dwalsh	You would want this to be labeled like httpd	13:41
dwalsh	semanage fcontext -a -t httpd_exec_t /usr/sbin/httpd	13:41
dwalsh	Slide 4 shows you system-config-selinux, which allows you to set the file context	13:41
dwalsh	Via a GUI, if you are so inclined.	13:41
dwalsh	One interesting addition to F11 is the idea of equivalence.	13:42
dwalsh	What I was looking for here was a way to say /srv/web should be labeled just like /var/www/	13:42
dwalsh	I want to copy all file context that match the directory /srv/web and substiture /var/www	13:43
dwalsh	So with semanage I can execute	13:43
dwalsh	semanage fcontext -a -e /var/www /srv/web	13:43
dwalsh	Which is a pretty powerfull concept.	13:43
dwalsh	Another example would be setting up an alternate location for home dirs.	13:44
dwalsh	Ok I got to get moving...	13:44
dwalsh	Slike 5	13:44
dwalsh	Covers the semange equivalence and genhomedircon	13:45
dwalsh	Lets move onto Slide 6	13:45
dwalsh	SELinux needs to KNOW.	13:45
dwalsh	The idea here is that confined apps can be configured to run in different ways.	13:45
dwalsh	So you need to tell SELinux how you are going to run the apps.	13:46
dwalsh	I am often asked why don't the writers of APPS write their own policy.	13:47
dwalsh	I think it would be a good idea but it also could be a bad idea.	13:47
moixs	(good question, who in Fedora actually does this?)	13:48
dwalsh	As an example, if I asked the writer of vsftpd to define their policy	13:48
dwalsh	They would say that ftp sometimes needs to read any file on disk,	13:48
dwalsh	Sometimes needs to write any file on disk.	13:48
dwalsh	So policy should allow ftp to read/write everything.	13:48
* maximus007 cringes		13:49
dwalsh	Except that 99% of all users of ftp just use it to share a very small directori	13:49
dwalsh	/var/ftp	13:49
dwalsh	So A better solution would to have rules built into policy for an admin to tell SELInux how he uses frp	13:49
dwalsh	ftp	13:49
dwalsh	These are called booleans	13:50
dwalsh	In the case of we have several booleans	13:50
dwalsh	# semanage boolean -l | grep ftp	13:51
dwalsh	ftp_home_dir -> off Allow ftp to read and write files in the user home directories	13:51
dwalsh	allow_ftpd_full_access -> off Allow ftp servers to login to local users and read/write all files on the system, governed by DAC.	13:51
dwalsh	allow_ftpd_use_nfs -> off Allow ftp servers to use nfs for public file transfer services.	13:51
dwalsh	allow_ftpd_anon_write -> off Allow ftp servers to upload files, used for public file transfer services. Directories must be labeled public_content_rw_t.	13:51
dwalsh	tftp_anon_write -> off Allow tftp to modify public files used for public file transfer services.	13:51
dwalsh	ftpd_connect_db -> off Allow ftp servers to use connect to mysql database	13:51
dwalsh	allow_ftpd_use_cifs -> off Allow ftp servers to use cifs for public file transfer services.	13:51
dwalsh	httpd_enable_ftp_server -> off Allow httpd to act as a FTP server by listening on the ftp port.	13:51
dwalsh	By default ftp is setup to only allow access to /var/ftp directory but you can alter these booleans to allow additional access.	13:51
dwalsh	Fedora 11 has about 127 booleans now.	13:52
dwalsh	Slide 7 and Slide 8 show you two different GUI tools used to manipulate booleans	13:53
dwalsh	You can also use audit2allow -w (audit2why) to look at an AVC and see if a boolean would have allowed the access	13:53
dwalsh	setroubleshoot uses this functionality to try to figure out if a boolean is ok	13:54
dwalsh	The tool is actually attempting all possible booleans to see if any one would allow the access.	13:54
dwalsh	In some cases multiple booleans are required to be turned on to allow access.	13:54
dwalsh	For example allowing httpd scripts to be executed in a users home directory requires two or three booleans.	13:55
dwalsh	Sadly audit2allow -w is not smart enough to figure this out.	13:55
dwalsh	Again the management guide and other tools are helpful in figuring this out.	13:56
dwalsh	Sline 9	13:56
dwalsh	Slide 9	13:56
dwalsh	Sadly SELINux Policy has bugs.	13:56
dwalsh	Apps have bugs also	13:56
dwalsh	Sometimes we release an app that causes new SELinux errors.	13:56
dwalsh	We try not to do this but Fedora is an active community.	13:57
dwalsh	I try not to add new confinement within a release.	13:57
dwalsh	Once Fedora 11 is released, we tend to only loosen the policy. Add new allow rules, but not implement new confinement.	13:57
dwalsh	And on RHEL releases we really frown on this.	13:58
dwalsh	But What does an administrator do if SELinux or an APP are just broken.	13:58
dwalsh	They labeling is correct.	13:58
dwalsh	No Boolean allows the access?	13:58
dwalsh	In this case the admin can fairly easily ad a custom policy module using audit2allow	13:58
dwalsh	grep avc /var/log/audit/audit.log | audit2allow -M mypol	13:59
dwalsh	This create a file called mypol.pp	13:59
dwalsh	which can then be installed via semodule	13:59
dwalsh	semodule -i mypol.pp	13:59
dwalsh	mypol.te also gets created,	14:00
dwalsh	You should really examine what this file is allowing, since it could be a security problem.	14:00
dwalsh	For example allowing a confined app to write to etc_t is not a good idea since the app would be allowed to overwrite /etc/passwd	14:00
dwalsh	But this method will allow you to customize policy to make the app just work.	14:01
dwalsh	When you generate a policy using this method it creates a policy called mypol. if you execute the command later with different avcs but use the same name, you will override the first rules and replace them with new rules.	14:01
dwalsh	So be carefull.	14:02
dwalsh	Questions?	14:02
BounceCat	"Slide 7 and Slide 8 show you two different GUI tools used to manipulate booleans" .. please name these tools	14:02
moixs	Maybe a more general one	14:02
moixs	How do you manage applications from external repositories, like RPM Fusion? Is the packager from RPM Fusion supposed to communicate with Fedora about SELinux policies? Does RPM Fusion maintain its own set of SELinux policies?	14:02
dwalsh	BounceCat, system-config-selinux launces both.	14:03
BounceCat	ok	14:03
dwalsh	One is the default view, the second is created by going to the booleans screen and hitting lockdown.	14:03
BounceCat	thx	14:03
dwalsh	moixs, These apps may or maynot have policy defined for them.	14:04
dwalsh	But they should all be able to run on a Fedora 11 box.	14:04
moixs	So it's the packager's responsibility do do that?	14:04
dwalsh	As I stated in an internal list, I think we can get the biggest piece of crap software to be able to run with SELinux in enforing mode.	14:05
moixs	ok :p	14:05
dwalsh	Of course this does not mean that the app will be secure...	14:05
dwalsh	moixs, I will open a bugzilla or whatever with any app that is struggling with SELinux, or tell the bugreporter to do that	14:05
dwalsh	Most problems with these software apps is badly built shared libraries.	14:06
dwalsh	Which causes execmod violations.	14:06
dwalsh	So we end up having to label them textrel_shlib_t w	14:06
moixs	So you, in Fedora, will implement a "workaraound" even for an "external" application?	14:06
dwalsh	which tells SELinux to now worry about a library being hacked,	14:06
dwalsh	moixs, Yes, My goal is not to stop you from running you app.	14:07
moixs	okay	14:07
dwalsh	Otherwise you will dump us for some other unnamed distro.	14:07
dwalsh	:^)	14:07
moixs	It's more that I package an daemonized application in RPM Fusion and never cared about SELinux :p	14:07
dwalsh	So in RHEL and Fedora by default apps run as an unconfined domain, and should be able to run unconstrained by SELinux	14:07
dwalsh	moixs, Fine, It should run.	14:08
dwalsh	Well let me complete the talk since I got to go to a Baseball tryout in 15 minutes. Not mine  :^(	14:08
linuxguru	oh. hehe	14:08
underscores	lol	14:08
dwalsh	When I originally wrote the blog, I talked about fixing SELinux problems. but ignored the most important thing about SELinux	14:08
dwalsh	How do I tell if I my machine was cracked.	14:09
dwalsh	Well Sadly we have not done a good job of that, and I hope to make strides on this in F12.	14:09
dwalsh	setroubleshoot is a tool that shows you selinux reported a problem. And it tries to give you a remedy of a solution.	14:10
dwalsh	We want to build some plugins that look for obvious breakins.	14:10
dwalsh	For example, no confined application on an SELinux box is allowed to turn off SELinux.	14:10
dwalsh	So it would be real nice to ring alarms if httpd_t tries to setenforce	14:11
dwalsh	If I was breaking into an SELInux system, the first thing I would try is to turn it off.	14:11
maximus007	well if anything but uid 0 tries it i think we have a problem and maybe even then	14:11
dwalsh	Similarly loading kernel modules, writeing to security directories/files	14:11
dwalsh	/etc/shadow	14:12
dwalsh	/etc/selinux/*	14:12
dwalsh	Kernel directories ...	14:12
maximus007	my question is how to prevent even root from turning it off, will i have to directly modify the policy?	14:12
dwalsh	So we need to change setroubleshoot to give a clearer indication that we believe the machine is compromised, and take action.	14:13
dwalsh	But you can try to read the AVCs or look at the rules generated by audit2allow to figure out if something very strange is going on.	14:13
dwalsh	That is the end of the formal presentation I can stay for about 5 more minutes.	14:14
dwalsh	Any other questions?	14:14
dwalsh	Or did I miss any?	14:14
dwalsh	Was this useful?	14:14
dgrift_	yes	14:14
mamurdian	it would be nice if you could provide more such lessons	14:14
linuxguru	sure it was dwalsh , very useful. got to know about some of the new stuff in selinux planned for f11.	14:15
underscores	yes :)	14:15
BounceCat	It was great Dan, thanks. I have 1 more question, but others can go first.	14:15
dwalsh	Ok Maybe next month I will cover svirt.	14:15
pluvo	Thanks dwalsh!	14:15
maximus007	how do i keep root from turning off selinux? I can confine root but root can unconfine himself can he not? so i think i have to modify policy directly	14:15
moixs	Yep, thank you :)	14:16
mamurdian	maximus007> i think you can give root another role	14:16
dwalsh	maximus007, Read my blog about confined users/administrators.	14:16
dwalsh	THe simple answer is you define a confined admin	14:16
dwalsh	Another talk.	14:16
maximus007	ok thanks for the class, well done	14:16
Bighusky	we had to install a newer package (rsync) on rhel5. Rsync and Rsyncd were not able to function properly, so we ended up using audit2allow -M to write a custom policy for this package. Was that the right approach or should we have used other means?	14:16
linuxguru	maximus007, i don't think that requires changing the policy directly. semanage is your friend.	14:16
dwalsh	Bighusky, yes	14:16
dgrift_	Bighusky didyou read man rsync_selinux first?	14:17
dwalsh	Bighusky, Although send me your changes dwalsh@redhat.com And I will see about getting them upstream	14:17
dwalsh	Bighusky, dgrift_ Is write that a boolean might have helped you	14:17
maximus007	linuxguru : yes but root is root and can generally run unconfined, so i'd have to remove the unconfined role i think but i will try it and see	14:17
DiscordianUK	Thanks very much dwalsh	14:17
dwalsh	rsync booleans in F11	14:18
dwalsh	semanage boolean -l | grep rsync	14:18
dwalsh	rsync_client -> off Allow rsync to run as a client	14:18
dwalsh	rsync_export_all_ro -> off Allow rsync to export any files/directories read only.	14:18
dwalsh	allow_rsync_anon_write -> off Allow rsync to modify public files used for public file transfer services. Files/Directories must be labeled public_content_rw_t.	14:18
dwalsh	maximus007, You need to define an admin role and then don't give the admin the root password	14:18
dgrift_	BounceCat questions?	14:19
Bighusky	dwalsh, dgrift, thanks very much, I have to admit that I didn't read man rsync_selinux. I will go back and give these suggestions a try.	14:19
BounceCat	Subdirectories I've created in my home dir are either unconfined_u, system_u, or user_u, apparently at random. What has caused that and does it matter ?	14:19
dwalsh	This confined admin would then log in as a confined user and use sudo to become root.	14:19
dwalsh	By default all files are created and relabeled to system_u	14:19
dwalsh	If you log onto a box as a particular user type	14:20
dwalsh	user_u or confined_u	14:20
dwalsh	user_u or uconfined_u	14:20
dwalsh	Files created by you will end up using that user type	14:20
dwalsh	user_u	14:20
maximus007	dwalsh, ok and thanks again excellent talk I have to get going myself but i will be sure to pester you about it in fedora-selinux :^)	14:20
dwalsh	So user_u:user_r:user_t creates /home/dwalsh/foo	14:20
dwalsh	will get created as user_u:object_r:user_home_t	14:21
dwalsh	But the user componant of the file type is seldom used for denials.	14:21
dwalsh	Bye.	14:21
dgrift_	thanks	14:21
BounceCat	thank you	14:21
dgrift_	anyone have any questions feel free to join #fedora-selinux	14:22
linuxguru	yes. over and out to #fedora-selinux	14:22
dgrift_	we might not be available right aways but we are glad if we can help	14:22
Bighusky	than you very much and have a good sunday	14:23

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!