From Fedora Project Wiki
(Add basic instructions to get kernel backtraces for denials)
(Elaborated more on advanced techniques. Confined users moved to a separate page.)
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
= How to debug SELinux issues =
= How to debug SELinux issues =
This page is currently a draft.
This page was created to set up the system to gather additional information to help with debugging issues related to SELinux.


== Install packages useful for debugging ==
== Install packages useful for debugging ==
Line 24: Line 24:
  $ sudo ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today
  $ sudo ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today


== Obtain kernel backtraces for AVC denials ==
This command output needs to be further troubleshooted.


In some cases it may help to determine the kernel code path through which the denial occurs. It is possible to use the kernel's tracing support to get the kernel (or even userspace) backtraces for SELinux denials.
== Switch the system to SELinux permissive mode ==
For testing purposes and to gather as many denials as possible, the permissive mode is useful so that the debugging person is not blocked.
 
Open the `/etc/selinux/config` file in an editor, change the `SELINUX=enforcing` line to
SELINUX=permissive
and reboot the system.
After tests finish, switch the system back to `enforcing`.


=== Using tracefs ===
For a one-time change to permissive, execute
$ sudo setenforce 0
This setting will be valid till the next reboot, or till `setenforce 1` was executed.


1. Run the following commands as root:
Alternatively, one particular SELinux domain can be made permissive (httpd_t in this example), leaving the rest of the system in enforcing mode:
$ sudo semanage permissive -a httpd_t
 
== Use `ausearch` to gather audited information ==
Numerous switches can be used with the ausearch command to gather audited data, interpret them, limit the results, etc. These switches have particular importance:
 
Interpret numeric values:
-i
Display only entries with a particular key:
-k key-value
Look only for data starting at some point - 10 minutes ago, today, reboot:
-ts recent
-ts today
-ts boot
 
Refer to ausearch(1) for additional information.
 
 
= Advanced debugging =
In many cases it is helpful to track the failing syscalls or the kernel code path through which the denial occurs. For that, use the strace and perf commands, or use the kernel's tracing support to get the kernel (or even userspace) backtraces for SELinux denials. Strace helps with a failing syscall, ltrace with userspace library call, perf in intricate problems, tracefs where kernel is directly involved.
 
Install additional tools and debugging information for affected packages (systemd in this example).
$ sudo dnf -y install dnf-utils strace ltrace perf
$ debuginfo-install "systemd*"
 
== Use `strace` to trace process syscalls ==
The strace tool can be used to trace system calls to isolate a particular syscall which fails.
 
$ sudo strace -vxy -s 256 -o /tmp/command.strace /path/command -options
 
To track also SELinux context changes, use the `--secontext` switch:
 
$ sudo strace -vxy -s 256 --secontext -o /tmp/command.strace /path/command -options
 
Refer to strace(1) for additional information.
 
== Use `ltrace` to trace library calls ==
The ltrace tool can be used in case it is a userspace library call which fails.
 
$ sudo ltrace -C -s 256 -o /tmp/command.ltrace /path/command -options
 
Refer to ltrace(1) for additional information.
 
== Use `auditctl` to audit a particular syscall ==
The audit service can be set to audit particular system calls.
 
1. Read the syscall's man page in the 2nd man section to identify all related syscalls. For example, to create a directory, 2 distinct syscalls can be used: mkdir and mkdirat.
 
2. Open the `/etc/audit/rules.d/syscalls.rules` file in an editor.
 
3. Add the following line to the end of the file (example for mkdir):
-a always,exit -F arch=b64 -S mkdir,mkdirat -F key=mkdir


# echo stacktrace >/sys/kernel/tracing/trace_options
4. Restart the audit daemon using the legacy `service` command, or reboot the system:
  # echo 1 >/sys/kernel/tracing/events/avc/selinux_audited/enable
  $ sudo service auditd restart


2. Run the scenario which triggers the SELinux denials.
5. Run the scenario which uses the syscalls.


3. Dump the backtraces of captured AVC events:
6. Collect the audited events for further troubleshooting:
$ sudo ausearch -i -k mkdir -ts recent


# cat /sys/kernel/tracing/trace
Refer to auditctl(8) for explanation and additional information.
[...]


4. (Optional) Reset the tracing settings by running the following commands (or just rebooting the machine):
== Using `perf` to trace denials triggered by a command ==
Install perf and debugging information for the involved package and its libraries. Execute a command through perf:
$ sudo perf record -o perf.data -a -g --call-graph dwarf -e avc:selinux_audited -- /path/command -options
The `perf.data` file with information about audited denials caused by the command is stored.


# echo nostacktrace >/sys/kernel/tracing/trace_options
== Using `perf` to trace all system denials ==
  # echo 0 >/sys/kernel/tracing/events/avc/selinux_audited/enable
Install perf and debugging information for the involved package and its libraries. Execute perf and terminate it when the scenario is finished:
  $ sudo perf record -o perf.data -a -g --call-graph dwarf -e avc:selinux_audited
^C


=== Using `perf` to trace denials from a command ===
The `perf.data` file with information about all denials audited during the perf command run is stored.


TBA
== Interpret data gathered by perf ==
Run `perf script` to read stored data and display stack trace:
$ sudo perf script -i perf.data
Run `perf report` to display call chains:
$ sudo perf report -g


=== Using `perf` to trace denials globally ===
Refer to perf-record(1) for additional information.


TBA
== Using tracefs ==


== Setting up confined users ==
1. Run the following commands as root:
=== Create new users assigned to a particular SELinux user ===


  PWD=${PWD-"my_p4ss-w0rd"}
  # echo stacktrace >/sys/kernel/tracing/trace_options
for username in guest xguest user staff
  # echo 1 >/sys/kernel/tracing/events/avc/selinux_audited/enable
  do
  adduser -Z ${username}_u ${username}
  echo "${PWD}" | passwd --stdin "${username}"
done
=== Assign a SELinux user to an existing Linux user ===
$ sudo semanage login -a -s staff_u existinguser
$ sudo semanage login -l
Login Name          SELinux User        MLS/MCS Range        Service
...
existinguser        staff_u              s0-s0:c0.c1023      *
=== Assign a SELinux user an additional role ===
By default, the staff user is not allowed to access the dbadm role.
$ sudo semanage user -l
                Labeling  MLS/       MLS/
SELinux User    Prefix    MCS Level  MCS Range                      SELinux Roles
...
staff_u        user      s0        s0-s0:c0.c1023                staff_r sysadm_r system_r unconfined_r
$ sudo semanage user -m -R "staff_r sysadm_r system_r unconfined_r dbadm_r" staff_u
$ sudo semanage user -l
                Labeling  MLS/       MLS/
SELinux User    Prefix    MCS Level  MCS Range                      SELinux Roles
...
staff_u        user      s0        s0-s0:c0.c1023                staff_r sysadm_r system_r unconfined_r dbadm_r


=== Assign admin roles to Linux users when they use sudo ===
2. Run the scenario which triggers the SELinux denials.
On the sudo commands execution, sudo can be configured so that the user id changes as well as the SELinux role and the corresponding type.
$ sudo cat > /etc/sudoers.d/admin-roles << EOF
# staff can become sysadm for all commands and shell
staff        ALL=(ALL)      ROLE=sysadm_r TYPE=sysadm_t NOPASSWD: ALL
# staff2 can only run networking commands
#Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool
#staff2        ALL=(ALL)      ROLE=sysadm_r TYPE=sysadm_t NOPASSWD: NETWORKING
# staff3 can become dbadm for databases administration
#CMND_Alias DATABASES = /usr/bin/mariadb-admin /usr/bin/mysqladmin /usr/bin/psql
#staff3        ALL=(ALL)      ROLE=dbadm_r TYPE=dbadm_t NOPASSWD: DATABASES
EOF


== Switch the system to SELinux permissive mode ==
3. Dump the backtraces of captured AVC events to report them:
For testing purposes and to gather as many denials as possible, the permissive mode is useful not to be blocked in actual work.


Open the `/etc/selinux/config` file in an editor, change the `SELINUX=enforcing` line to
# cat /sys/kernel/tracing/trace
  SELINUX=permissive
  [...]
and reboot the system.
After tests finish, switch the system back to `enforcing`.


For a one-time change to permissive, execute
4. (Optional) Reset the tracing settings by running the following commands (or just rebooting the machine):
$ sudo setenforce 0
The setting will be valid till the next reboot.


== Advanced debugging ==
  # echo nostacktrace >/sys/kernel/tracing/trace_options
Install additional tools and debugging information for affected packages (systemd in this example).
  # echo 0 >/sys/kernel/tracing/events/avc/selinux_audited/enable
  $ sudo dnf -y install dnf-utils strace perf
  $ debuginfo-install "systemd*"
t.b.c.

Latest revision as of 13:09, 6 June 2023

How to debug SELinux issues

This page was created to set up the system to gather additional information to help with debugging issues related to SELinux.

Install packages useful for debugging

$ sudo dnf -y install setools-console selinux-policy-devel policycoreutils-newrole strace initscripts-service bzip2

Enable full auditing

For performance reasons, full auditing is not enabled by default. Instructions how to enable it:

1. Open the /etc/audit/rules.d/audit.rules file in an editor.

2. Remove the following line if it exists:

-a task,never

3. Add the following line to the end of the file:

-w /etc/shadow -p w

4. Restart the audit daemon using the legacy service command, or reboot the system:

$ sudo service auditd restart

5. Run the scenario which effects in SELinux denials.

6. Collect AVC denials:

$ sudo ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today

This command output needs to be further troubleshooted.

Switch the system to SELinux permissive mode

For testing purposes and to gather as many denials as possible, the permissive mode is useful so that the debugging person is not blocked.

Open the /etc/selinux/config file in an editor, change the SELINUX=enforcing line to

SELINUX=permissive

and reboot the system. After tests finish, switch the system back to enforcing.

For a one-time change to permissive, execute

$ sudo setenforce 0

This setting will be valid till the next reboot, or till setenforce 1 was executed.

Alternatively, one particular SELinux domain can be made permissive (httpd_t in this example), leaving the rest of the system in enforcing mode:

$ sudo semanage permissive -a httpd_t

Use ausearch to gather audited information

Numerous switches can be used with the ausearch command to gather audited data, interpret them, limit the results, etc. These switches have particular importance:

Interpret numeric values:

-i

Display only entries with a particular key:

-k key-value

Look only for data starting at some point - 10 minutes ago, today, reboot:

-ts recent
-ts today
-ts boot

Refer to ausearch(1) for additional information.


Advanced debugging

In many cases it is helpful to track the failing syscalls or the kernel code path through which the denial occurs. For that, use the strace and perf commands, or use the kernel's tracing support to get the kernel (or even userspace) backtraces for SELinux denials. Strace helps with a failing syscall, ltrace with userspace library call, perf in intricate problems, tracefs where kernel is directly involved.

Install additional tools and debugging information for affected packages (systemd in this example).

$ sudo dnf -y install dnf-utils strace ltrace perf
$ debuginfo-install "systemd*"

Use strace to trace process syscalls

The strace tool can be used to trace system calls to isolate a particular syscall which fails.

$ sudo strace -vxy -s 256 -o /tmp/command.strace /path/command -options

To track also SELinux context changes, use the --secontext switch:

$ sudo strace -vxy -s 256 --secontext -o /tmp/command.strace /path/command -options

Refer to strace(1) for additional information.

Use ltrace to trace library calls

The ltrace tool can be used in case it is a userspace library call which fails.

$ sudo ltrace -C -s 256 -o /tmp/command.ltrace /path/command -options

Refer to ltrace(1) for additional information.

Use auditctl to audit a particular syscall

The audit service can be set to audit particular system calls.

1. Read the syscall's man page in the 2nd man section to identify all related syscalls. For example, to create a directory, 2 distinct syscalls can be used: mkdir and mkdirat.

2. Open the /etc/audit/rules.d/syscalls.rules file in an editor.

3. Add the following line to the end of the file (example for mkdir):

-a always,exit -F arch=b64 -S mkdir,mkdirat -F key=mkdir

4. Restart the audit daemon using the legacy service command, or reboot the system:

$ sudo service auditd restart

5. Run the scenario which uses the syscalls.

6. Collect the audited events for further troubleshooting:

$ sudo ausearch -i -k mkdir -ts recent

Refer to auditctl(8) for explanation and additional information.

Using perf to trace denials triggered by a command

Install perf and debugging information for the involved package and its libraries. Execute a command through perf:

$ sudo perf record -o perf.data -a -g --call-graph dwarf -e avc:selinux_audited -- /path/command -options

The perf.data file with information about audited denials caused by the command is stored.

Using perf to trace all system denials

Install perf and debugging information for the involved package and its libraries. Execute perf and terminate it when the scenario is finished:

$ sudo perf record -o perf.data -a -g --call-graph dwarf -e avc:selinux_audited
^C

The perf.data file with information about all denials audited during the perf command run is stored.

Interpret data gathered by perf

Run perf script to read stored data and display stack trace:

$ sudo perf script -i perf.data

Run perf report to display call chains:

$ sudo perf report -g

Refer to perf-record(1) for additional information.

Using tracefs

1. Run the following commands as root:

# echo stacktrace >/sys/kernel/tracing/trace_options
# echo 1 >/sys/kernel/tracing/events/avc/selinux_audited/enable

2. Run the scenario which triggers the SELinux denials.

3. Dump the backtraces of captured AVC events to report them:

# cat /sys/kernel/tracing/trace
[...]

4. (Optional) Reset the tracing settings by running the following commands (or just rebooting the machine):

# echo nostacktrace >/sys/kernel/tracing/trace_options
# echo 0 >/sys/kernel/tracing/events/avc/selinux_audited/enable