From Fedora Project Wiki

About Xen

The Xen virtual machine system is now available in Fedora Core 4 and above versions. More information on Xen itself can be found at http://wiki.xensource.com/xenwiki/ and the Fedora Xen page.

Fedora is following the -unstable (3.0.x) Xen development line at the moment, which does occasionally lead to things being broken but also lets us track a lot of the more interesting work going on there. It also prevents the current NetBSD/xen, FreeBSD 5.3 and Solaris 11beta ports from running because they are all based on Xen 2.0.x.

About This Quick Start

Since setting up to run Xen isn't entirely straight-forward, here's a run-through of what should work for configuring a Fedora Core 5 system as a Xen host, and setting up a single Xen guest.

This Quick Start is an update version of the Fedora Core 4 Xen Quick Start page. A number of things have changed since FC4 and Xen 2 such as package names and configuration options, this guide is aimed at being a fresh start rather than an upgrade guide.

Note.png
This Quick Start is currently based on Fedora Core 5 Test 2 and some things may change between now and the final release of Fedora Core 5

System Requirements

  • Your system must use GRUB, the default boot loader for Fedora [[FootNote(This is required because you actually boot the Xen hypervisor and it then starts the Linux kernel. It does this using the MultiBoot standard.)]
  • 256 MB of RAM with the default setup [[FootNote(You can conceivably get by with less, but you'll have to reduce the dom0_memory line in /boot/grub/grub.conf. The Xen hypervisor itself requires ~32 MB. Plus the memory required for your guest systems.)]
  • Sufficient storage space for the guest operating systems. A minimal command-line Fedora system requires around 600MB of storage, a standard desktop Fedora system requires around 3GB.
  • SELinux must not be enforcing it's policy, or some of the operations below will fail. Use system-config-securitylevel to set SELinux to not enforce, and reboot to let this change take effect.

Conventions

Commands which require root privileges are prefixed with the character '#'. To become root, issue the command 'su' as a normal user and supply the root password. Alternatively, use another system such as 'sudo' if you have it setup.

Installing the Xen Software

Before we start we need to set up the basics of the Xen system. To install the main software, type the command:


The xen package contains the Xen hypervisor and tools needed to run virtual machines on x86 systems; kernel-xen-hypervisor is the Xen-enabled host system kernel (domain 0) and kernel-xen-guest is the Xen guest system kernel.

Once this is done, you should have an entry set up in the file /boot/grub/grub.conf for booting the xen-hypervisor kernel. The xen-hypervisor kernel is not set as the default boot option.

To set GRUB to boot with xen-hypervisor by default, edit /boot/grub/grub.conf and set the default to the xen-hypervisor kernel:

default=2
Note.png
The above assumes that xen-hypervisor is the third entry in /boot/grub/grub.conf, these entries are numbered starting from zero.
Warning.png
If the graphical boot screen or X-Windows fails on the next bootup, you may need to disable the graphical boot (rhgb) and switch to using runlevel 3 (command-line only mode).

This is an example /boot/grub/grub.conf, with the init 3 option to start the host system on runlevel 3:

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora Core (2.6.15-1.29_FC5hypervisor)
root (hd0,0)
kernel /xen.gz-2.6.15-1.29_FC5
module /vmlinuz-2.6.15-1.29_FC5hypervisor ro root=/dev/VolGroup01/LogVol00 init 3
module /initrd-2.6.15-1.29_FC5hypervisor.img

Enabling Xen

Once you've rebooted with the xen-hypervisor GRUB option, you should be running with a Xen-enabled kernel [[FootNote(To check that you are running a xen-hypervisor kernel use 'uname -r' - the name of the kernel should contain xen-hypervisor)] .

Now start the Xen service:


To check that Xen is running successfully, enter the following command:


You should see Domain-0 (the Xen-enabled host) running:

[root@xen-hypervisor ~] # xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      940     1 r-----   267.8

To enable the Xen service to automatically start on bootup, enter the following:


Creating A Guest System

These instructions set up a simple base Fedora installation as a Xen guest system.


Create a Virtual Disk for the Guest System

{{Admon/warning | This example system uses the kernel from the kernel-guest package, as the kernel for your guest domain must currently be kept outside of the guest itself.

First, create an image file to use as the storage for our Fedora guest system. This acts as a virtual hard drive for the guest system. This command creates an image file with a size of 1 GB. To create a larger filesystem increase the seek= value.

dd if=/dev/zero of=fedora-guest1.img bs=1M count=1 seek=1024

Now, format the image file by creating an ext3 filesystem on it.

/sbin/mke2fs -F -j fedora-guest1.img

You should now be able to mount the image on a temporary mountpoint, say /mnt


Setup the Base Environment

To install a guest system on to the image we need to set up some basic files on the filesystem of the image that have to be different for Xen (and for the system installation with yum).

First up create some diectories that we will need in the following steps:


Create some required device nodes in the image. [[FootNote(Using an initrd and thus udev seems to work as well with a Fedora Core development tree as of 2005-09-16. Just run depmod and mkinitrd from xen-hypervisor for the xen-guest kernel; make sure to install a xen-guest kernel into the guest system; perform a special depmod run as this is still missing from the kernel rpm.)]  :


Create the guests /etc/fstab file inside the image (i.e. /mnt/etc/fstab), something simple like the following should work:

/dev/sda1               /                       ext3    defaults 1 1
none                    /dev/pts                devpts  gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults 0 0
none                    /proc                   proc    defaults 0 0
none                    /sys                    sysfs   defaults 0 0

Finally, make sure /sys/ is mounted inside the image so that mkinitrd won't fail



Configure YUM

We need to set up a temporary configuration for yum so we can install our guest system. Create a yum-install.conf file that we will use to find our repositories, this file can be created in a temporary location as it will not be needed after we successfully install our guest.

For a stable release the contents of the yum-install.conf file should contain:

[main] 
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
reposdir=/dev/null

[base] 
name=Fedora Core 5 - $basearch - Base
mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-core-5
enabled=1

[updates-released] 
name=Fedora Core 5 - $basearch - Released Updates
mirrorlist=http://fedora.redhat.com/download/mirrors/updates-released-fc5
enabled=1
Warning.png
At the time of writing Fedora Core 5 has not been released and the above configuration will not yet work as the mirrorlist files and repositories will not exist

For a rawhide development system the contents of the yum-install.conf file should instead contain:

[main] 
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
reposdir=/dev/null

[development] 
name=Fedora Core Development Tree
mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-core-rawhide
enabled=1

If you want to use a local Fedora mirror, then you'll want to set the baseurl= parameter and comment out the mirrorlist= parameter before using it; otherwise, it will point to the default set of mirror Websites.

Install a Base Operating System

Firstly we need to install the kernel-xen-guest package into the image. This will pull in dependencies that total to a bit over 60 MiB.


After the the kernel-xen-guest package and dependencies have installed you should update the initrd:


Then, decide which package group(s) you want to install. Start with Core and Base (or for the space constrained just Core, however this is more difficult as you will be missing utilities such as yum).

Note.png
NOTE
In Fedora Core 4 installing Base would pull in the Core group as a dependency. As of Fedora Core 5 this is no longer the case.

Install the guest system with the package groups by running yum. For example, to install with Core and Base use the command:


Grab a coffee and have a snack, it's going to take a little while depending on your connection speed to the yum repository :-)

Note.png
NOTE
SELinux must be disabled to do this. SELinux does path based checks for whether or not things are allowed which do not apply when you're doing installs into a chroot environment.

Come back and if everything went okay, you'll have a minimal install in /mnt.

Do any other configuration you want to on the file system and then unmount it:


Registering a Guest System as a Domain

Now, we just have to create a configuration file in /etc/xen/. As a simple example, the following configuration file is /etc/xen/fedora-guest1:

kernel ="/boot/vmlinuz-2.6.15-1.29_FC5guest"
memory = 64
name = "fedora-guest1"
vif = [''] 
disk = ['file:/root/fedora-guest1.img,sda1,w'] 
root = "/dev/sda1"
extra = "ro selinux=0 3"

Substitute /boot/vmlinuz-2.6.15-1.29_FC5guest with the xen-guest kernel you installed. Additionally, the device sda1 listed here as the disk is related to the /etc/fstab you created above.

Before actually creating the guest domain you may want to decrease the amount of memory being allocated for your host system (domain0), in order to leave adequate room in RAM for your guest domains. This is especially important as the Xen system won't page your guest domains to your host domain's swap space. Constraining the RAM space used by your host domain can help assure that adequate RAM is available for your guests.

The way to do this is to run the following commands as root:

xm mem-max 0 384
xm mem-set 0 384

The mem-max command restricts the host domain (the 0 parameter signifies domain 0) from ever increasing its memory usage beyond 384 megabytes. The mem-set command forces the host domain to immediately shrink its memory allocation down to 384 megabytes.

It may take a bit of time for the domain 0 kernel to page out memory over the new limit, so you should give it a minute or so to complete.

When this is done, your system should have enough RAM free to allow you allocate one or more pretty good sized guest domains.

Now, create a new domain:

xm create -c fedora-guest1

You should now see the ever familiar boot process, at the end you should see the login prompt at which point you can login as 'root' and begin playing around.

Warning.png
If you don't have enough free RAM available to create your guest domain, you will get a message similar to Error: Error creating domain: (2, 'No such file or directory'). This means you need to be sure to use the xm mem-max and xm mem-set commands, above.
Note.png
NOTE about missing shell errors
If you get errors about a missing shell, despite /etc/passwd and /bin/bash (or whatever) being present, you need to disable SELinux for this new domain, as well as domain 0.

If you want to disconnect from the running Xen guest console, this can be done with the key control key combination:

^ 

To reconnect to any running xen guest console type:

xm console fedora-guest1

To restart the guest domain can be done by connecting to the host domain with another window and running the following:

xm shutdown fedora-guest1
xm create -c fedora-guest1

This is pretty early and rough, but it's enough to starting playing with. The next step for development is getting to where you can do actual installations in a Xen guest environment and then being able to boot kernels which are on the guest's file system.

Automatically Loading Domains On Startup

If you want your domains to automatically start when rebooting the system make sure that you enable the xendomains service, and place the Xen configuration file for the domains in the /etc/xen/auto/ directory.

To enable the xendomains service:


Note that the /etc/init.d/xendomains script, as of RPM xen-3.0-0.20050912.fc4, seems to be broken with respect to its use of the LSB init functions. This is arguably due to a bug in Fedora's LSB support, and has been filed in Fedora's bugzilla as bug 171052 and bug 171056 . The latter bug includes a modified version of the /etc/init.d/xendomains script as an attachment which you can use instead.

Troubleshooting

If you're running into issues with your Xen installation there are two log files on the host system to help you out. The file /var/log/xend.log holds the same information as you receive when running 'xm log', and contains basic information. Unfortunately these log messages are often very short and contain little useful information. The following is the output of trying to create a domain running the kernel for NetBSD/xen.

[2005-06-27 02:23:02 xend]  ERROR (SrvBase:163) op=create: Error creating domain:(0, 'Error')
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvBase.py", line 107, in _perform
val = op_method(op, req)
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line 71, in op_create
raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: (0, 'Error')

The second file, /var/log/xend-debug.log usually contains much more detailed information. Trying to start the NetBSD/xen kernel will result in the following log output:

ERROR: Will only load images built for Xen v3.0
ERROR: Actually saw: 'GUEST_OS=netbsd,GUEST_VER=2.0,XEN_VER=2.0,LOADER=generic,BSD_SYMTAB'
ERROR: Error constructing guest OS

If you are reporting errors or need help with your Xen setup you should always include both the xend.log and xend-debug.log messages that you receive.

Q: I am trying to start then xend service and nothing happens, then when I do a xm list I get the following:

Error: Error connecting to xend: Connection refused.  Is xend running?

Alternatively, I run xend start manually and get the following error:

ERROR: Could not obtain handle on privileged command interface (2 = No such file or directory)
Traceback (most recent call last):
File "/usr/sbin/xend", line 33, in ?
from xen.xend.server import SrvDaemon
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line 21, in ?
import relocate
File "/usr/lib/python2.4/site-packages/xen/xend/server/relocate.py", line 26, in ?
from xen.xend import XendDomain
File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line 33, in ?
import XendDomainInfo
File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 37, in ?
import image
File "/usr/lib/python2.4/site-packages/xen/xend/image.py", line 30, in ?
xc = xen.lowlevel.xc.xc()
RuntimeError: (2, 'No such file or directory')

A: You have rebooted your host into a kernel that is not a xen-hypervisor kernel. Yes I even did this myself in testing :)

You either need to select the xen-hypervisor kernel in the grub menu at boot time or set the xen-hypervisor kernel as default in your grub.conf file.

Getting Help

If the Troubleshooting section above does not help you to solve your problem, check the Red Hat Bugzilla for existing bug reports on Xen in FC5. The product is "Fedora Core", and the component is "Xen". These reports contain useful advice from fellow xen testers and often describe work-arounds.

Discussion on Fedora Xen support issues occur on the Fedora Xen mailing list

The Xen 3 User Manual can be found at http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user/user.html

For general Xen issues and useful information check the Xen project documentation , and mailing list archives .

Footnotes