From Fedora Project Wiki
(first cut at this page)
 
(→‎How To Test: link to test day docs)
 
(18 intermediate revisions by 5 users not shown)
Line 10: Line 10:


* Targeted release: [[Releases/11|Fedora 11]]
* Targeted release: [[Releases/11|Fedora 11]]
* Last updated: 2009-01-21
* Last updated: 2009-03-11
* Percentage of completion: 60%
* Percentage of completion: 100%
 
=== TODO ===
 
# Ability to assign devices in virt-manager
# Ability to use assigned devices for guest installation in virtinst
# Filter host devices to only include devices which may be assigned (check FLR capability, whether behind bridge, non-page-aligned MMIO BARs)
# Before assignment, reset the device, unbind its driver and bind <code>pci-stub.ko</code>


=== Completed ===
=== Completed ===
Line 24: Line 17:
# Support in <code>kvm.ko</code> and <code>qemu-kvm</code>
# Support in <code>kvm.ko</code> and <code>qemu-kvm</code>
# <code>pci-stub.ko</code> to reserve host devices
# <code>pci-stub.ko</code> to reserve host devices
# libvirt host device enumeration and assignment
# libvirt host device enumeration and assignment [in libvirt 0.6.1]
# libvirt host device assignment
# libvirt host device assignment [in libvirt 0.6.1]
# libvirt patches for dettach/reattach [in libvirt 0.6.1]
# [https://bugzilla.redhat.com/487103 Backport remove_id]
# Before assignment, unbind its driver, bind to <code>pci-stub.ko</code> and reset the device. [in libvirt 0.6.1]
# Ability to use assigned NICs for guest installation in virtinst [in 0.400.2 release]
# Ability to assign NICs in virt-manager [Expected before beta release] [in 0.7.1 release]


== Detailed Description ==
== Detailed Description ==
Line 37: Line 35:
* All of the guest's memory must kept permanently in memory. This is because the guest may program the device with any address in its address space and the hypervisor has no way of handling a DMA page fault.
* All of the guest's memory must kept permanently in memory. This is because the guest may program the device with any address in its address space and the hypervisor has no way of handling a DMA page fault.
* It isn't possible to migrate the guest to another host. Even if the exact same hardware exists on the remote host, it is impossible for the  hypervisor to migrate the device state between hosts.
* It isn't possible to migrate the guest to another host. Even if the exact same hardware exists on the remote host, it is impossible for the  hypervisor to migrate the device state between hosts.
* Graphics cards cannot currently be assigned because they require access to the videio BIOS.
* Graphics cards cannot currently be assigned because they require access to the video BIOS.


In summary, PCI device assignment is possible given the appropriate hardware, but it is only suitable in certain situations where the flexibility of memory over-commit and migration is not required.
In summary, PCI device assignment is possible given the appropriate hardware, but it is only suitable in certain situations where the flexibility of memory over-commit and migration is not required.
Line 51: Line 49:
The core device assignment support in the kernel includes:
The core device assignment support in the kernel includes:


  * IOMMU API
* IOMMU API
  * VT-d and AMD IOMMU support
* VT-d and AMD IOMMU support
  * Device assignment in kvm.ko
* Device assignment in kvm.ko


Support is also required in <code>qemu-kvm</code>, the userspace component of KVM.
Support is also required in <code>qemu-kvm</code>, the userspace component of KVM.
Line 73: Line 71:
# Start the guest install process, check that it completes successfully
# Start the guest install process, check that it completes successfully
# Check the newly installed guest reboots and has access to the assigned network card
# Check the newly installed guest reboots and has access to the assigned network card
Also see: Procedures from [[Test_Day:2009-05-07_Virtualization_KVM_PCI_Device_Assignment|Virtualization Test Day]].


Notes:
Notes:
Line 85: Line 85:
== Dependencies ==
== Dependencies ==


Outside of the changes described above for the kernel, kvm, libvirt, python-virtinst and virt-manager, the only other changes which should be needed is to fix any devices drivers which fail to reset devices correctly and leave devices in a broken state.
The changes described above are for the kernel, kvm, libvirt, python-virtinst and virt-manager.
 
Ensuring that VT-d support is solid enough that <code>intel_iommu=on</code> is the default.


== Contingency Plan ==
== Contingency Plan ==
Line 102: Line 104:


The [http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/34434.pdf AMD IOMMU spec] is also available.
The [http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/34434.pdf AMD IOMMU spec] is also available.
A [http://marc.info/?l=kvm&m=123454366317045&w=2 post to kvm@vger.kernel.org] describing some of the requirements around device unbinding, reset and filtering. [https://bugzilla.redhat.com/487103 bug #487103] tracks the needed backport of <code>remove_id</code>.
[https://bugzilla.redhat.com/479996 bug #479996] tracks some of the major issues which were fixed with VT-d support.


== Release Notes ==
== Release Notes ==
Line 113: Line 119:
* See [[Talk:Features/KVM PCI Device Assignment]]
* See [[Talk:Features/KVM PCI Device Assignment]]


----
[[Category:Virtualization|KVM PCI Device Assignment]]
 
[[Category:F11 Virt Features|KVM PCI Device Assignment]]
[[Category:Virtualization]]


[[Category:FeaturePageIncomplete]]
[[Category:FeatureAcceptedF11]]
<!-- Category:FeatureReadyForWrangler -->

Latest revision as of 16:52, 17 May 2009

Summary

Assign PCI devices from your KVM host machine to guest virtual machines. A common example is assigning a network card to a guest.

Owner

Current status

  • Targeted release: Fedora 11
  • Last updated: 2009-03-11
  • Percentage of completion: 100%

Completed

  1. Support in kvm.ko and qemu-kvm
  2. pci-stub.ko to reserve host devices
  3. libvirt host device enumeration and assignment [in libvirt 0.6.1]
  4. libvirt host device assignment [in libvirt 0.6.1]
  5. libvirt patches for dettach/reattach [in libvirt 0.6.1]
  6. Backport remove_id
  7. Before assignment, unbind its driver, bind to pci-stub.ko and reset the device. [in libvirt 0.6.1]
  8. Ability to use assigned NICs for guest installation in virtinst [in 0.400.2 release]
  9. Ability to assign NICs in virt-manager [Expected before beta release] [in 0.7.1 release]

Detailed Description

KVM guests usually have access to either virtio devices or emulated devices. If the guest has access to suitable drivers, then virtio is preferred because it allows high performance to be achieved.

On host machines which have Intel VT-d or AMD IOMMU hardware support, another option is possible. PCI devices may be assigned directly to the guest, allowing the device to be used with minimal performance overhead.

However, device assignment is not always the best option even when it is available. Problems include:

  • All of the guest's memory must kept permanently in memory. This is because the guest may program the device with any address in its address space and the hypervisor has no way of handling a DMA page fault.
  • It isn't possible to migrate the guest to another host. Even if the exact same hardware exists on the remote host, it is impossible for the hypervisor to migrate the device state between hosts.
  • Graphics cards cannot currently be assigned because they require access to the video BIOS.

In summary, PCI device assignment is possible given the appropriate hardware, but it is only suitable in certain situations where the flexibility of memory over-commit and migration is not required.

Benefit to Fedora

Fedora users will be able to assign PCI network cards, hard disk controllers, phone line termination cards etc. to their virtual machines.

Device assignment is an important feature for any virtualization platform. As such, the feature will improve Fedora's virtualization standings in any competitive analysis.

Scope

The core device assignment support in the kernel includes:

  • IOMMU API
  • VT-d and AMD IOMMU support
  • Device assignment in kvm.ko

Support is also required in qemu-kvm, the userspace component of KVM.

libvirt, python-virtinst and virt-manager also require support to be added in order to allow users to easily assign devices. Not only is the ability to assign devices needed, but also the ability to list what devices are available to assign.

Further complications include:

  1. Preventing a guest assigned device from being used in the host
  2. Detecting which host devices are assignable
  3. Ensuring that devices are properly reset.

How To Test

Perhaps the most straightforward test is to use virt-manager create a Fedora 11 KVM guest with a PCI network card.

  1. Run virt-manager, click on New and go through the usual process of creating a guest
  2. In the network configuration screen, choose the "assign physical device" option and select an available network card from the list
  3. Start the guest install process, check that it completes successfully
  4. Check the newly installed guest reboots and has access to the assigned network card

Also see: Procedures from Virtualization Test Day.

Notes:

  • You need to have a machine with Intel VT-d or AMD IOMMU support.
  • If you have only one network card on the host machine, the host will not be able to access the network during the test. So, e.g. you would need to local install media to start the installation.

User Experience

The user experience is similar to that described above. Users will be able to easily assign PCI devices to their KVM guests.

Dependencies

The changes described above are for the kernel, kvm, libvirt, python-virtinst and virt-manager.

Ensuring that VT-d support is solid enough that intel_iommu=on is the default.

Contingency Plan

If the tools support is not complete in time, no contingency is needed. The funcionality just wouldn't be available.

However, if the functionality was so broken that we didn't want to expose it to users, we might have to disable the support in the tools - e.g. remove the device assignment UI from virt-manager.

Documentation

Searching the KVM mailing list archives may prove helpful.

Some documentation for the libvirt support is available on the libvirt site.

Wikipedia references some information for Intel VT-d, including the full specification.

The AMD IOMMU spec is also available.

A post to kvm@vger.kernel.org describing some of the requirements around device unbinding, reset and filtering. bug #487103 tracks the needed backport of remove_id.

bug #479996 tracks some of the major issues which were fixed with VT-d support.

Release Notes

Fedora 11 expands its virtualization capabilities to include KVM PCI device assignment support. KVM users can now give virtual machines exclusive access to physical PCI devices using Fedora's virtualization toools, including the Virtual Machine Manager application.

Intel VT-d or AMD IOMMU hardware platform support is required in order for this feature to be available.

Comments and Discussion