From Fedora Project Wiki
(Created page with "{{admon/important | Comments and Explanations | The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To ...")
 
No edit summary
 
(31 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{admon/important | Comments and Explanations | The page source contains comments providing guidance to fill out each section.  They are invisible when viewing this page.  To read it, choose the "edit" link.<br/> '''Copy the source to a ''new page'' before making changes!  DO NOT EDIT THIS TEMPLATE FOR YOUR FEATURE.'''}}
{{admon/important | Set a Page Watch| Make sure you click ''watch'' on your new page so that you are notified of changes to it by others, including the Feature Wrangler}}
{{admon/note | All sections of this template are required for review by FESCo.  If any sections are empty it will not be reviewed }}
<!-- All fields on this form are required to be accepted by FESCo.
We also request that you maintain the same order of sections so that all of the feature pages are uniform.  -->


<!-- The actual name of your feature page should look something like: Features/Your_Feature_Name.  This keeps all features in the same namespace -->
<!-- The actual name of your feature page should look something like: Features/Your_Feature_Name.  This keeps all features in the same namespace -->
Line 15: Line 6:
== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this feature is and what it will do.  This information is used for the overall feature summary page for each release. -->
<!-- A sentence or two summarizing what this feature is and what it will do.  This information is used for the overall feature summary page for each release. -->
Multiqueue virtio-net provides an approach that scales the network performance as the increasing of the number of vcpus by allowing them to transfer packets through more than one virtqueue pairs.


== Owner ==
== Owner ==
<!--This should link to your home wiki page so we know who you are-->
<!--This should link to your home wiki page so we know who you are-->
* Name: [[User:jasowang| Jason Wang]]
* Name: [[User:jasowang| Jason Wang]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your feature, status is requested, or  technical issues need to be resolved-->
* Email: <jasowang@redhat.com>
* Email: <jasowang@redhat.com>
* Name: [[User:crobinso| Cole Robinson]]
* Email: <crobinso@redhat.com>


== Current status ==
== Current status ==
* Targeted release: [[Releases/<number> | Fedora 19 ]]
* Targeted release: Fedora 20
* Last updated: (DATE)
* Last updated: 2013-03-15
* Percentage of completion: XX%
* Percentage of completion: 66%


<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
== Detailed Description ==
== Detailed Description ==
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
Today's high-end server have more processors, guests running on them tend have an increasing number of vcpus. The scale of the protocol stack in guest in restricted because of the single queue virtio-net:
* The network performance does not scale as the number of vcpus increasing: Guest can not transmit or retrieve packets in parallel as virtio-net have only one TX and RX, virtio-net drivers must be synchronized before sending and receiving packets. Even through there's software technology to spread the loads into different processor such as RFS, such kind of method is only for transmission and is really expensive in guest as they depends on IPI which may brings extra overhead in virtualized environment.
* Multiqueue nic were more common used and is well supported by linux kernel, but current virtual nic can not utilize the multi queue support: the tap and virtio-net backend must serialize the co-current transmission/receiving request comes from different cpus.
In order the remove those bottlenecks, we must allow the paralleled packet processing by introducing multi queue support for both back-end and guest drivers. Ideally, we may let the packet handing be done by processors in parallel without interleaving and scale the network performance as the number of vcpus increasing.
The following parts were changed to parallize the packet processing:
* tuntap: convert the driver to multiqueue by allowing multiple socket/fd to be attached to the device, each socket/fd exposed by the device could be treated as a queue.
* qemu:
** net: Add multiple queue infrastructure to qemu
** let qemu can create multiple vhost threads for a virtio-net device
** userspace multiple queue virtio-net
* guest driver: let the driver can use multiple virtqueues to do packet sending/receiving.


== Benefit to Fedora ==
== Benefit to Fedora ==
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
Improve the performance of the virtio-net with smp guests on Fedora Host.


== Scope ==
== Scope ==
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
* tuntap driver in kernel (DONE, in 3.8-rc)
* guest virtio-net driver (DONE, in 3.8-rc)
* qemu changes (DONE, in qemu 1.4)
* libvirt (Not done)
* virt-manager (optional, Not done)


== How To Test ==
== How To Test ==
Line 53: Line 66:
3. What are the expected results of those actions?
3. What are the expected results of those actions?
-->
-->
* hardware requirement: no special requirements but better have a multiqueue ethernet card to test
* qemu command line (e.g. you want to start a guest with 4 queues)
** qemu -netdev tap,id=hn0,queues=4,vhost=on -device virtio-net-pci,netdev=hn0,mq=on,vectors=9
* test cases:
** boot & reboot
*** boot a guest with 2 queues then reboot
** enable multiqueue support in guest
*** boot a guest with 2 queues
*** enable the multiqueue support by ethtool -L $interface combined 2
*** check whether 2 txqs and 2 rxqs are existed in /sys/class/net/$interface/queues
*** run 20 cocurrent netperf test in guest: for i in `seq 20` do netperf -H $ip -t TCP_RR &; done
*** check packets goes to each queue by check interrupt are distrucited to each queue through /proc/interrupts in guest
*** change back to single queue mode by ethtool -L $interface combined 1
*** measure the 4 cocurrent netperf test in test again
** migration test
*** boot a src vm with 2 queues
*** boot a dst vm with 2 queues
*** enable multiqueue in src guest by ethtool -L $interface combined 2
*** scp a file from host or external host to a src vm
*** migrate the vm from src to dst
*** scp should finish as usual without any error
** set_link test
*** boot a guest with 2 queues
*** enable multiqueue in guest by ethtool -L $interface combined 2
*** use set_link to off the link in qemu monitor
*** check the network link is down in guest
*** use set_link to on the link in qemu monitor
*** check the network link is up in guest
** hotplug test
*** boot a guest with 2 queues
*** use netdev_del and device_del in qemu monitor to delete the virtio-net device
*** check in guest and monitor (info network) that the device is removed
*** use netdev_add tap,id=hn1,queues=2,vhost=on and device_add in the monitor to hot add a device in guest
*** enable the multiqueue support in the new added deivce by ethtool -L $interface combined 2
*** run 20 cocurrent netperf test in guest: for i in `seq 20` do netperf -H $ip -t TCP_RR &; done
*** check packets goes to each queue by check interrupt are distrucited to each queue through /proc/interrupts in guest
** pktgen stress test
*** boot a guest with 2 queues
*** enable the multiqueue support in the new added deivce by ethtool -L $interface combined 2
*** use pktgen to generate the load on both queues in guest
*** use pktgen to generate the load on both queues in host tap device
** other stress test
*** boot and enable multiqueue in guest
*** ordinary network stress test such as apache ab, netperf, stress ...


== User Experience ==
== User Experience ==
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
The perfomrance of network application/server which use multiple sessions in parallel will be improved.


== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
None


== Contingency Plan ==
== Contingency Plan ==
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
Since this is brand new functionality, if it isn't ready in time, nothing has changed. We just drop this feature page.


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
*
* [http://www.linux-kvm.org/page/Multiqueue Docs in kvm wiki]
* [https://lkml.org/lkml/2012/12/7/272 mq virtio-net patchset]
* [https://lkml.org/lkml/2012/11/1/13 mq tuntap patchset]
* [http://lists.gnu.org/archive/html/qemu-devel/2013-01/msg04719.html mq qemu patchset]
* [https://www.redhat.com/archives/libvir-list/2013-March/msg00533.html Libvirt semi-proposal (march 12)]


== Release Notes ==
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
*
* KVM/qemu in Fedora can start a guest with multiqueue virtio-net support. And a Fedora guest have multiqueue virtio-net driver.


== Comments and Discussion ==
== Comments and Discussion ==
* See [[Talk:Features/Your_Feature_Name]]  <!-- This adds a link to the "discussion" tab associated with your page. This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
None yet.
 


[[Category:FeaturePageIncomplete]]
[[Category:FeatureReadyForWrangler]]
<!-- When your feature page is completed and ready for review -->
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 09:56, 10 August 2015


Multiqueue virtio-net

Summary

Multiqueue virtio-net provides an approach that scales the network performance as the increasing of the number of vcpus by allowing them to transfer packets through more than one virtqueue pairs.

Owner

Current status

  • Targeted release: Fedora 20
  • Last updated: 2013-03-15
  • Percentage of completion: 66%

Detailed Description

Today's high-end server have more processors, guests running on them tend have an increasing number of vcpus. The scale of the protocol stack in guest in restricted because of the single queue virtio-net:

  • The network performance does not scale as the number of vcpus increasing: Guest can not transmit or retrieve packets in parallel as virtio-net have only one TX and RX, virtio-net drivers must be synchronized before sending and receiving packets. Even through there's software technology to spread the loads into different processor such as RFS, such kind of method is only for transmission and is really expensive in guest as they depends on IPI which may brings extra overhead in virtualized environment.
  • Multiqueue nic were more common used and is well supported by linux kernel, but current virtual nic can not utilize the multi queue support: the tap and virtio-net backend must serialize the co-current transmission/receiving request comes from different cpus.

In order the remove those bottlenecks, we must allow the paralleled packet processing by introducing multi queue support for both back-end and guest drivers. Ideally, we may let the packet handing be done by processors in parallel without interleaving and scale the network performance as the number of vcpus increasing.

The following parts were changed to parallize the packet processing:

  • tuntap: convert the driver to multiqueue by allowing multiple socket/fd to be attached to the device, each socket/fd exposed by the device could be treated as a queue.
  • qemu:
    • net: Add multiple queue infrastructure to qemu
    • let qemu can create multiple vhost threads for a virtio-net device
    • userspace multiple queue virtio-net
  • guest driver: let the driver can use multiple virtqueues to do packet sending/receiving.

Benefit to Fedora

Improve the performance of the virtio-net with smp guests on Fedora Host.

Scope

  • tuntap driver in kernel (DONE, in 3.8-rc)
  • guest virtio-net driver (DONE, in 3.8-rc)
  • qemu changes (DONE, in qemu 1.4)
  • libvirt (Not done)
  • virt-manager (optional, Not done)

How To Test

  • hardware requirement: no special requirements but better have a multiqueue ethernet card to test
  • qemu command line (e.g. you want to start a guest with 4 queues)
    • qemu -netdev tap,id=hn0,queues=4,vhost=on -device virtio-net-pci,netdev=hn0,mq=on,vectors=9
  • test cases:
    • boot & reboot
      • boot a guest with 2 queues then reboot
    • enable multiqueue support in guest
      • boot a guest with 2 queues
      • enable the multiqueue support by ethtool -L $interface combined 2
      • check whether 2 txqs and 2 rxqs are existed in /sys/class/net/$interface/queues
      • run 20 cocurrent netperf test in guest: for i in seq 20 do netperf -H $ip -t TCP_RR &; done
      • check packets goes to each queue by check interrupt are distrucited to each queue through /proc/interrupts in guest
      • change back to single queue mode by ethtool -L $interface combined 1
      • measure the 4 cocurrent netperf test in test again
    • migration test
      • boot a src vm with 2 queues
      • boot a dst vm with 2 queues
      • enable multiqueue in src guest by ethtool -L $interface combined 2
      • scp a file from host or external host to a src vm
      • migrate the vm from src to dst
      • scp should finish as usual without any error
    • set_link test
      • boot a guest with 2 queues
      • enable multiqueue in guest by ethtool -L $interface combined 2
      • use set_link to off the link in qemu monitor
      • check the network link is down in guest
      • use set_link to on the link in qemu monitor
      • check the network link is up in guest
    • hotplug test
      • boot a guest with 2 queues
      • use netdev_del and device_del in qemu monitor to delete the virtio-net device
      • check in guest and monitor (info network) that the device is removed
      • use netdev_add tap,id=hn1,queues=2,vhost=on and device_add in the monitor to hot add a device in guest
      • enable the multiqueue support in the new added deivce by ethtool -L $interface combined 2
      • run 20 cocurrent netperf test in guest: for i in seq 20 do netperf -H $ip -t TCP_RR &; done
      • check packets goes to each queue by check interrupt are distrucited to each queue through /proc/interrupts in guest
    • pktgen stress test
      • boot a guest with 2 queues
      • enable the multiqueue support in the new added deivce by ethtool -L $interface combined 2
      • use pktgen to generate the load on both queues in guest
      • use pktgen to generate the load on both queues in host tap device
    • other stress test
      • boot and enable multiqueue in guest
      • ordinary network stress test such as apache ab, netperf, stress ...

User Experience

The perfomrance of network application/server which use multiple sessions in parallel will be improved.

Dependencies

None

Contingency Plan

Since this is brand new functionality, if it isn't ready in time, nothing has changed. We just drop this feature page.

Documentation

Release Notes

  • KVM/qemu in Fedora can start a guest with multiqueue virtio-net support. And a Fedora guest have multiqueue virtio-net driver.

Comments and Discussion

None yet.