From Fedora Project Wiki

Libvirt, bridging and bonding - a primer

Introduction

Recently, Adam Williamson wrote up a nice long blog post on getting bridging to work with the current state of Fedora. His write up is targeted at desktop users, so he didn't cover two areas - he uses the GUI to configure some of it, and he does not address bonding (which is generally only required for a server, anyhow). So, I dove in to solve this problem for a headless server that is doing some virtualization on the side. We will be using nmcli to configure our bond, bridge, and physical ethernet devices. On the example system, I also have a management interface for out-of-band access to the system.

Procedure

User: root

First, you will need to be root for this - it just makes it that much easier.

$ sudo su -

Example non-working configuration

When I started down this path, my networking was severely mis-configured - nothing really was working properly. It had reached this point by using the intuitive tools provided by Virtual Machine Manager (VMM) and virsh. Once I worked through the "suggested" process with those tools, I ended up with a broken network.

# nmcli con show conf
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:04:46 PM CDT 
System em1          1dad842d-1912-ef5a-a43a-bc238fb267e7  802-3-ethernet  never                           
System p4p1         5dd47203-fffb-671a-4fd0-4cff98347a3b  802-3-ethernet  never                           
System bond0        ad33d8b0-1f7b-cab9-9447-ba07f855b143  802-3-ethernet  never                           
Wired connection 2  f5984df2-83d8-42e9-bd8c-7830680cb708  802-3-ethernet  Mon 04 Aug 2014 05:39:46 PM CDT 
Bridge br1          2ee981ca-5ff4-4f9b-03fe-32879aa3dc85  bridge          Mon 04 Aug 2014 05:47:56 PM CDT 
kvm-br0             6338b494-9e37-4677-b3e4-837938cfd860  bridge          Mon 04 Aug 2014 04:17:38 PM CDT 

# brctl show
bridge name     bridge id               STP enabled     interfaces
nm-kvm-br0              8000.000000000000       no
virbr0          8000.000000000000       yes

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:76ff:fef4:5c38/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::223:aeff:feab:2054/64 scope link 
       valid_lft forever preferred_lft forever
4: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1b:21:ab:d6:a1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::21b:21ff:feab:d6a1/64 scope link 
       valid_lft forever preferred_lft forever
5: nm-kvm-br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether 82:2c:4a:76:7f:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::802c:4aff:fe76:7f16/64 scope link 
       valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 12:27:18:b3:35:1d brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

# virsh iface-list 
Name                 State      MAC Address
--------------------------------------------
lo                   active     00:00:00:00:00:00

As you can see, the only IP address I had was my management interface, and libvirt didn't see any interfaces at all. So, I decided to start from scratch.

Reset

The first step is to remove all configuration from the system.

# nmcli con show conf
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:19:46 PM CDT 
System em1          1dad842d-1912-ef5a-a43a-bc238fb267e7  802-3-ethernet  never                           
System p4p1         5dd47203-fffb-671a-4fd0-4cff98347a3b  802-3-ethernet  never                           
System bond0        ad33d8b0-1f7b-cab9-9447-ba07f855b143  802-3-ethernet  never                           
Wired connection 2  f5984df2-83d8-42e9-bd8c-7830680cb708  802-3-ethernet  Mon 04 Aug 2014 05:39:46 PM CDT 
Bridge br1          2ee981ca-5ff4-4f9b-03fe-32879aa3dc85  bridge          Mon 04 Aug 2014 05:47:56 PM CDT 
kvm-br0             6338b494-9e37-4677-b3e4-837938cfd860  bridge          Mon 04 Aug 2014 04:17:38 PM CDT 

# nmcli con delete 6338b494-9e37-4677-b3e4-837938cfd860

# nmcli con delete 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85

# nmcli con delete ad33d8b0-1f7b-cab9-9447-ba07f855b143

# nmcli con delete 5dd47203-fffb-671a-4fd0-4cff98347a3b

# nmcli con delete 1dad842d-1912-ef5a-a43a-bc238fb267e7

That deleted all of my current network configuration except for my management interface.

Verify starting point

# nmcli con show conf 
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:24:46 PM CDT 

So, now I have a clean slate to work with. From here, I want to verify that my management interface is configured properly and won't interfere with the real task at hand:

# nmcli con show conf 56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.id:                          Wired connection 1
connection.uuid:                        56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.interface-name:              --
connection.type:                        802-3-ethernet
connection.autoconnect:                 yes
connection.timestamp:                   1407194686
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          yes
802-3-ethernet.mac-address:             00:04:76:F4:5C:38
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.mac-address-blacklist:   
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            
ipv4.method:                            manual
ipv4.dns:                               
ipv4.dns-search:                        
ipv4.addresses:                         { ip = 10.0.0.3/24, gw = 0.0.0.0 }
ipv4.routes:                            
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.never-default:                     yes
ipv4.may-fail:                          yes

the key thing we were looking for there were ipv4.never-default: yes and ipv4.may-fail: yes. This ensures that the system does not try to use the management network as it's default route, and that if the connection isn't present it won't affect the system.

Finally, reboot (if possible) to clean every thing up.

Configuring the interfaces

This is where it gets tricky. The key thing to remember here is that the physical interfaces are subordinate to the bond, which is subordinate to the bridge. So, we configure the bridge first.

Interface Chain
System -> bridge -> bond -> physical -> network

Bridge interface

The bridge is the main interface to the system. This is what will hold the IP information and be presented to the kernel network stack and libvirt.

nmcli usage
I am showing everything on one line here for brevity's sake, but you can also stop after defining the con-name or if-name, then entering the nmcli shell to finish the configuration
# nmcli con add con-name br0 ifname br0 type bridge autoconnect yes stp off ip4 172.16.4.3/24 gw4 172.16.4.1
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) successfully added.

# nmcli con show ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.id:                          br0
connection.uuid:                        ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.interface-name:              br0
connection.type:                        bridge
connection.autoconnect:                 yes
connection.timestamp:                   0
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
ipv4.method:                            manual
ipv4.dns:                               172.16.4.2
ipv4.dns-search:                        
ipv4.addresses:                         { ip = 172.16.4.3/24, gw = 172.16.4.1 }
ipv4.routes:                            
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.never-default:                     no
ipv4.may-fail:                          yes
bridge.interface-name:                  br0
bridge.stp:                             no
bridge.priority:                        128
bridge.forward-delay:                   15
bridge.hello-time:                      2
bridge.max-age:                         20
bridge.ageing-time:                     300

Bond Interface

Bonding is used on servers to provide a higher bandwidth, fault tolerance, and/or load balancing. To get the benefit of the faster bandwidth, a switch that supports 802.3AD Link Aggregation is required. To take advantage of the fault tolerance or certain load balancing options, specific hardware is not required. For this example, I am setting up an 802.3AD aggregate link to my switch in order to establish a 2Gbps circuit to the server.

# nmcli con add con-name bond0 ifname bond0 type bond autoconnect yes mode 802.3ad miimon 100
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) successfully added.

Now the command line options for the bond configuration do not allow me to set it as a bridged interface, so we will need to drop into the nmcli shell to finish up the config.

# nmcli con edit 58fe56dd-a30b-483a-9e35-3727ccc6ab0e

===| nmcli interactive connection editor |===

Editing existing 'bond' connection: '58fe56dd-a30b-483a-9e35-3727ccc6ab0e'

Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.

You may edit the following settings: connection, bond, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> print
===============================================================================
                              Connection details
===============================================================================
connection.id:                          bond0
connection.uuid:                        58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name:              bond0
connection.type:                        bond
connection.autoconnect:                 yes
connection.timestamp:                   0
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
bond.interface-name:                    bond0
bond.options:                           miimon=100,mode=balance-rr
-------------------------------------------------------------------------------
nmcli> set connection.master br0
nmcli> set connection.slave-type bridge
nmcli> save
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) sucessfully saved.
nmcli> quit

You now have a bond interface bridged with the bridge interface we set up previously. Now we need to add the physical interfaces to the bond.

Physical Interfaces

Bonded interfaces are pseudo interfaces just like bridges - they need to have physical interfaces slaved to them in order to be operational. Here we will do this all from the command line. You need to know what device name udev gave the interfaces, the easiest way to do this is with the old ifconfig command - it will spit out the udev device names as well as any configuration already applied. Using the udev device name of your physical interfaces, add them as slaves to the bond0 interface we just created.

# nmcli con add con-name eth0 ifname em1 type bond-slave master bond0
Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added.

# nmcli con add con-name eth1 ifname p4p1 type bond-slave master bond0
Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.

Verification and Activation

The configuration should be complete at this point. To verify our setup, we will use nmcli, brctl, ip address, and <virsh>.

# nmcli con show conf
NAME   UUID                                  TYPE            TIMESTAMP-REAL                  
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:35:20 PM CDT 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  802-3-ethernet  never                           
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  802-3-ethernet  never                           
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            never                           
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          never                           

# nmcli con up br0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)

# nmcli con up bond0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)

# nmcli con up eth0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)

# nmcli con up eth1
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/9)

# nmcli con show act
NAME   UUID                                  DEVICES    DEFAULT  VPN  MASTER-PATH                               
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  p2p1       no       no   --                                        
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  nm-bridge  yes      no   --                                        
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  nm-bond    no       no   /org/freedesktop/NetworkManager/Devices/5 
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  em1        no       no   /org/freedesktop/NetworkManager/Devices/6 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  p4p1       no       no   /org/freedesktop/NetworkManager/Devices/6 

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:76ff:fef4:5c38/64 scope link 
       valid_lft forever preferred_lft forever
3: em1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
4: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master nm-bridge state UP group default 
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a8e4:e3ff:fe2b:12af/64 scope link 
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet 172.16.4.3/24 brd 172.16.4.255 scope global nm-bridge
       valid_lft forever preferred_lft forever
    inet6 fe80::5c18:76ff:fef5:e166/64 scope link 
       valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 42:59:71:b6:95:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             0080.0023aeab2054       no              bond0
virbr0          8000.000000000000       yes

# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
lo                   active     00:00:00:00:00:00
br0                  active     00:23:ae:ab:20:54

Libvirt is now bridged with the bonded interfaces to the network, 2GB pathway to and from the virtual machines to be built.