QEMU and Tuntap
- QEMU and Tuntap
- Explanation of how tuntap networking driver works
- Explanation by ASCII (and partly configuration examples)
- Possibility 0: routing with one "external" LAN, and multiple "virtual internal" LANs in subnets
- Possibility 1: NAT routing with one "external" LAN, and multiple "virtual internal" LANs in subnets
- Possibility 2: NAT routing with one "external" net, and one "virtual internal" LANs
- Possibility 3: NO routing, one LAN made up of real and virtual devices
- Problems and error messages
- Footnotes
Explanation of how tuntap networking driver works
This is not really qemu related but it should help curious users understand how qemu takes advantage of tuntap. It is also written from the perspective of a Linux/GNU host, however BSD systems should work the same way. They will just have different names for the ethernet devices.
The tuntap driver supports 2 modes: tun and tap:
tap is based on the older ethertap driver and deals with ethernet frames
tun deals with ip frames
qemu only uses tap devices [#Note1].
Accessing a network outside the qemu guest with tuntap
Suppose you set up a single qemu guest to use tuntap networking (with a single emulated network card inside the qemu guest). If you now want to access a network outside of the guest you will need an interface where that network is connected to.
If you only have a local connection, a so called loopback device, then that interface will be {{lo}}. If you will access an external network, then that interface will probably be eth0. Each interface has its own mac address [#Note2] - in the form of 00:40:CA:8A:7E:40 and an IP address like 192.168.1.1. We will call this interface the host nic (where nic means "Network Interface Card").
Qemu's internal nic
qemu sets up its own ethernet device as well, but this one is emulated. The default mac address that qemu uses for its nic, known as the guest nic, is hard coded. It is possible, and recommended if running multiple guests, to change the mac address (by passing the one you want qemu to use via the command line via -net macaddr=11:22:33:44:55:66). It also has its own IP address, but qemu doesn't know what this is - you give it to the guest OS. Within the qemu quest this internal nic will be visible as eth0, not to be confused with the nic of the same name of the host.
The ''tap'' device
There is also a third nic, one that most people forget about once everything is up and running. This is tap0, and it exists on the host. In tuntap mode, a new tap device is created for each instance of qemu, in addition to a new guest nic (so you can have tap0, tap1, tap2, etc.) This is counterintutive at first, since you already have eth0 on the host and the emulated nic in the guest. However, qemu doesn't directly access the hosts eth0. Instead, it manipulates tap0. So host<->guest communication is achieved by connecting tap0 to the emulated guest nic eth0 [#Note3]. Normal tools (such as /sbin/ip or /sbin/route) can then be used to connect eth0 to tap0.
Seeing qemu's nic from the outside - bridging
Why does qemu do this? Good question. There is a big advantage to doing things this way. It is very difficult (if it's possible at all) to set up a nic (such as eth0) to have multiple subnets. So you can't place all qemu guests in one subnet and have eth0 (and the rest of your LAN) in a different subnet, unless you have a separate tap0. The host nic which is connected to the emulated nic must be on the same subnet, so if you want different subnets you set up tap0 and the guest nic on one subnet and eth0 (and the LAN) on a different subnet.
But if you don't want to have separate subnets for eth0 and the guest nic, then what do you do? You aren't supposed to have 2 host nics on the same subnet. The answer is to use kernel bridging. You create br0, connect eth0 and tap0 to br0, and then use br0 as the host nic [#Note4].
You now technically have 4 nics (br0, eth0, and tap0 on the host; and the guest nic) but you only have to think about br0 and the guest nic. eth0 and tap0 effectively don't exist anymore, they only stick around to do the dirty work of the kernel [#Note5].
Making multiple qemu instances visible to each other
Now if each guest creates its own tap device, then how do you get them to to talk to each other? Also, how do you put them on the same subnet? The answer to the first question: if you place each tap device (and guest nic) in its own subnet, then you can use regular iptables and routing rules in order let the host forward packets between them. The second question has two answers: if you are going to use bridging anyways, or don't mind it, you can just connect all the tapNs into br0 and don't worry about it (make sure you give the guests unique mac and IP addresses though).
Virtual Distributed Ethernet (VDE)
If you can't or don't want to use bridging, then you use VDE. The VDE wrapper for qemu overrides the mechanism that creates a new tuntap device, and instead connects qemu to a socket. At the other end of the socket is a server known as vde_switch. The job of vde_switch is to forward ethernet packets (or frames) between different qemu guests. In addition, it creates its own tap device to allow forwarding between the host and the guests. qemu never notices the difference. When qemu is talking to vde_switch it believes it's talking to the tuntap driver.
ZMan says : IMHO, examples would able all the rest of us to get the know-how behind this mass of text.
obi says: ASCII diagrams might explain this one a bit better
Explanation by ASCII (and partly configuration examples)
This is the relationship between host and guest networkcards - nic0 can be a "virtual" NE2000 nic in Windows, or eth0 in Linux (in GUEST mode) - I just have to call it something.
HOST GUEST
+--------------+ +--------------+
| | | |
LAN ---+---- eth0 | | |
| | | |
| tap0 ----+---+---- nic0 |
| | | |
+--------------+ +--------------+
Now how to get the HOST, GUEST and the computers on the real LAN talking to each other? There are a number of possibilities...
(note: in the example I'll use the 10.0.0.0/8 subnet as the "real" LAN's subnet, and 192.168.X.0/24 as the "virtual" LANs' subnets)
Possibility 0: routing with one "external" LAN, and multiple "virtual internal" LANs in subnets
You can set the HOST up as a router.
In the following example, eth0 is connected to the existing "external" LAN nic and tap0 and tap1 are the new, "internal" nics. With multiple LAN ports, it's like you have multiple separate LANs, connected via the same router. A bit cumbersome, as you have to set up different subnets (192.168.0.0/24 and 192.168.1.0/24) for each GUEST.
HOST GUEST1
+--------------+ +--------------+
| 10.0.0.10 | | |
LAN ---+---- eth0 | | |
| | | | GUEST2
| tap0 ----+---+---- nic0 | +--------------+
| 192.168.0.1 | | 192.168.0.10 | | |
| | +--------------+ | |
| | | |
| tap1 ----+----------------------+---- nic0 |
| 192.168.1.1 | | 192.168.1.10 |
+--------------+ +--------------+
Here is an configuration example of that setup under Debian:
Configuring the tap devices: In Debian in /etc/network/interfaces:
iface tap0 inet static
pre-up sudo tunctl -u YourUserNameHere -t tap0
address 192.168.0.1
netmask 255.255.255.0
iface tap1 inet static
pre-up sudo tunctl -u YourUserNameHere -t tap1
address 192.168.1.1
netmask 255.255.255.0
[#Note6]
Enabling forwarding between NICs:
echo 1 > /proc/sys/net/ipv4/ip_forward
Starting the guests:
qemu -boot c -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=no GUEST0.img qemu -boot c -net nic,vlan=0 -net tap,vlan=0,ifname=tap1,script=no GUEST1.img
Inside GUEST0 (Debian) in /etc/network/interfaces:
iface eth0 inet static address 192.168.0.10 netmask 255.255.255.0 gateway 192.168.0.1
Inside GUEST1 (Debian) in /etc/network/interfaces:
iface eth0 inet static address 192.168.1.10 netmask 255.255.255.0 gateway 192.168.1.1
Possibility 1: NAT routing with one "external" LAN, and multiple "virtual internal" LANs in subnets
This is the same setup as Possibility 1, except that we put the two subnets of GUEST1 and GUEST2 behind NAT. Thus their IPs will not be visible in the LAN.
This setup reflects the common home internet uplink scenario, that is it's just as if the HOST was a little cable/broadband router. (To make this point even more, you can imagine multiple virtual pcs (multiple GUESTs), trying to share an internet connection via the HOST.
In the following example, eth0 is the "external" nic (like the WAN port on your routing) and tap0 is the "internal" nic (LAN port). With multiple LAN ports, it's like you have multiple separate LANs, that all connect to the internet through the same router.
To set this up you need to:
configure the network just like we did in Possibility 0
set up NAT routing on the HOST between the virtual subnets and the real LAN - something like this:
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.0.0/24 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j MASQUERADE
Possibility 2: NAT routing with one "external" net, and one "virtual internal" LANs
Now, a first "simplification" is to do away with the multiple LANs, and connect them together. In real life we'd plug the nic0's in the 2 GUESTs in a hub or switch, and have one network card in the HOST also plugged in in the switch. We can do something similar with a "bridge". We set up a br0 with the bridge-utils, and add (on-the-fly, if we so choose) the tapX devices to it. A bridge makes it so that all the traffic seen by one card (tap0) is also seen by the other one (tap1). Of course the nic0's in the GUESTs will only respond if the traffic is meant for them (has the right MAC address or IP address). Point is, the nic0's in GUEST1 and GUEST2 will see each other. This starts to look like a home LAN+internet set-up, doesn't it?
HOST GUEST1
+--------------+ +--------------+
| 10.0.0.10 | | |
LAN ---+---- eth0 | | |
| | | | GUEST2
| +------+ +-+---+---- nic0 | +--------------+
| | tap0---+ | | 192.168.0.10 | | |
| | tap1---+ | +--------------+ | |
| +------+ | | | |
| br0 +-+----------------------+---- nic0 |
| 192.168.0.1 | | 192.168.0.11 |
+--------------+ +--------------+
Possibility 3: NO routing, one LAN made up of real and virtual devices
Another step further - why bother with NAT routing at all? Why not bridge the "virtual" tapX devices together with eth0? that way the GUESTs would see the traffic of real LAN. This is what you'd get:
HOST GUEST1
+--------------+ +--------------+
| | | |
LAN ---+-+ | | |
| | +------+ | | | GUEST2
| +---eth0 | +-+---+---- nic0 | +--------------+
| | tap0---+ | | 10.0.0.11 | | |
| | tap1---+ | +--------------+ | |
| +------+ | | | |
| br0 +-+----------------------+---- nic0 |
| 10.0.0.10 | | 10.0.0.12 |
+--------------+ +--------------+
Problems and error messages
ping
If you start QEMU with '-net user' pinging the outside world doesn't work; Try 'telnet www.qemu.org 80' instead. See QEMU and network for further QEMU network related articles.
set -net sections for both host and guest system
If you see the following error: error: qemu Invalid vlan (0) with no nics
Then you forgot to add both a -net section for the guest and the host. Try to call qemu with -net nic -net tap,ifname=tap0,script=no
configuring tap under Debian
If you use debian you can create a if section in /etc/network/interfaces:
auto tap0 iface tap0 inet static pre-up tunctl -u $(yourUserName) -t tap0 address 172.20.0.1 netmask 255.255.255.0
Footnotes
Note1:
Versions 0.7.3 and older named them as if they were tun devices (tun0, tun1, etc) due to a misunderstanding, but later versions use the correct tap names by default. I'll talk about tap devices having their proper names, so I'll be using tap0, tap1, etc in the examples in order to reduce confusion.
Note2:
also known as a hardware address since the manufacturer normally writes the address directly into the card
Note3:
This is in contrast to usernet mode, or user mode networking, where you only have the host eth0 and the guest nic.
Note4:
If you want to continue calling the main host nic eth0, you can use ifrename or nameif to rename the original eth0 into reth0 and br0 into eth0. For simplicity I'll continue to call the bridge br0, but it is recommended that you do this renaming (preferably in a boot/init script) if you use bridging.
Note5:
You may ask, why doesn't qemu provide a networking mode that gives access to eth0, instead of having to muck around with tap0s and br0s. The answer is because the Linux kernel (and afaik, the BSD kernels) doesn't support sending packets directly to eth0. The best you can do is either send packets through eth0 to other workstations on the LAN, or go above the ethernet frame layer and send IP packets to the host. If you want to send ethernet frames directly to the host OS, you either have to do it via tap0 or you have to write a custom device driver to inject the frames on your behalf (this is what some versions of VMware do, btw). And if you send it via tap0, then to share a subnet you have to use bridging.
Note6:
If you want to do the same configuration manually - that is not through /etc/network/interfaces, because you're not running Debian, then the same would be achieved like this:
tunctl -u YourUserNameHere -t tap0 ifconfig tap0 192.168.0.1 netmask 255.255.255.0
Adding the default gateway in the second part of the configuration would be done like this:
route add default gw 192.168.0.1
Back to FrontPage