Dominik - hierarchical jails -> networking

New About Yours API Help
5.5 KB, Plain text
Hi Guys,

I am also a long time listener to the bsdnow podcast (since episode one) and wanted to thank you guys for the great work you are putting in week after week. 
As you guys were able to help me out with all questions regarding BSD so far, I wanted to see if I could tap into your brains once again. 

Recently I started experimenting with hierarchical jails and ran into some networking issues. So far I was using a cloned lo1 interface with aliases to make networking available within my jails, which worked pretty well.
My new approach is to make parent jails with their own network stack for the children jails. The idea is to create easy to deploy, scalable environments with separate networking using ZFS templates.

Problem:
I have  a connection to the outside world within my parent jail when I ping IP addresses directly but I don't seem to be ale to resolve DNS.

To create the parent jails I am using iocell with a custom kernel where the following options are set (VIMAGE was the only one I had to set really):
nooptions       SCTP   # Stream Control Transmission Protocol
options         VIMAGE # VNET/Vimage support
options         RACCT  # Resource containers
options         RCTL   # same as above
Networking for the jails in  /etc/rc.conf on the host looks like this:
## DNS unbound local server ##
local_unbound_enable="YES"

## set up bridge interface for iocell ##
cloned_interfaces="bridge0"
ipv4_addrs_bridge0="10.0.0.1/24"

## plumb interface igb0 into bridge0 ##
ifconfig_bridge0="addm igb0 up"

After successfully creating a jail (test1) I get the following output with ifconfig on the host: 
# ifconfig
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:aa:5e:2b:5b:00
inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 6 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 1 priority 128 path cost 55

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:aa:5e:2b:5b:01
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet1:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 7 priority 128 path cost 2000

vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: 47f7e3c9-abe4-11e8-92d4-001e67b65f08
options=8<VLAN_MTU>
ether 02:ff:60:47:f7:e3
hwaddr 02:95:d0:00:06:0a
inet6 fe80::ff:60ff:fe47:f7e3%vnet0:2 prefixlen 64 scopeid 0x6
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

vnet1:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: 47f7e3c9-abe4-11e8-92d4-001e67b65f08
options=8<VLAN_MTU>
ether 02:ff:60:47:f7:e5
hwaddr 02:95:d0:00:07:0a
inet6 fe80::ff:60ff:fe47:f7e5%vnet1:2 prefixlen 64 scopeid 0x7
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

unbound is set as the internal DNS on the host:
server:
        username: unbound
        directory: /var/unbound
        chroot: /var/unbound
        pidfile: /var/run/local_unbound.pid
        auto-trust-anchor-file: /var/unbound/root.key
        interface: 10.0.0.1
        interface: 127.0.0.1
        access-control: 10.0.0.0/24 allow
        access-control: 127.0.0.1 allow

include: /var/unbound/forward.conf
include: /var/unbound/lan-zones.conf
include: /var/unbound/control.conf
include: /var/unbound/conf.d/*.conf

Host --> /etc/resolv.conf:
nameserver 127.0.0.1
options edns0

The defaultrouter on the jail is set to the brdige0 IP 10.0.0.1

Jail --> /etc/resolv.conf:
nameserver 10.0.0.1

ifconfig from the test1 jail:
root@test1:~ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog

vnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:ff:60:47:f7:e4
hwaddr 02:95:d0:00:07:0b
inet6 fe80::ff:60ff:fe47:f7e4%vnet0 prefixlen 64 tentative scopeid 0x3
inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet1: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:ff:60:47:f7:e6
hwaddr 02:95:d0:00:08:0b
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

I am stuck at this point and don't know how to go about it so I was wondering if you guys have any ideas. Any suggestions would be much appreciated.

Thx in advance and best regards. 

Dominik
Pasted 2 months, 2 weeks ago — Expires in 291 days
URL: http://dpaste.com/0SZJ0V4