mceith.com

One issue with zfs send over network is that the documentation available from Oracle only shows examples with ssh. This is not the best case if one is dealing with large datasets. It is very slow over and does not applicable to 10 Gigabit networks.

I have noticed that mbuffer is very capable to deliver high speeds on zfs send. It works client based and needs mbuffer client on sending and receiving end.

Set up receiver to accept connection from senders ip-address 192.168.1.100:

# mbuffer -s 128k -m 1G -I 192.168.1.100:50001 | zfs receive tank/backup/data


mbuffer listens on port 50001 and uses 1 Gigabyte of memory to buffer incoming data. Blocksize is set to 128k which is default on ZFS.

Send the data to client on address 192.168.1.10:

# zfs send tank/nfs/data@today | mbuffer -s 128k -m 1G -O 192.168.1.10:50001


One should see significant increase of speed to zfs send. If the dataset is busy with bursty writes, mbuffer will hold it on its own buffer on both ends.

Remember that datastream is not crypted and this can be issue on some cases. So use dedicated networks for sending snapshots with mbuffer.

mbuffer can be found here

One of the most confusing property of ZFS is the complex space allocation of datasets if you are used to terms like thin provisioning, snapshot reserve and other common storage technobabbels. For me at least 🙂

I want to create 50G dataset which user sees as 50G of available space, it is all ways reserved for that user and snapshots cannot never consume space reserved for the user. User cant use space more than 50G even if there is space available on pool. I also want to have 50G space available for snapshots guaranteed.

So we have:
50G reserved for user
50G reserved for snapshots
Total: 100G

First we have to reserve that 100G from pool so no one else cannot use our precious space:

A ZFS reservation is an allocation of space from the pool that is guaranteed to be available to a dataset. As such, you cannot reserve space for a dataset if that space is not currently available in the pool. The total amount of all outstanding unconsumed reservations cannot exceed the amount of unused space in the pool. A dataset cannot consume space that has been reserved for another dataset.

Now we want to give exactly 50G for that user1 without freebies on space possibly available on pool:

refquota on a dataset limits the amount of space that the dataset can consume. This hard limit does not include space that is consumed by snapshots and clones.

Okay so now we have 100G of guaranteed space for our dataset and 50G limit to user1. Next thing is make sure that the user1 will have that 50G available no matter what:

You can set a refreservation to guarantee space for a dataset that does not include space consumed by snapshots and clones. If refreservation is set, a snapshot is only allowed if enough free pool space exists outside of this reservation to accommodate the current number of referenced bytes in the dataset

user1 is now one happy puppy.

Nice new command in Solaris 11 express and Solaris 11 is ipadm. The command is soon to replace good old ifconfig. Here is little cheat sheet for basic operations.

Use dladm to list info about physical interfaces:

# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         100    full      bnx0
net4              Ethernet             down       0      unknown   ixgbe0

Notice that Oracle has changed link names to be netX by default and not after link name.

This is not the case if you upgraded from express to 11. You can rename the link name with dladm rename-link old-linkname new-linkname

To configure address to interface we first need to bring the interface up (plumb)

# ipadm create-ip net4
# dladm show-phys net4
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net4              Ethernet             up         10000  full      ixgbe0

Configure address to newly created interface.

Static IP:

DHCP:

Change netmask:

List active interfaces:

# ipadm show-if
IFNAME     CLASS    STATE    ACTIVE OVER
lo0        loopback ok       yes    --
net0       ip       ok       yes    --
net4       ip       ok       yes    --

Show current addresses:

# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net0/v4           static   ok           1.1.1.1/24
net4/lan          static   ok           10.0.0.1/24

Add default gateway, -p for persistance so it will be activated on reboot:

# route -p add default 1.1.1.254

Delete address:

Delete interface: