zrep troubleshooting

zrep is, by design, fairly simple. It is based around zfs properties. Therefore, when in doubt, check the properties. zrep gives an easy way to do this.

Here is an example of a set of properties used when zrep is replicating to a filesystem on the same host. Note that src-host and dest-host are the same:

# ./zrep list -v scratch/zreptest

scratch/zreptest:
zrep:src-fs     scratch/zreptest
zrep:src-host   hamachi
zrep:dest-fs    scratch/zreptest_2
zrep:master     yes
zrep:savecount  5
zrep:dest-host  hamachi
For the destination filesystem (in this case scratch/zreptest_2), the properties would be mostly the same. The only difference would be, instead of "zrep:master yes", you would see "readonly on"

ZFS property overview

Fixing zfs properties

To manually change a zfs property, use
zfs set zrep:something=newval  file/system
To remove a property, zfs is a little odd. The standard (and in fact, only) way of removing a property in zfs, is as follows:
zfs inherit zrep:master  file/system
This tells zfs to set the value type of that property, to be "inherited" from the parent filesystem. Since the parent usually does not have any zrep properties, that effectively 'removes' a property from a filesystem or snapshot.

Fixing snapshot properties

There is only one property that zrep currently cares about for snapshots:
zrep:sent

This contains a timestamp based value of the time it was successfully synced to the target system. Both the src snapshot, and the target-side snapshot, have this value, since it is crucial in allowing smooth failover from src to test.

Bad zrep status output

You will hopefully never see the following:
# zrep status
scratch/zreptest                              synced as of [NEVER]
If you do, it means that either there is no zrep snapshot present, or the snapshot does not have the "zrep:sent" property.

If there is no snapshot, then unless you have a separate fully sync'd snapshot, you will need to do a full reinitialization of zrep sync and data, via

zrep clear scratch/zreptest
zrep init scratch/zreptest desthost destfs
If for some reason, you DO have a common snapshot on both sides, then you can see the
"Converting normal snapshot" section lower down.

Setting the zrep:sent property

If there IS a zrep snapshot of the form @zrep_######, but "zrep status" is not picking it up, then you will have to manually set a "sent" property value on the snapshot.

Normally, it is expected that it be set to a timestamp value. However, the simplest and safest thing to do in this specific case, is to set it to "1", via the following:

zfs set zrep:sent=1 scratch/zreptest@zrep_######
Presuming that "zrep list -v scratch/zreptest" shows the usual properties (example at the top of this page), you should then be able to once more do "zrep sync" for it.

Converting normal snapshot to a zrep snapshot

If for some reason, you have lost the "real" zrep snapshots, but have a snapshot that you PREVIOUSLY created by hand on the master, and it also exists on the target, you should be able to rename the snapshot on both sides to be "zrep_000000".

Once that is done, you need to set the appropriate zfs properties that zrep expects. The EASY WAY now, is to use the special sentsync subcommand, documented lower down in this page.

Ugly Manual way:

set the zrep:sent value on the snapshot (not the filesystem) as mentioned above.
You will also need to make sure there is a "zrep:master" property set to "yes".

You also need a lastsent property. See which properties zrep likes to set on your version of zfs, by using "zrep list -v" on a working paired filesystem.

You should then be able to once more do zrep sync again.

Do note that you cannot just randomly create snapshots on both sides for this purpose. The snapshot on the target side, must have been created as a result of a zfs send/receive combination. That is the only way that zfs incremental receives can function.

Marking a sync as valid (sentsync)

Occasionally, it has been reported to me, that a zfs send appears to have succeeded, yet the setting of zrep sent properties has failed.
For those situations, I have added the recovery command of:

zrep [-t tag] sentsync fs@snapshotnamehere

(think of this as "Syncing the last Sent snapshot")

Critical Warning: Only use this command on the master side, and only on the most resently zrep-sent snapshot for that filesystem/tag combination (and of course, only when you are not trying to do another sync for it)

The sentsync subcommand, will set "sent" properties on both sides, and also set the "master" property on the side you run it on.

Warning #2 it will now CLEAR all other timestamps on other snapshots. This is to allow a special edge case usage, whereby the remote side has somehow "lost" a recently synced snap or so, so the master needs to be told, "no, dont sync from most recent, sync from a few snaps previously".

This is a mostly convenience feature. To still preserve the "sent" timestamp of earlier snapshots, you would need to either DELETE the more recent snapshots on the master, or clear the zrep:sent values from the more recent snapshots.


Written by:Philip Brown
Bolthole Top Search Bolthole.com