Using the DeltaCopy Server to Keep The Drives In Sync – Using Deltas!

Once again, being not very happy with the previous not-quite-right way of doing things, I had a little poke, a nudge, and a further poke at DeltaCopy and its configuration to try to get the rsync working as rsync should work.  Ideally, you`ll want to use rsync through a server, and in this case the rsync server is DeltaCopy, or specifically, the DeltaCopy server installed with the application.  The previous post provides a method to implement a simple “if the file or directory exists, check it`s up-to-date, and if not, re-copy it; if it doesn`t exist, copy it”, but one of rsync`s biggest features is that of copying deltas, hence the name of the DeltaCopy tool.  The deltas are differences in files, and rsync will attempt to copy only changes in files to bring them up to date, thus saving in the amount of time required to copy.  Now, copying deltas can be oversold (depending on the type of file, it`s not always possible just to apply the changes, so a full re-copy is necessary) and it is more resource intensive, but for a large number of files, particularly text files (HTML, log files, code, etc.) can produce a massive speed up in the sync time.

rsync-ing the Photos Directory

Here`s the scenario, if you`ve already forgotten, or can`t be arsed reading the previous post:

USB drive mounted on U:
Lacie NetworkSpace network drive mounted on M:

U: is my primary work drive with all my gubbins on it, and M: is a network-attached drive for keeping a backup of everything.  I don`t always want to re-sync everything, as even with rsync, just finding the files and directories in order to test whether or not they`re up to date can take ages.  What we`ll do here is just rsync the photos directory, U:\photos to M:\photos.

The file we`re interested in here is deltacd.conf, located if you`ve installed using defaults, to

C:\Program Files\Synametrics Technologies\DeltaCopy\deltacd.conf

Open this file in Notepad and take a look.  What we see here are a number of “modules” which can be source or destination locations.  We can`t use a source module and a destination module if they`re both going through the server.  So we need to specify a module pointing to the U drive, and hack our little batch file to point to the M drive.

Here`s my module representing the photos directory on the U: (USB-attached) drive:

[usbphotos]

path = /cygdrive/u/photos/
comment = photos on usb drive
read only = false
auth users =
secrets file =

You can see this just points to U:\photos using the cygwin notation.  In order to make use of this, I then edit my little batch file, backup-photos-to-network.bat:

“C:\Program Files\Synametrics Technologies\DeltaCopy\rsync.exe” –log-file=”/cygdrive/c/rsync-photos-out.txt” -h –progress –delete-during -v -v -rlt localhost::usbphotos// “/cygdrive/m/photos”
pause

So, same parameters as before, but different source and destination.  The localhost::usbphotos// tells rsync to talk to a rsync server (“daemon”) on localhost and use the usbphotos module to grab its actual location, which of course is /cygdrive/u/photos AKA U:\photos  There`s also a pause in there to keep the DOS prompt open when the job completes – always nice to be able to see the status, else the DOS prompt will vanish after completion, which won`t be helpful if you`ve an error somewhere in the batch file.

One thing you`ll notice when using rsync deltas is that you`ll see a lot more CPU usage on your machine.  However, assuming you see uplift from the delta copying, this can make it all worthwhile.  Also, if you`ve a lot of files, it can be worth specifying a different log file for each job as the files can grow very large very quickly.  Finally, if you`re that way inclined, you can use the nasty Windows task scheduler to schedule regular runs of these batch files to keep everything up to date on the network drive.

That concludes my quick and nasty DeltaCopy rambling; basically I found the GUI too awkward for something I wanted to do quickly, and then ended up bypassing it completely to do what I needed to do.  I now however have a lovely mirror of my files.  Now to find somewhere to hide the drive!  I`ve been thinking about online file storage, but currently trying to keep 600GB and counting online would take too long to sync, and it`s just easier to have another drive, although that`s of little comfort if the house burns down.

Output From The Batch File

C:\>”C:\Program Files\Synametrics Technologies\DeltaCopy\rsync.exe” –log-file=”/cygdrive/c/rsync-work-out.txt” -h –progress –delete-during -v -v -rlt localhost::usbwork// “/cygdrive/m/work”
opening tcp connection to localhost port 873
sending daemon args: –server –sender -vvltre.iLs . usbwork//
receiving incremental file list
delta-transmission enabled
./
DSCF6251.JPG is uptodate
DSCF6261.JPG is uptodate
DSCF6277.JPG is uptodate
Smarty-2.6.22.zip is uptodate
Thumbs.db is uptodate
databases/headstaggers.mdb is uptodate
databases/headstaggers_gallery.txt is uptodate
databases/hscv.sql is uptodate
databases/hscv.sql_original_test is uptodate
databases/mydyson.txt is uptodate
databases/mydyson.txt.save is uptodate
databases/v.jsp is uptodate
[...]

I obviously need to do a bit of clean up!  Again, if you`re playing around and testing, avoid using –delete-during until you`re sure things are working!  “YMMV” as they seem to say.