Amazon Clouddrive, Encfs and Rsync – Part II

Welcome Back!

So now that you set up your own syncmachine that will push all your data encrypted to your Amazon Clouddrive, it’s time to up the ante. A quick recap: At this point you can mount and decrypt your Drive into a local folder and read files with ‘ok’ speeds. Writing actually takes ages because if you copy/move a file into your clouddrive decrypted folder the whole process of encrypting and uploading takes place. This is annoying for large files and some programs have serious problems with that.

So if you are a CouchPotato like me, that has some issues coming up on your Sonarr, err, Radar you want those issues to be gone. So like magic you raise your wand and begin to cast the “Abra-ca sabnzbd” spell.

– Anonymous post on usenet.

If that sentence made no sense to you, well no worries. Here is an analogy:

  1. You download you favorite Linux ISO with your favorite command line tool.
  2. You save said ISO in a temporary folder somewhere.
  3. You wrote or got a neat program that scans and sorts your temp folder into your CloudDrive.
  4. The program starts a copy/move from the temporary directory into the Clouddrive.
  5. Encfs and acd_cli start working on encrypting and uploading.
  6. Before the copy/move command completes your tool throws a timeout.
  7. Nothing happened.

Also if you share your CloudDrive folder with Samba it is annoying to transfer several GB worth of data. The wait kills at least me. I don’t like to wait.

Let’s fix all that. Now.

The Blueprint

In short: We are going to add a persistent cache on top of your clouddrive. This cache will speed writes up to whatever the underlying hard disk or SSD allows and the sync will take place in the background. Oh, and your files are immediately available to you. For the You-Ser (get it? You and User) it is entirely transparent. On top of that it is persistent. Crash or Blackout while there are data in the cache? Got you covered. Interested? Let’s all go Tim Taylor on this.

For this example I am picking up on Part I of this Howto, and assume these defaults:

  • /media/Storage – The decrypted folder of your CloudDrive.
  • /media/cache – An empty folder either on the root directory or inside a mounted Harddisk. This directory needs to be able to hold enough files to your liking. I am using 1000gb (1tb) for this.
  • /root/clouddrive/configs/Storage.xml – your encryption xml file used to access the Storage.
  • Your password for the Storage.

If you picked a different directory to mount your Clouddrive (other than ‘/media/Storage’) replace my suggestion with your actual directory. This goes for all the defaults I am assuming.

Got enough space on /media/cache? Good. Lets get cracking. If not: You do need space for caching.

Home Improvement

What we are really going to do is mounting an overlay-filesystem, join your real-data with an empty directory and route all changes to the latter. This will yield a directory full of changed files that is already on a permanent storage (harddisk) and can be easily synced. As we access the drive now via the cache, the local cache dir always has preference over the remote directory. So speedy all the way!

First things first, check that you have a recent kernel (3.10+) and support for overlayfs:

$ grep overlay /proc/filesystems
nodev overlay

If there is no output, be sure you are running a recent kernel. Try loading the module anyway:

# modprobe overlay
# grep overlay /proc/filesystems
nodev    overlay

The first command should exit without any comment. If there is output you might need to upgade your kernel first. Next, create needed directories:

mkdir -p /media/cache/workdir
mkdir -p /media/cache/data
mkdir -p /media/cache/data.reverse
mkdir -p /media/cache/combined

Risking that I am repeating myself: Be sure to have lots of space under /media/cache.  Now, we will jump right in and  mount the overlay:

mount -t overlay overlay -o lowerdir=/media/Storage,upperdir=/media/cache/data,workdir=/media/cache/workdir /media/cache/combined

It should silently exit. Check that the filesystem is mounted:

df -h /media/cache/combined
 Filesystem      Size  Used Avail Use% Mounted on
 overlay          30G  1.3G   27G   5% /media/cache/combined

Filesystem must be overlay. If it is anything else, you have a typo somewhere up there. Go into that /media/cache/combined directory, have a look around. You should see the contents of your regular CloudDrive. Now copy any file in there, even a large file and notice the speed. Also the file is instantly there. But if you were to check the ‘real’ CloudDrive folder (/media/Storage) your file is not there. It’s in the local cache dir. Change into ‘/media/cache/data’ and check the contents. This directory keeps all the changes you need to sync. Also: Yay, you found your missing file! \o/

Syncing back to the Cloud

But in order to sync you will need to encrypt the data directory again. This is done the same way you did when initially uploading your files to the cloud: reverse encfs. You even need the exact same files and passwords from back then. So like before:

ENCFS6_CONFIG="/root/clouddrive/configs/Storage.xml" encfs --reverse -o allow_other --no-default-flags /media/cache/data /media/cache/data.reverse

Check out `/media/cache/data.reverse`. Should look familiar. The actual copy is done (like usual) with

rclone sync /media/cache/data.reverse remote:Storage/

Now this rclone command will take its time. But that is ok: You can keep working and accessing your files as if nothing is happening. The only caveat is that only this one server/computer currently has these new files. All the rest needs to wait for the sync to complete.

After the run is complete you need to dismount the reverse encrypted directory, dismount the overlayfs, dismount the clouddrive and remount everything so that you cache directory is empty again and that the clouddrive will now serve the files out of the cloud:

umount /media/cache/data.reverse
umount /media/cache/data
~/encryption/amazon-mount.sh umount
~/encryption/amazon-mount.sh mount
rm -vf /media/cache/data/*
mount -t overlay overlay -o lowerdir=/media/Storage,upperdir=/media/cache/data,workdir=/media/cache/workdir /media/cache/combined

Not tedious at all. Wait it is. Can’t have that. Lets use a script instead:

#! /bin/bash

#
# Configuration
#
ENCFS6_CONFIG="~/encryption/reverse.xml"
export ENCFS6_CONFIG
ENCFS_PASS="[YourStoragePasswordHere]"

#
# Locking
#
function locking {
        LOCKFILE="/tmp/copy.lock"
        # Check if the lockfile exists.
        if [ -e "${LOCKFILE}" ] ; then
                # it does exist. Check if the process is still running.
                PID="`cat ${LOCKFILE}`"
                if [ "`ps --no-heading -q ${PID} | wc -l`" != "0" ] ; then
                        # it is. Don't touch anything, walk away slowly.
                        echo "Still running."
                        exit 1
                else
                        # it is not. Stale lockfile. Keep calm and remove the lockfile.
                        rm ${LOCKFILE}
                fi
        fi
        # No lockfile present (anymore, stale would have been deleted)
        echo $BASHPID > ${LOCKFILE} || error "unable to create lockfile."
        # Remove lockfile on script exit.
        trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT
}
locking

#
# Only run if there are files to be synced.
#
COUNT=$(find /media/cache/data -type f | wc -l)

# Only act if we have new stuff.
if [ ${COUNT} -lt 1 ] ; then
    echo "No data files found."
    exit 0
fi

# reverse-encrypt the delta.
umount /media/cache/data.reverse 2>/dev/null
encfs --reverse --extpass="/bin/echo ${ENCFS_PASS}" -o allow_other --no-default-flags /media/cache/data /media/cache/data.reverse

# Copy everything into the cloud. (Twice; things might have been added)
/usr/local/bin/rclone copy /media/cache/data.reverse remote:Storage/
/usr/local/bin/rclone copy /media/cache/data.reverse remote:Storage/

# If this a only-copy call, then quick afterwards.
if [ "$1" == "onlycopy" ] ; then
    umount /media/cache/data.reverse 2>/dev/null
    exit 0
fi

# Stop the Services
umount /media/cache/combined 2>/dev/null

# Add custom STOP commands here (samba, nfs, ftp...)

# umount the clouddrive
/encryption/amazon-mount.sh umount

# Move everything into the cloud
/usr/local/bin/rclone move /media/cache/data.reverse remote:Storage/ && rm -rf /media/cache/data.reverse/*

# Umount the reverse-encrypted dir
umount /media/cache/data.reverse 2>/dev/null

# Remount CloudDrive
/encryption/amazon-mount.sh mount

# Re-enable the new overlayfs
mount -t overlay overlay -o lowerdir=/media/Storage,upperdir=/media/cache/data,workdir=/media/cache/workdir /media/cache/combined

# Add custom START commands here (samba, nfs, ftp...)

Save this as ‘/root/copy.sh’. ‘This script will pre-sync everything from your local cache twice, then stop the clouddrive, sync-move everything and re-mount the clouddrive. As there are two syncs before the clouddrive is stopped, the downtime should be minimal. And if you place the sync job somewhere into the night you won’t notice anything at all. Let’s setup crontab:

crontab -e

And add:

0 0,3,9,12,15,18,21 * * * /root/copy.sh onlycopy 
0 6 * * * /root/copy.sh >>/root/usenet.log

This will copy (but not restart) the data file directory into the cloud every 3 hours and a copy-and-restart am 6am. Don’t worry about the script running several times in parallel: It refuses to launch if it is already running.

One final thing: If you are using Samba point your shares to ‘/media/cache/combined’. Or any other service for that matter. Always use the new mount point to get the new speeds. You can and should add stop/start commands for your used services in the script. Add those below the “Custom Command” comments.

Estimating the damage

If you followed my guidance and went with me through all this step-by-step you should be running a fully encrypted Amazon CloudDrive that decrypts on-the-fly. Adding files is a bliss as no one (or no tool) needs to wait for any upload to finish. Syncing is done in the background and safely (protected against crashes et all). So if you have a fast download but slow(er) upload connection this should net you a very fast encrypted network drive.

Enjoy.

-Christian.

Christian

Touched base with Linux back in 1995, got hooked up on it ever since. I am using Linux for both private and office for two decades. Working as a System Administrator at a medium sized hosting company I get in touch with all kinds of trouble. All of which can be solved with Linux. In my blog I am sharing solutions to problems that I had to search for myself in hope that someone else out there might find them useful.

Leave a Reply

Your email address will not be published. Required fields are marked *