Amazon Clouddrive, Encfs and Rsync

Cloudy, with a chance of Unlimitness.

Amazon.$tld just released an unlimited Storage Tier for non-commercial applications for ~70 Bucks a year (depending on your location). And they are not kidding. Not only does the Web-Interface say ‘x used of unlimited’ but once mounted you’ll notice 100TB Quota. And reports on github actually state people reached that limit and even increased that by contacting support. So, plenty of space for your ‘documents’.

In its most basic form you can manage, down- and upload your files via the web-interface. That’s nice for on-the-go, but we are talking serious data here. So we need a client. Amazon offers the usual stuff: Windows Client, OSX Client. No Linux? Don’t be sad. Those clients can’t even do a real ‘sync’. Plus so far all clients upload your files unencrypted. Of course amazon goes ad-fishing with buzzwords from the ‘secure’-department… But they do have a list of compatible file types available. This is just for the web-client that it can index and play media; but this also means they are scanning your files (even if only automatically).

Err, no thanks.

What we want is the ability to mount the CloudDrive in Linux and encrypt everything before uploading.
Let’s get cracking.

Part I: acd_cli

yadayada wrote acd_cli, hosted on github. This jewel is a python application that connects and mounts(!) the CloudDrive via fuse to Linux. Enough words, more installing! For CentOS7 it’s:

yum install epel-release
yum install python34-setuptools w3m
easy_install-3.4 pip
pip3 install --upgrade git+https://github.com/bgemmill/acd_cli.git@master

First we install the epel repository that is needed for Python 3.4 and pip (3.4 edition) which we are installing with command 2 and 3 (w3m is a text based browser). Command 4 installs a fork(!) of acd_cli that does some work-around needed, but more on that later on.

Try launching the command:

[arrakis ~]# acd_cli
usage: acd_cli [-h] [-v | -d] [-nl] [-c {never,auto,always}]
               [-i {full,quick,none}] [-u]
               {version,v,sync,s,old-sync,clear-cache,cc,tree,t,children,ls,dir,find,f,find-md5,fh,find-regex,fr,upload,ul,overwrite,ov,stream,st,download,dl,cat,create,c,mkdir,list-trash,lt,trash,rm,restore,re,move,mv,rename,rn,resolve,rs,add-child,ac,remove-child,rc,usage,u,quota,q,metadata,m,mount,umount,delete-everything,init,i,test}
               ...
acd_cli: error: the following arguments are required: action

Now you need to authorize acd_cli with your amazon account, launch:

[arrakis ~]# acd_cli init
For the one-time authentication a browser (tab) will be opened at https://tensile-runway-92512.appspot.com/.
Please accept the request and save the plain-text response data into a file called "oauth_data" in the directory "/root/.cache/acd_cli".
Press a key to open a browser.

Hit a key, and (I am assuming you are running a headless server) w3m will open. Navigate in there, enter password, click OK all the way to something like this:

{
    "access_token": "4c8lwGmDqF2WZbcYDHI9MZ3040ATLKPU0fQkKTMNYYtx4gqC5bqAlUVvKyZ",
    "exp_time": 1472910000.2017406,
    "expires_in": 3600,
    "refresh_token": "nDQVHrRvgU1MI1QoLuUAPHtz7CsLp3RIANxyQ2KyMf5EvoxyyEAUeLNarax",
    "token_type": "bearer"
}

Copy that text or directly save that in a file and place it at ~/.cache/acd_cli/oauth_data, then(!) resume the process. Keep that file as private as possible, that opens your vault without any further authorizations. Now try your connection:

$ /bin/acd_cli sync
Getting changes..
Inserting nodes.

If that’s all you’re seeing: acd_cli works and just synced to your account.

Part II: The mounting

At this point we can actually do our first mount, but before let’s agree on some basics:

  • We will use ‘/var/clouddrive’ as a base for all other stuff.
  • We will store all encryption related files in ‘~/encryption’
  • We will mount the clouddrive on ‘/var/clouddrive/encrypted/’ (encrypted files view)
  • Your current, unencrypted data that you want to encrypt and upload resides in ‘/media’

Create the directories:

base='/var/clouddrive'
mkdir -p ${base}
mkdir -p ${base}/encrypted
mkdir -p ${base}/decrypted

Let’s mount Amazon CloudDrive right in there for the first time:

acd_cli mount ${base}/encrypted/

The command should exit without any errors, check that it works:

[plex@arrakis ~]$ df -h
Filesystem        Size  Used Avail Use% Mounted on
[...]
ACDFuse           100T  1G  100T   1% /var/clouddrive/encrypted

[plex@arrakis ~]$ ls -lha /var/clouddrive/encrypted
total 0
drwxrwxrw- 1 acd acd  0 Aug 31 13:14 .
drwxr-xr-x 4 acd acd 38 Aug 31 10:33 ..
drwxrwxrw- 1 acd acd  0 Aug 31 13:13 Documents
drwxrwxrw- 1 acd acd  0 Aug 31 13:14 Pictures
drwxrwxrw- 1 acd acd  0 Aug 31 13:14 Videos

The mount is active and you can even list the things that are in the CloudDrive.  Compare that to the web-interface and it should match. If you don’t care about encryption then your journey can end here. It should not, however.

We do not need the mount for now, dismount with

umount /var/clouddrive/encrypted

Part III: The disciphers of Data

Disciples, Disciphers… As long as you keep reading and preach the word of encryption it should matter not. Less talk, we have data to encrypt. I did create a “Storage” directory inside the CloudDrive which I’ll use for the encfs. You can encrypt all of it, but you might want to have, at some point, an unencrypted area for sharing or whatnot. Better keep your options open.

install encfs:

yum install encfs

Here is the thing: You do have a lot of data right now that you want to upload and fast. rsync comes to mind; but rsync and acd_cli do not play nice at this time (extremely high cpu load, extremely low throughput), that’s why we clone the fork of acd_cli– still, uploading is a mess. This is why we will not use rsync for the initial upload but some other file. More on that later; lets set up the encryption. For that we will utilize encfs in its reverse mode, that is, it will mount a virtual encrypted view of the data that it will encrypt on the fly. Normally encfs works the other way around: It mounts encrypted data that that it decrypts on the fly.

  • Normal Mode: On Harddisk: Encrypted files; encfs-mount shows decrypted files.
  • Reverse Mode: On Harddisk: Decrypted files; encfs-mount shows encrypted files.

But since your data is decrypted on your, say, NAS, we will upload the encrypted view of that. Setup time.

Before you start the commands below, create a password. I suggest using ‘pwgen’ with ‘pwgen -s 128 1’, this will create a single, secure 128 chars long password.

base='/var/clouddrive'
mkdir -p ${base}/reverse
encfs --reverse /media /var/clouddrive/reverse
Creating new encrypted volume.
Please choose from one of the following options:
 enter "x" for expert configuration mode,
 enter "p" for pre-configured paranoia mode,
 anything else, or an empty line will select standard mode.
?> x

Manual configuration mode selected.
The following cypher algorithms are available:
1. AES : 16 byte block cipher
 -- Supports key lengths of 128 to 256 bits
 -- Supports block sizes of 64 to 4096 bytes
2. Blowfish : 8 byte block cipher
 -- Supports key lengths of 128 to 256 bits
 -- Supports block sizes of 64 to 4096 bytes

Enter the number corresponding to your choice: 1

Selected algorithm "AES"

Please select a key size in bits. The cypher you have chosen
supports sizes from 128 to 256 bits in increments of 64 bits.
For example: 
128, 192, 256
Selected key size: 256

Using key size of 256 bits

Select a block size in bytes. The cypher you have chosen
supports sizes from 64 to 4096 bytes in increments of 16.
Alternatively, just press enter for the default (1024 bytes)

filesystem block size: 4096

Using filesystem block size of 4096 bytes

The following filename encoding algorithms are available:
1. Block : Block encoding, hides file name size somewhat
2. Block32 : Block encoding with base32 output for case-sensitive systems
3. Null : No encryption of filenames
4. Stream : Stream encoding, keeps filenames as short as possible

Enter the number corresponding to your choice: 1

Selected algorithm "Block""

reverse encryption - chained IV and MAC disabled
Enable per-file initialization vectors?
This adds about 8 bytes per file to the storage requirements.
It should not affect performance except possibly with applications
which rely on block-aligned file io for performance.
y/[n]: 

Configuration finished. The filesystem to be created has
the following properties:
Filesystem cypher: "ssl/aes", version 3:0:2
Filename encoding: "nameio/block", version 4:0:2
Key Size: 256 bits
Block Size: 4096 bytes
File holes passed through to ciphertext.

Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism. However, the password can be changed
later using encfsctl.

New Encfs Password: 
Verify Encfs Password: 

That should have created your initial reverse view, try ‘ls -lha /var/clouddrive/reverse/’. If you see a bunch of garbage: It worked. You also have a new file in /media/, named .encfs6.xml. This is your supplemental encryption file that, combined with your password will decrypt the data. Keep both at a super-safe place. I you lose either you will no longer be able to access your data. Move your file to our per-determined location:

mkdir ~/encryption
mv /media/.encfs6.xml ~/encryption/reverse.xml

We will store all encryption related things in ~/encryption. Also create a file ‘~/encryption/reverse.pass’ and on a single line paste your password there. This is optional and will help with auto-mounting.

Part IV: The Syncing

At this point you should be able to mount amazon CloudDrive (which is not mounted currently) and having a reverse-encfs mount up and running, let’s sync. Like I said, syncing with rsync is currently not possible. So we will employ rclone, a command line tool that can handle most Cloud-Provides, including Amazon CloudDrive. Depending on your server, download the appropriate binary from http://rclone.org/downloads/. I am using a raspberry Pi3 for the upload, which maxes my 4Mb/s nicely with plenty of resources to space. The RP3 utilizes the ARM binary for Linux. Download, Unpack and place rclone in ‘/usr/local/bin/rclone’. For further installation help consult their installation docs.

You’ll then need to do a first-time setup that you only need to do once, they also have a very detailed howto for that: http://rclone.org/amazonclouddrive/

Done? Now simply sync the encrypted files to your CloudDrive:

rclone sync /var/clouddrive/reverse remote:Storage/

Replace ‘remote’ with whatever you named your remote in rclone during the setup. If you took my advice and created a ‘Storage’ directory then that command will upload all the encrypted files into your CloudDrive. I really, really recommend launching that sync command from within a screen or tmux shell. If all worked well you’ll see (once a minute) a status output like this:

2016/09/02 10:19:26 
Transferred:   928.978 GBytes (3.890 MBytes/s)
Errors:                 0
Checks:                 2
Transferred:          899
Elapsed time:  67h56m1.2s
Transferring:
 * ...1JrzoMcLjfGxQTbL1zsG3hCu23FI05r19H6Ek4l148: 94% done. avg: 1020.0, cur:  834.5 kByte/s. ETA: 4m1s
 * ...6jftoXHq-0n-AzZxg/JLV-hmzY,QGSqaLmmLKSPvcI:  3% done. avg:  739.5, cur:  775.2 kByte/s. ETA: 19m59s
 * ...ClFXawwAnT,8aoErCjOH-aZP2HyHDGrmdQ3XNHhp97: 66% done. avg:  892.0, cur:  717.1 kByte/s. ETA: 11m52s
 * ...VMBMmHAa23,LVkjy2zyNHHUvv4kOWI3xXBmRZg0Jk5: 31% done. avg:  905.0, cur:  734.7 kByte/s. ETA: 53m26s

Once some files have uploaded, check your CloudDrive web-interface, you should only see encrypted files inside ‘/Storage’:

cd2

So let it run and finish uploading. If the rclone command aborts/interrupts at some point it’s safe to restart it. It will ignore already-uploaded files so resume is a bliss. rclone survived several DSL disconnections so far without dropping out at all. I am still on my first-run on that command.

Part V: Accessing your Data

At some point you might want to access your data. Let’s also assume you want to access the data on another server. Repeat the acd_cli and encfs installation as we did earlier. You can skip reverse mounting and rclone completly. Create directories:

base='/var/clouddrive'
mkdir -p ${base}
mkdir -p ${base}/encrypted
mkdir -p ${base}/decrypted
mkdir -p ~/encryption

Copy the ‘~/.cache/acd_cli/oauth_data’ file from your rclone-upload server to this server so you can skip the authorization altogether. Also copy the ‘~/encryption/reverse.*’ files likewise. Next, create a ‘~/encryption/amazon-mount.sh’ file, paste this inside:

#!/bin/bash

base="/var/clouddrive"
encfs_pass="`cat ~/encryption/reverse.pass`"

if [ ! -e ${base} ] ; then
  mkdir -p ${base}
  mkdir -p ${base}/encrypted
  mkdir -p ${base}/decrypted
fi

if [ "$1" == "mount" ] ; then
  acd_cli sync
  acd_cli mount ${base}/encrypted/ || exit 1
  ENCFS6_CONFIG='~/encryption/reverse.xml' encfs --extpass="/bin/echo ${encfs_pass}" ${base}/encrypted/Storage ${base}/decrypted
  exit 0
fi

if [ "$1" == "umount" ] ; then
  fusermount -u ${base}/decrypted
  fusermount -u ${base}/encrypted
  exit 0
fi

echo "Please supply mount or umount."

and make it executable:

chmod 0700 ~/encryption/amazon-mount.sh

This script will mount amazon CloudDrive and encfs als dismount is after. Like so:

$ ~/encryption/amazon-mount.sh mount
Getting changes..
Inserting nodes.
$

This will result in two mounted directories:

$ df -h | grep cloud
ACDFuse     100T  912G  100T   1% /var/clouddrive/encrypted
encfs       100T  912G  100T   1% /var/clouddrive/decrypted

You can now use the decrypted directory to your hearts desire, minus rsync for now.

Conclusion

70 Bucks for unlimited storage is a steal. Mounting and encrypting in Linux is currently hell, and syncing even more so. But it does work and development is very, very active so we are bound to see massive progress in the weeks to come. In the meantime we can already store our data securely on amazon CloudDrive which will, depending on your data pile take quite a while (23 days in my case).

But even if we consider the downsides: This is stil oh-so-much cheaper than buying hard-disks that you have to manage yourself. I will never delete a file again, I got ‘/Attic’ on CloudDrive.

If you need help, consult each projects website for support/help.

-Christan.

Continue reading with the 2nd Part >>

Christian

Touched base with Linux back in 1995, got hooked up on it ever since. I am using Linux for both private and office for two decades. Working as a System Administrator at a medium sized hosting company I get in touch with all kinds of trouble. All of which can be solved with Linux. In my blog I am sharing solutions to problems that I had to search for myself in hope that someone else out there might find them useful.

15 thoughts on “Amazon Clouddrive, Encfs and Rsync

  1. Hello Christian,

    Thank you for your guide.
    I tried many others but even with yours I have a problem.
    Mounting the encrypted version of ACD looks okay.
    Sending it to ACD is okay (Via rclone or ACD-CLi)
    But decrypting the files never works…
    This is the result I get with your .sh script:

    Getting changes..
    Inserting nodes.
    fuse: mountpoint is not empty
    fuse: if you are sure this is safe, use the ‘nonempty’ mount option
    Traceback (most recent call last):
    File “/usr/local/bin/acd_cli”, line 9, in
    load_entry_point(‘acdcli==0.3.2’, ‘console_scripts’, ‘acd_cli’)()
    File “/usr/local/bin/acd_cli.py”, line 1609, in main
    ret = args.func(args)
    File “/usr/local/bin/acd_cli.py”, line 1096, in mount_action
    allow_root=args.allow_root, allow_other=args.allow_other)
    File “/usr/local/lib/python3.5/dist-packages/acdcli/acd_fuse.py”, line 716, in mount
    FUSE(ACDFuse(**args), path, subtype=ACDFuse.__name__, **kwargs)
    File “/usr/local/lib/python3.5/dist-packages/fuse.py”, line 480, in __init__
    raise RuntimeError(err)

    or this one now:
    (FileUtils.cpp:368) fatal: config file specified by environment does not exist: ~/encryption/reverse.xml

    Do you have any idea of what I did wrong ?

    1. “fuse: mountpoint is not empty” is your hint. You are trying to mount the encfs into an already (or not empty) directory. Please post your entire encfs-mount command and I’ll have a look.

  2. I’m running encfs 1.9.1 and was running into the follow error when executing the script:

    2016-10-11 19:12:41,244 ERROR [default] fatal: config file specified by environment does not exist: ~/encryption/encfs.xml

    > encfs 1.7.1 wants absolute paths
    So you need to change the path of the encfs.xml and pass files in the script.
    i.e ENCFS6_CONFIG=’ /root/encryption/encfs.xml’

    Thanks for the guide!

  3. This is hell, but a dam good way to learn Linux if you a noob!
    Just to add to the complexity of the setup I used the updated version of EncFS
    https://github.com/vgough
    I’ve know I dear if I have committed my self to that version but I seemed logical to use that until something like CryFS is off the groud

  4. Hi and thanks for this nice tutorial.

    I have a problem with the decrypted folder, it is empty all the time.

    root@debian:~# df -h | grep cloud
    ACDFuse 100T 20T 81T 20% /var/clouddrive/encrypted
    encfs 100T 20T 81T 20% /var/clouddrive/decrypted

    My Amazon cloud shows me encrypted files in /Storage.

    root@debian:~# ls -la /var/clouddrive/encrypted/Storage/
    total 0
    drwxrwxrw- 1 root root 0 Nov 1 14:07 .
    drwxrwxrw- 1 root root 0 Nov 1 13:36 ..
    drwxrwxrw- 1 root root 0 Nov 1 14:08 19TgxUm0wrlAaBlDquZ-6P34
    -rwxrwxrw- 1 root root 816 Nov 1 14:08 3EBTChPYExtylYi,fuYd7cy0
    -rwxrwxrw- 1 root root 1090 Nov 1 14:08 F8Sl9u-AbVeDeVjlH9qxmKEG

    root@debian:~# ls -la /var/clouddrive/decrypted/
    total 4
    drwxrwxrw- 1 root root 0 Nov 1 14:07 .
    drwxr-xr-x 5 root root 4096 Nov 1 15:01 ..

    i followed the tutorial 1 to 1

    any ideas ?

      1. Hi Christian,

        the problem was solved by using rclone.
        I tryed rsync for upload my files but the files was unvisible.

        After acd_cli sync i can see my files in the decrypted folder.

        I´m tested a upload with rclone direct to the decrypted folder it is possible too but much slower.

  5. Loved this! I was looking for a clever way to put my data into ACD securely and this was perfect! Curious, what is the advantage to using rclone of the reversed encfs mount versus using acdcli upload command? I’m testing now using acdcli -v upload reverse/ folder-in-acd/ and it seems to be working great and saturating my connection. Is there something I’m missing doing it this way?

    1. Hey Brian,

      see the comment above yours. 🙂
      At the time of writing acd_cli could not handle rsync at all (stalled/failed). This has improved a tiny bit. I am still using rclone; the cpu usage is way lower than using acd_cli (python vs native binary).

      But hey, if it works keep going!
      -Christian.

  6. Great tutorial! Thanks so much for taking the time to write this.

    I had a quick question regarding the encfs reverse mount.

    Is there a way to remount the reverse using the reverse.xml? It seems the normal encfs –reverse will just create a new encryption.

    Thanks!

    1. #1: This is a great tutorial, Christian. Thanks!

      #2: To remount the reverse drive using reverse.xml you run encfs like so:

      ENCFS6_CONFIG=’/home//encryption/reverse.xml’ encfs –extpass=”/bin/cat /home//encryption/reverse.pass” –reverse /media /var/clouddrive/reverse

      For the ENCFS6_CONFIG you need to use the full path name instead of depending on ~ (see earlier comments for more information).

      1. Ah I should have guessed the comment system would munch my command a bit. Change “/home//encryption” to “/home/YOURUSERNAMEHERE/encryption”.

  7. Hello Christian,

    This is a damn cool tutorial! Thank you for what you did and sharing it!

    I’m feeling a bit dumb to ask but i’m having a hard time to get passed “acd_cli init”. You hint “Hit a key, and (I am assuming you are running a headless server) w3m will open. Navigate in there, enter password”.
    So yes, i’m running a headless CentOS 7 VM. When i launch the command, i get a text browser and i can’t provide an answer to “What is you email (phone for mobile acount)”. I can “tab” to the password field but there is no field to fill for the account.

    Is there any way to pass this?

    1. Hey Guy,
      Thanks for your feedback 🙂
      Try tabbing there in W3 and hit enter once. This, if memory serves, activated the input field. If all else fails you can run it on your local PC (if you are running Linux) and simply copy over the created file.

      I did it remotely and it works. From there I just distributed the file.

Leave a Reply

Your email address will not be published. Required fields are marked *