(Updated) Syncing your Data: cloudhq.net

The Situation

After last weeks Amazons ban of rclone and acd_cli it was nigh impossible to access and sync your data reliable with pretty much anything. Sure, Amazon offers a free Windows client but that a) works only in Windows and b) does not make the drive available as a local drive. Plus rclone allows syncing of two clouds making them a de facto Raid1. It worked very well and due to the fact that it was scriptable you could fire-and-forget it. I never had any reason to look the other side.

Until now. I still don’t.

Update August 2018

I received several request about CloudHQ and a follow up review, ranging from pure curiosity to buyers recommendation to frustrated people. Update time!

After using CloudHQ for 3 months I am ready to share my findings. First off, let’s sort out some basic facts. My main Google Drive (source) has about 47TB of used data. That data should have been copied 1-way to another Google Drive and an Amazon Clouddrive. My daily delta (amount of new Data daily that pours into the Google drive) is about 200GB per day, that’s roughly 1.4TB per week of additional data.

Please keep in mind that CloudHQ aims for enterprise customers, so transferring large amount of data and transfering it fast should be a given. CloudHQ sends a weekly report of amount of data transferred, here are the last few weeks of data:

  • 06/19 – 06/26, Files copied last week: 2509, Data copied last week: 1.2 TB.
  • 06/26 – 07/03, Files copied last week: 5351, Data copied last week: 1.9 TB.
  • 07/03 – 07/10, Files copied last week: 130934, Data copied last week: 1.5 TB.
  • 07/10 – 07/17, Files copied last week: 116569, Data copied last week: 1.9 TB.
  • 07/17 – 07/24, Files copied last week: 98979, Data copied last week: 1.7 TB.
  • 07/24 – 07/31, Files copied last week: 38397, Data copied last week: 2.6 TB.

That’s roughly 1.8TB per week on average. That boils down to a transfer speed of 23.80Mbit/s. To put that in contrast: My home DSL Upload speed has a 40Mbit/s. Given the fact that the new data delta per week is roughly 1.4TB and CloudHQs weekly transfer speed of 1.8TB that leaves 0.4TB per week to catch up on the current 47TB. That would boil down to 117 weeks and change… or 2.2 years.

But wait! Didn’t you say you copied to Amazon and another Google drive?” Wow, you’re paying attention! And you’re right. The CloudHQ transfer report lists a combined value of transferred data across all sync pairs. Let’s ignore the fact that this would half the speed of the delta catchup, this will at the very least double the time needed to a whooping 4.4 years.

If I dedicate one of my servers to sync with it’s 1gbit port it can easily copy data from Google Drive to Amazon CloudDrive at 800Mbit/s, resulting in a completion of all data in 6 days. For a company that focuses on Enterprise customers I am surely doing something wrong. So I contacted the support with the above findings. At first there was no response, so I fired a second email:

Hey folks,

I have not received any reply from you regarding this matter yet.
Please be informed that I need to update my blog posting and inform my
visitors who are awaiting a reply this evening.

I will give you a heads up on the corrected review once alterations have
been made. If you need a statement on the blog other than "CloudHQ did
not respond to any inquiries" please at the least drop us a note in the
next few hours; editing of the article is already underway.

With kind regards,
Christian Reiss.

only a few hours they replied:

Hi Christian,

Thank you for reaching out to us about this and sorry for the hassle here.
I have checked logs for your account that sync pairs are running fine. These specification are definitely something we do and size like yours should not pose any problem for our system. As you probably noticed, we focus mainly on Business and Enterprise customers and terabytes of data is not really a lot for us.
Nevertheless, I have escalated your issue to the technical team to check the logs with Amazon and Google Drive if they are throttling your account.

Incidentally, we have checked that we are unable to access your blog post. May I remind you that part of ToS is that the blog must be accessible to the public at all times.

All the best,

My network is offline? That alerted me, I inquired further:

Hey [name removed],

thanks for escalating the issue.
I am currently on the road, so only a brief update here. I instructed to
hold off on updates regarding the post for now pending your findings.

The site https://alpha-labs.net/ is mirrored on 4 locations across the
globe via HA with an yearly uptime of 99.9997% . I am taking
unavailability issues seriously. Would you mind telling me your country
of origin and your requesting IP (you can leave out the last part for
anonymity if you desire). I will check the firewall block logs.

With kind regards,
-Christian.

I checked monitoring across the globe, looked through all the firewall logs. Nothing. A reply arrived:

Hi Christian,

Our technical team had looked into this and your sync pairs are running fine.
Sadly, there is nothing we can do here with regards to the speed as everything is ok.
My apologies for the hassle here. If you have other concerns, please let me know and I'll be glad to assist.

Sincerely,

I am not here to judge. That is up to you, but here are my suggestions:

If you are a corporate admin and need to transfer large amounts of data, use your own infrastructure combined with rclone. It can transfer data as fast as your network allows. You can even throttle it if network throughput if of any concern.

If you are a private user, pretty much the same applies. Copying your files over your local DSL Uplink at 23Mbit or more is on par.

Closing thoughts: The idea of a fire and forget method to sync your digital belongings is sweet; but that needs to be fast. There are convenient tools out there to do the job yourself, but you want to offload the burden of the bandwidth usage to someone else; you want the speed. On the other hand I guess you get what you pay for at a price of 20 bucks.

I am leaving the original posted article below for prosperity.

-Christian


It broke down

So I woke up this morning and noticed nothing was syncing, everything was breaking down. Re-Authing does not work as Amazon revoked all the API Keys for those applications leaving the internet out there with nothing to work on/with. And to make things even worse: I just got an additional Google Drive account that had zero files in it. Without any working 3rd party apps (and I also tried netrdrive, clouddsync, multicloud) my data was not deleted- but stuck.

Introducing CloudHQ

Then, more by chance, I discovered cloudhq.net. This is a hosted synchronization service that syncs Clouds of your choice, including Amazon and Google Drive. Registration is done via the usual form which will, after activation drop you inside a basic web-interface. But don’t let that scare you off; basic in this case means reduced to the maximal stuff needed. The Dashboard is where you configure your sync jobs which I set as a two-way sync. A multi-sync (many-to-one) is also available and they cover most all of the major players out there.
In my case I setup a sync job between Amazon CloudDrive and Google Drive, where the first copies data into the second one. I am using my CloudDrive as cold storage (read only) while my Google Drive is my main read-write repository. During the setup of a job you get to choose (among what clouds you want to sync) whether you want a one-way, two-way, with file deletion, without file deletion, backups… It pretty much covers all the settings I could think of.

The beauty is that you set up a job on their servers; a running instance on your premises is not required. I am okay with the transfer speeds: Roughly 2.5-3.0TB/day on average- but that is entirely Amazons fault. I set up a test sync with a second Google Drive and it was roughly twice as fast.

I even opened up a support ticket with their staff to query about transfer speeds and whether two sync jobs would count toward an account side limit; turns out there is no limit. The only limitation are the syncing clouds. Sure it could sync a tiny bit faster but I don’t need an online server, pay for traffic or power. I chose the convenience here.

In Conclusion

If you need to move massive amounts of data beyond single-digit Terra-bytes you do not want to download and re-upload everything. Let someone else do that for you. It’s not a free service but considering the bandwidth usage and the fees on my infrastructure this is a huge savings for me. Check them out here: http://www.cloudhq.net/

On a sidenote: I am not invited by cloudhq.net; I am paying for my own account.

Hope this helps choosing on how you transfer.
-Christian.

Christian

Touched base with Linux back in 1995, got hooked up on it ever since. I am using Linux for both private and office for two decades. Working as a System Administrator at a medium sized hosting company I get in touch with all kinds of trouble. All of which can be solved with Linux. In my blog I am sharing solutions to problems that I had to search for myself in hope that someone else out there might find them useful.

Leave a Reply

Your email address will not be published. Required fields are marked *