rm-rf.ca

I've been hearing rumblings of awesomeness about Amazon S3 as a backup service from a couple friends lately. My current system could stand some improvement and I'd love something incremental and easier to do regularly, and with S3 being so highly recommended at only 15c a gig it looks like an ideal storage mechanism.

The next step is locating a tool to encrypt the data and do the actual uploads, I hear great things about Jungledisk but I'm not thrilled about a non-open source solution. The search led me to duplicity, a tool quite similar to rsync that does encrypted incremental backups to many different backends including S3.


export AWS_ACCESS_KEY_ID=X
export AWS_SECRET_ACCESS_KEY=Y
duplicity -v 5 --num-retries=50 --encrypt-key="YOURGPGKEY" --exclude-filelist ~/your_exclude_patterns.txt /home/whoeveryouare/ s3+http://yourbucket


I used an exclude file to keep things I don't want sent out, one line per directory. (no wildcards will be expanded) s3cmd was useful for working with buckets and examining their contents even as the upload was underway. It may be important to note that if duplicity is interrupted during a backup, you start over on the next run.

This kicked off great, my ISP appears to shape all encrypted traffic (to 30kb/s) but because this is over http I get proper upload speeds of about 125kb/s. This however saturates my bandwidth and suddenly web browsing is getting really slow, IRC over ssh starts chugging, and gaming is out of the question.

There appear to be many options to limit your own bandwidth, the first I found was tc but sadly you need a bloody phd to understand it. Thankfully the interweb has a few adaptable examples for what I was hoping to do, the best I found was here. I made a couple modifications to filter based on traffic going to a certain port (in our case https port 443):


#!/bin/bash

# Name of the traffic control command.
TC=/sbin/tc

# The network interface we're planning on limiting bandwidth.
IF=eth0 # Interface

# Upload limit (in mega bits)
UPLD=90kbps # UPLOAD Limit

# Filter options for limiting the intended interface.
U32="$TC filter add dev $IF protocol ip parent 1:0 prio 1 u32"

start() {

$TC qdisc add dev $IF root handle 1: htb default 30
#$TC class add dev $IF parent 1: classid 1:1 htb rate $DNLD
$TC class add dev $IF parent 1: classid 1:2 htb rate $UPLD
#$U32 match ip dst $IP/32 flowid 1:1
$U32 match ip dport 443 0xffff flowid 1:2
}

stop() {
$TC qdisc del dev $IF root
}

restart() {
stop
sleep 1
start
}

show() {
$TC -s qdisc ls dev $IF
}

case "$1" in
start)
echo -n "Starting bandwidth shaping: "
start
echo "done"
;;

stop)
echo -n "Stopping bandwidth shaping: "
stop
echo "done"
;;

restart)
echo -n "Restarting bandwidth shaping: "
restart
echo "done"
;;

show)
echo "Bandwidth shaping status for $IF:"
show
echo ""
;;

*)
pwd=$(pwd)
echo "Usage: tc.bash {start|stop|restart|show}"
;;

esac

exit 0

Adjust it as you see fit in terms of the upload speed you want to max out at and even the destination port if you're looking to limit some other kind of bandwidth. I'm particularly fond of this solution because it allows me to turn throttling on/off or adjust it's rate any time during the upload.

There's also trickle which allows you to limit bandwidth on a per process basis from user-land. Just start trickled, then run trickle and tell it what command you want to run. This does seem to imply you must start it from the get-go and can't adjust the rate dynamically during the transfer, but still a very nice solution.