a0002

Friday, February 1, 2013

serve static files with Amazon S3

Using Amazon S3 to serve static files:
Using lighttpd with secdownload to serve files
Amazon S3

It is a lot cheaper than renting a server which is just sitting idle 99.9% of the time.
Unlimited storage. I can just keep uploading files to S3 without having to worry about how much space I am using.
It should be far more reliable than just a single static file server.

Instead of rsync, there is a tool called S3Sync that can be used to transfer files to amazon S3. It took some time getting it to work though. Initially it didn't support EU buckets (it does now), and it is less resilient. Whenever the connection got interrupted, the script would display a Broken Pipe message and get stuck in retries. But thanks to a post on the forum I was able to modify the script so that it would recover from such errors. Note that by now a new version has been released which probably fixes this problem - but I haven't upgraded yet. Currently the only drawback is that there is no support for bandwidth limiting so I'll have to use another tool for that.
On the PHP side I had some difficulties getting the signing to behave correctly. Most of the examples I found online had issues with filenames containing spaces or slashes. I eventually used the function below, I can't guarantee that it is correct but haven't seen any problems with it so far.

<?php
// grab this with "pear install --onlyreqdeps Crypt_HMAC"
require_once('Crypt/HMAC.php');

// Amazon S3 credentials
define('S3_ACCESS_KEY_ID', 'your-S3-access-key');
define('S3_SECRET_ACCESS_KEY', 'your-S3-secret-access-key');

/**
 * Generate a link to download a file from Amazon S3 using query string
 * authentication. This link is only valid for a limited amount of time.
 *
 * @param  $bucket    The name of the bucket in which the file is stored.
 * @param  $filekey   The key of the file, excluding the leading slash.
 * @param  $expires   The amount of time the link is valid (in seconds).
 * @param  $operation The type of HTTP operation. Either GET or HEAD.
 */
function mymodule_get_s3_auth_link($bucket, $filekey, $expires = 300, $operation = 'GET') {
$expire_time = time() + $expires;

$filekey = rawurlencode($filekey);
$filekey = str_replace('%2F', '/', $filekey);

$path = $bucket .'/'. $filekey;

/**
         * StringToSign = HTTP-VERB + "\n" +
         *                           Content-MD5 + "\n" +
         *                           Content-Type + "\n" +
         *                           Expires + "\n" +
         *                           CanonicalizedAmzHeaders +
         *                           CanonicalizedResource;
         */
$stringtosign =
$operation ."\n".   // type of HTTP request (GET/HEAD)
"\n".               // Content-MD5 is meaningless for GET
"\n".               // Content-Type is meaningless for GET
$expire_time ."\n". // set the expire date of this link
"/$path";           // full path (incl bucket), starting with a /

$signature = urlencode(mymodule_constructSig($stringtosign));

$url = sprintf('http://%s.s3.amazonaws.com/%s?AWSAccessKeyId=%s&Expires=%u&Signature=%s', 
$bucket, $filekey, S3_ACCESS_KEY_ID, $expire_time, $signature);

return $url;
}

function mymodule_hex2b64($str) {
$raw = '';
for ($i=0; $i &lt; strlen($str); $i+=2) {
$raw .= chr(hexdec(substr($str, $i, 2)));
}
return base64_encode($raw);
}

function mymodule_constructSig($str) {
$hasher =& new Crypt_HMAC(S3_SECRET_ACCESS_KEY, 'sha1');
$signature = mymodule_hex2b64($hasher-&gt;hash($str));
return $signature;
}
?>