Archive for December, 2011

Testing Paperclip generated expiring S3 urls with RSpec, Cucumber and Timecop

December 21st, 2011

The need

I have a Rails app that is using Paperclip to generate expiring urls for files stored in S3. The urls are set to expire after 1 minute. As much as i trust Paperclip and Amazon, I need tests that prove that these generated urls do in fact expire on time, and that visitors to those files after they’ve expired are prevented from accessing the file.

Disclaimer

This has been a bit of a rush, so no doubt i’ll refactor and tidy the code and this post laster today / in the week.

The solution

I’ve used RSpec and Cucumber to check expiring urls that the system generates to ensure they expire successfully. RSpec simply checks that a generated url includes the Expires parameter and it’s value is set exactly to 60 seconds from now. Cucumber goes further than this by uploading files and checking if they are accessible before and after expiration.

RSpec to simply test that the expiration time generated for a link is correctly set to 1 minute

This test simply asks the model containing the attachment (in this case an “Asset” model), how many seconds from now remain before the attachment expires.

Spec

describe Asset do

it “should return an attachment link that expires within 1 minute” do

asset = Factory.build(:asset)
asset.seconds_until_attachment_expires.should == 60

end

end

This depends on a few new methods in the Asset model class, which take care of extracting the Expires param from the expiring url, and comparing to Time.now.

Asset Model Class

First, we create an instance helper method that returns the number of seconds an object’s url has left before it expires

def seconds_until_attachment_expires

Asset.seconds_until_attachment_expires(expiring_attachment_url)

end

I decided to pass the responsibility of calculating this number to a class method. I did this because the Cucumber tests need to request the same calculation for urls that were generated in the past. If they interacted with an instance of the Asset class, by default it would return a new url each time it was asked. So, rather than clutter up the instance method with a decision about whether to issue a new url or return an existing one, i simply passed the responsibility to the class. That seems to work for now, although I might refactor it later.

Next, we create the class level method that calculates time left until expiration. This accepts a url, meaning we can test urls generated now or in the past

def self.seconds_until_attachment_expires(url)

seconds = attachment_expiration_in_seconds_from_epoch(url) – Time.now.strftime(”%s”).to_i
seconds.round

end

This method simply strips the time from the generated url (via the attachment_expiration_in_seconds_from_epoch method) and rounds the value.

def self.attachment_expiration_in_seconds_from_epoch(url)

url.split(”&”).second.split(”=”).last.to_i

end

Clearly, this is tightly coupled to the format of the generated url string, so a cleaner way should be sought. However, for now, this method is only used in the tests and it does work, so it’ll do for the moment.

Finally, to ensure that Rspec, Cucumber and the app all interact with a url generated exactly 60 seconds from now, we create a model instance method that generates the link. All requests for the link call this method.

def expiring_attachment_url

attachment.expiring_url(60)

end

Cucumber to test actual file access via the browser

Cucumber takes things 1 step further. It interacts with all the same methods that we created on the Asset model, but also goes off and uploads attachments and then tries to access them before and after they’ve expired. We use Timecop to create expired urls, and a Cucumber before hook to ensure all scenarios run from the current time by default.

Scenarios

@selenium
Scenario: Viewing an active attachment on an object

Given some object has been created and a plain text file attached
When I visit the object’s attachment url
Then I should see the contents of the uploaded attachment
And I should not see “Request has expired”

@selenium
Scenario: Viewing an expired attachment on an object

Given some object has been created and a plain text file attached
When I visit the object’s attachment url after it has expired
Then I should not see the contents of the uploaded attachment
And I should see “Request has expired”

features/support/hooks.rb

Before do

Timecop.return

end

NB: For the sake of completeness (even though we’re not calling Timecop from our Rspec specs), to be completely satisfied that Timecop isn’t affecting our specs in any unexpected way, we add the same to spec_helper.rb too:

spec/spec_helper.rb

config.before do

Timecop.return

end

config.before do
Timecop.return
end

steps

This is where Timecop offers a wonderfully simple way of generating expired urls.

And /^I visit the question’s attachment url after it has expired$/ do

#First, go back in time 2 minutes and generate the expiring url, and make sure it’s set to expire in 1 minute
Timecop.freeze(Time.now – 2.to_i.minutes) do

@url = current_object.asset.expiring_attachment_url
Asset.attachment_expires_in(@url).should == 60

end

#Next, return to the current time and make sure the previously generated expiring url has now been expired for 1 minute
Timecop.return
Asset.attachment_expires_in(@url).should == -60

#Finally, go visit the expired url
visit @url

end

When run, cucumber correctly reports that expired urls result in the user seeing the message “Request has expired”, and non-expired urls correctly provide access to the uploaded file.

Summary

Although a rough and ready solution, and most likely needing refactoring, it does provide us with a way to test expiration of uploads to S3.

I hope you found this useful.

Making S3 file uploads private yet accessible via your rails app using Paperclip

December 20th, 2011

The need

I needed to make file uploads to S3 secure. Files must be private in S3 (i.e. not accessible if someone guessed the url), yet accessible via the rails based web app.

The solution

Add this to the rails model:

has_attached_file :attachment,

:storage => :s3,

:s3_credentials => ‘config/amazon_s3.yml’,

:s3_permissions => :private

Replaced this in the view:

Replaced: asset.attachment.url

With: asset.attachment.expiring_url(60)

This keeps the files private, and the app creates expiring urls as users request them, which expire after a minute.

What Amazon Say

Query String Request Authentication Alternative

You can authenticate certain types of requests by passing the required information as query-string parameters instead of using the Authorization HTTP header. This is useful for enabling direct third-party browser access to your private Amazon S3 data, without proxying the request. The idea is to construct a “pre-signed” request and encode it as a URL that an end-user’s browser can retrieve. Additionally, you can limit a pre-signed request by specifying an expiration time.

Creating a Signature

Following is an example query string authenticated Amazon S3 REST request.

GET /photos/puppy.jpg
?AWSAccessKeyId=0PN5J17HBGZHT7JJ3X82&Expires=1141889120&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D HTTP/1.1
Host: johnsmith.s3.amazonaws.com
Date: Mon, 26 Mar 2007 19:37:58 +0000

The query string request authentication method doesn’t require any special HTTP headers. Instead, the required authentication elements are specified as query string parameters:

Query String Parameter Name Example Value Description
AWSAccessKeyId 0PN5J17HBGZHT7JJ3X82 Your AWS Access Key Id. Specifies the AWS Secret Access Key used to sign the request, and (indirectly) the identity of the developer making the request.
Expires 1141889120 The time when the signature expires, specified as the number of seconds since the epoch (00:00:00 UTC on January 1, 1970). A request received after this time (according to the server), will be rejected.
Signature vjbyPxybdZaNmGa%2ByT272YEAiv4%3D The URL encoding of the Base64 encoding of the HMAC-SHA1 of StringToSign.

The query string request authentication method differs slightly from the ordinary method but only in the format of the Signature request parameter and the StringToSign element. Following is pseudo-grammar that illustrates the query string request authentication method.

Signature = URL-Encode( Base64( HMAC-SHA1( YourSecretAccessKeyID, UTF-8-Encoding-Of( StringToSign ) ) ) );

StringToSign = HTTP-VERB + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Expires + "\n" +
    CanonicalizedAmzHeaders +
    CanonicalizedResource;

YourSecretAccessKeyID is the AWS Secret Access Key ID Amazon assigns to you when you sign up to be an Amazon Web Service developer. Notice how the Signature is URL-Encoded to make it suitable for placement in the query-string. Also note that in StringToSign, the HTTP Date positional element has been replaced with Expires. The CanonicalizedAmzHeaders andCanonicalizedResource are the same.

Example Query String Request Authentication

Request StringToSign
GET /photos/puppy.jpg?AWSAccessKeyId=0PN5J17HBGZHT7JJ3X82&
    Signature=rucSbH0yNEcP9oM2XNlouVI3BH4%3D&
    Expires=1175139620 HTTP/1.1

Host: johnsmith.s3.amazonaws.com
GET\n
\n
\n
1175139620\n

/johnsmith/photos/puppy.jpg

We assume that when a browser makes the GET request, it won’t provide a Content-MD5 or a Content-Type header, nor will it set any x-amz- headers, so those parts of the StringToSign

has_attached_file :attachment,
:storage => :s3,
:s3_credentials => ‘config/amazon_s3.yml’,
:s3_permissions => :private