LocalStack S3 setup for .net core development

If you use AWS S3 and don’t want to use the actual bucket for one or another reason for your local development, you’ve probably come across LocalStack S3 mock for this. Especially, if you are running your apps or integration tests in Docker.

TL;DR;

The source code of my console app which runs several tests against LocalStack S3 in Docker containers is on my GitHub.

LocalStack S3

In a nutshell, LocalStack is a mock server for many of AWS services including S3, and allows to run them locally, e.g. in a Docker container, so you could isolate your application from external dependencies, e.g. network connection, access/security policies in S3 bucket etc. And there is a nice UI to see your AWS services enabled.

NOTE – if you need 100% of features S3 is offering for your local development, it might be easier to create a real S3 bucket for the local development instead.

The problem

While it’s relatively easy to run the LocalStack S3 using the original docker image, however if you needed more advanced features of S3 you would probably run into a few technical challenges…

In this blog post I will be looking into following LocalStack S3 issues when using AWS SDK for .NET Core:

• Specify bucket name only, i.e. without the absolute URL.
• Use HTTP.
• Generate Pre-Signed URL that would work right away.

All of these issues were somewhat related.

NOTE – these issues apply to AWS SDK only. If you use AWS CLI, it’s enough to specify service URL to make LocalStack S3 work. See screenshot below.

Bucket name only

Using the absolute URL when specifying the bucket name looks strange but does work right away! This sets the correct endpoint URL and protocol to HTTP implicitly. However, if you tried to get an object or perform any other operation in AWS SDK by just using the bucket’s name, you would get Bucket Not Found response. The AWS SDK uses HTTPS by default.

// The code below works without much effort! 
var result = await _client.GetObjectAsync(new GetObjectRequest { BucketName = "http://localstack:4572/mybucket", Key = "file.txt" });

To setup the correct local endpoint by default you need to enable proxy, and configure it to use exactly the same hostname and port as LocalStack S3 service. It’s looks like a hack but forces communication to the localhost. Screenshot from config below.

Use HTTP

Forcing AWS S3 client to use HTTP would solve the issue described above. Luckily, there is a UseHttp flag in the config. Now getting bucket or objects works!

Too good to be true? Unfortunately, yes! Some request objects like GetPreSignedRequest don’t respect that and would return you a URL starting with https://.

// The request below returns HTTPS URL 
var result = _client.GetPreSignedURL(new GetPreSignedUrlRequest { BucketName = "mybucket", Key = Filename, Expires = DateTime.Now.AddHours(1) });

An ideal solution, however, would be running LocalStack under HTTPS but this would require more setup and I am tempted to leave this for another blog post :)

Generate Pre-Signed URL

Getting Pre-Signed URLs as mentioned above is a little bit more tricky. First, you need to explicitly set protocol to HTTP, and secondly need to enable ForcePathStyle flag in AWS S3 client’s config to have the URL formatted in hostname/bucket/file pattern. Otherwise, you will have the default bucket.hostname/file pattern, which would require different hosts setup.

// The following code generates working URL. 
var result = _client.GetPreSignedURL(new GetPreSignedUrlRequest { BucketName = "mybucket", Key = "file.txt", Expires = DateTime.Now.AddHours(1), Protocol = Protocol.HTTP });

The reward for this is double though! The URL works, and using the plain URL without the signature parameter would deny access to the object like in the real S3! Proof below :)

Dockerized solution

My solution (see docker-compose.yml) to run a few tests against Docker instance of LocalStack S3 looks following:

• LocalStack – LocalStack server running under http://localstack:4572.
• TestsApp – a set of tests demonstrating cases mentioned above. All tests implement ILocalstackTest interface, and new test cases could be added easily. It’s important to register tests in DI container in the correct order, i.e. download object *only* after it has been uploaded.

NOTE – to use LocalStack in the test app locally, e.g. for debugging etc., you need to setup localstack hostname in hosts file to point to the local machine IP address, e.g. see my hosts file below.

Testing

To run the LocalStack S3 and the tests app locally, clone my source code repo from GitHub, and run the command below in the root of the repo.

docker-compose up

Running any subsequent time when the source code has been modified, make sure to add –build flag. Otherwise, Docker would just load previously built images.

docker-compose up --build

Anyhow, you should see the following output.

All requests to LocalStack S3 have been completed in under 1s, and returned a successful response.

HINT – I use Cmder for convenient and user-friendly command line experience.

Conclusions

I use LocalStack for most of my AWS local development, also to run integration tests, and I find it really useful. The features set is good enough for a common scenario as well. For extreme cases you can always create a real S3 bucket in AWS.

Update 27/01/2019

Upgraded source code to the latest .NET Core 2.1.