Quick Explanation Of The S3 Consistency Model
**Update 1/12/2021**
This year AWS announced an update to the S3 consistency model, making the caveats represented in this post old and outdated. I have put together a new post, The S3 Consistency Model Got an Upgrade, that you can check out for the latest breakdown.
******
Folks that are switching to AWS are often looking to leverage S3 in some form or another. It is one of the foundational services in the AWS ecosystem.
It also fits a wide variety of use cases. It fits anything from an event driven processing pipeline to a blob store for lots of objects. There is a lot of unique ways to use S3, and a lot of ways you should not use it. Every piece of technology has limits and tradeoffs.
A common tradeoff that gets asked about a lot is the consistency model of S3. There is a number of different consistency models across computer science. The AWS documentation for S3 states the following for their consistency model.
Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD or GET request to the key name (to find if the object exists) before creating the object, Amazon S3 provides eventual consistency for read-after-write.
Consistency Scenarios
Many developers have fallen into the trap of only reading the first part of this documentation. But there is caveats here that should be thought about.
First let’s walk through the happy path scenario.
PUT /key-prefix/cool-file.jpg 200
GET /key-prefix/cool-file.jpg 200In this scenario we have read-after-write consistency. We PUT a new object, cool-file.jpg, into the bucket. The next operation we performed was a GET for that same object. We see that both operations were successful and returned 200 response codes.
Let’s go through a caveat scenario as mentioned in the AWS documentation.
PUT /key-prefix/cool-file.jpg 200
PUT /key-prefix/cool-file.jpg 200 (new content)
GET /key-prefix/cool-file.jpg 200Here we are making a PUT request for our new object. Then the next operation we perform is another PUT request. This second request is overwriting the previous object with new content. Our final operation is then a GET request for that object.
This is a key caveat to the AWS documentation. We are overwriting an existing key in the bucket before we read it back out. We no longer have read-after-write consistency in this scenario. Instead we have eventual consistency. Our GET request could return the content of the first PUT or the content of the second PUT. That will remain true until the change propagates.
Now let’s examine the other caveat scenario.
GET /key-prefix/cool-file.jpg 404
PUT /key-prefix/cool-file.jpg 200
GET /key-prefix/cool-file.jpg 404This is a view of the caveat mentioned in the documentation. Here we make a GET request for our object before it is created. Of course the object does not exist at this point so the response is 404 Not Found. Then we PUT our new object in the bucket and get a 200 response code. The final operation is then another GET request for the object, but we get a 404 Not Found again.
In this scenario, S3 has eventual consistency. The 404 response for the first GET is returned until the PUT propagates.
Conclusion
AWS is a very powerful ecosystem. But, there are times when their documentation is hard to decipher. This can make learning AWS difficult and lead to more questions then answers. My hope is that I have provided at least a little more clarity on the caveats of the S3 consistency model. If you have any questions or comments please feel free to reach out.
Learn AWS By Actually Using It
If you enjoyed this post and are hungry to start learning more about Amazon Web Services, I have created a new course on how to host, secure, and deliver static websites on AWS! It is a book and video course that cuts through the sea of information to accelerate your learning of AWS. Giving you a framework that enables you to learn complex things by actually using them.
Head on over to the landing page to get a copy for yourself!
