When AWS CloudFront Breaks WordPress

Awhile ago I moved this blog to an AWS EC2 instance and, to try and reduce load on the instance, put the blog behind Amazon CloudFront, since CloudFront makes a decent front-side cache and has a free tier.

All seemed well until one day I went to update an old blog post, and upon saving the post was met with this response:

Updating failed. The response is not a valid JSON response.

Uhhh… what?

So I did some digging, and I immediately saw HTTP 403 (Forbidden) errors in my web browser’s console from an endpoint called autosaves. I then checked my webserver logs, but saw no sign of the URL there, nor any errors. Turns out that CloudFront was blocking the request, but why?

Even stranger was when I tried doing a simple post to the autosaves endpoint, I had no issues. But if I went into the Network tab in Chrome, click Copy as cURL command on the offending request in question, and ran that on the command line, I was able to reproduce the issue. So CloudFront was only blocking some requests… but why?

It turns out that, by default, the web ACLs that CloudFront installs on new distributions will prevent you from making a POST or PUT request where the body is larger than 8 K. Reasonable for most websites, but sadly not WordPress.

I’ll show you how to fix that.

Continue reading “When AWS CloudFront Breaks WordPress”

What’s Taking Up So Much Space in AWS S3?

I’m a big fan of Amazon S3 for storage, and I like it so much that I use Odrive to sync folders from my hard drive into S3 use S3 to store copies of all of my files from Dropbox as a form of backup.  I only have about 20 GB of data that I truly care about, so that should be less than a dollar per month for hosting, right? Well…

“You are not your job or how much data you have in S3!”

Close to 250 GB billed for last month. How did that happen?

Continue reading “What’s Taking Up So Much Space in AWS S3?”

Dockerizing Discord’s Music Bot in Amazon ECS

I’m a big fan of the Discord Musicbot, and run it on some Discord servers that I admin. Wanting to run it on a server, I first created an Ansible playbook and launched a server on Digital Ocean. But after a few months, I noticed that the server was sitting over 90% idle. Surely there had to be a better way.

So I next tried Docker, and created a Dockerized version of the Musicbot. I was quite happy with how much easier it was to spin up the bot, but still didn’t want to run it on a dedicated server on Digital Ocean. Aside from having unused capacity, if that machine were to go down, I’d have to do something manually.

I thought about running it in some sort of hosted Docker cluster, and came across Amazon’s container service. So this post is about creating your own cluster in ECS and hosting a Docker container in it. I found the process slightly confusing when I did it the first time, and wanted to share my experience here.

Continue reading “Dockerizing Discord’s Music Bot in Amazon ECS”

How to Undelete Files in Amazon S3

While S3 is a great storage platform, what happens if you accidentally delete some important files? Well, S3 has a mechanism to recover deleted files, and I’d like to go into that in this post.

First, make sure you have versioning enabled on your bucket. This can be done via the API, or via the UI in the “properties” tab for your bucket. Versioning saves every change to a file (including deletions) as a separate version of said object, with the most recent version taking precedence. In fact, a deletion is also a version! It is a zero-byte version which has a “DELETE” flag set. And the essence of recovering undeleted files simply involves removing the latest version with the “DELETE” flag.

This is what that would look like in the UI:

To undelete these files, we’ll use a script I created called s3-undelete.sh, which can be found over on GitHub:

Continue reading “How to Undelete Files in Amazon S3”

Notes from “Scaling on AWS for the First 10 Million Users”

Earlier tonight, I had the pleasure of attending a presentation from Chris Munns of Amazon at the offices of First Round Capital about scaling your software on AWS past the first 10 million users. I already had some experience with AWS, but I learned quite a few new things about how to leverage AWS, so I decided to write up my notes in a blog post for future reference, and as a service to other members of the Philadelphia tech community.

Without further preamble, here are my notes from the presentation:

Continue reading “Notes from “Scaling on AWS for the First 10 Million Users””