dmuth's blog

Notes from "Scaling on AWS for the First 10 Million Users"

Earlier tonight, I had the pleasure of attending a presentation from Chris Munns of Amazon at the offices of First Round Capital about scaling your software on AWS past the first 10 million users. I already had some experience with AWS, but I learned quite a few new things about how to leverage AWS, so I decided to write up my notes in a blog post for future reference, and as a service to other members of the Philadelphia tech community.

Without further preamble, here are my notes from the presentation:

  • "Auto scaling is a tool and a destination"
  • AWS regions are connected via the public Internet, but availability zones in the same region are connected via Amazon-owned fiber. So AZ to AZ communication is entirely across Amazon's infrastructure.
  • More capacity is added to AWS's infrastructure each day than powered all of in 2003. That's a lot of servers.
  • DynamoDB is based on a whitepaper that Amazon wrote back in 2003 or so.
  • DynamoDB is multi-AZ.
  • SQL is a well established technology with clear patterns to scalability (explained further down), and no matter what you think--your data is not special.
  • "Everything fails all the time"
    • Build with that in mind
    • "You cannot have greater than 4 nines in a single data center"
  • Elastic Load Balancers are Availability Zone independent
    • "session stickiness" can route repeat requests from the same person to the same EC2 instance
  • S3 and CloudFront should be used for static assets from your website.
    • This falls under "outsource your infrastructure" -- use Amazon's CDN instead of implementing your own
  • ElasticCache is better than a database for caching (less load) or local files (less invalidation issues)
    • It is hosted Memcache
  • S3 has 11 9s of durability
  • DynamoDB scales really well. Individual customers have done 500,000+ reads/writes with no issue.
  • Auto scaling lets you match capacity to load
Average: 3.3 (6 votes)
Your rating: None

An American's Visit to Sweden and Denmark

I found myself between jobs a short awhile ago, and suddenly had a few weeks of downtime. Since I had an open invitation to visit Sweden and Denmark from friends living there, I figured I would take the opportunity and head over. I took a few pictures, and noted some observations about those countries.

First, Know Some Locals

If you want to take a package tour and see the parts of the country that are tourist attractions, feel free. But you'll get much more out of the experience if you know locals who can show you things that they think are interesting. I was fortunate to know both Pinky Fennec and Joel Fox, who were more than happy to host me while I visited.

Public Transit is Amazing

A subway station on the Hågsatra line

The train system in Stockholm is pretty extensive. Trains and busses run pretty much everywhere you'd want to go. Both systems used a wireless card that you could buy with a week's worth of fare, and just swipe it at the turn style or at the front of the bus. Subway stations had displays above the tracks stating how much time until the next train would arrive. When bus schedules said that busses arrived on the quarter of every hour, they meant it.

It was also pretty much the same in Copenhagen, except I spent more time on busses. The bus service there was simply phenomenal. They had busses that would arrive every 8 minutes, and they meant it. I regularly commuted from a friend's place on the outskirts of Copenhagen to the central part of the city every single day.

Getting from city to city was also fun. I rode on what amounted to the Swedish version of Amtrak from Stockholm to Copenhagen. It was a single train ride, about 400 miles in 5 hours. That's an average of 80 mph, but with the stops we made I'd say we did somewhere between 90 and 100 mph most of the time. The train ride was exceptionally smooth, and several times when we started moving I thought that the train next to us was moving. Naturally, these trains were also on time.

Danish Cheetah
This Danish cheetah tried to sell me a scarf!

Public transit here in the US could learn a few things from how they do it in these two cities.

Be Prepared to Walk Everywhere

What comes with taking lots of public transit? Lots of walking. Be ready to walk everywhere. This is actually a good thing, because of the benefits of the exercise it provides. One thing I noticed in both countries was pretty much a total lack of obesity in the native populations. This was especially noteworthy given the amount of drinking I saw in both countries.

Denmark Has Amazing Pastries

My first morning in Denmark, Pinky said "BRB" and ran a half-block down the street to the local bakery. She returned with a bundle of amazing pastries that were no more than a few hours old. They made for an amazing breakfast.

I wish I could remember what these delicious pastries were called, but I do know that the Danish just can't get enough of them!

Average: 3.3 (6 votes)
Your rating: None

So I got invited to try out Google Glass

"Whoa. This is heavy."

So I got invited to try out Google Glass the other day. I'm looking forward to this, since this something that really hasn't been done before, at least not in the "voice-activated computer glasses with an Internet connection" sense.

A few ideas that I'm thinking of using the glasses for:

  • Using facial recognition on my friends, possibly with GPS
  • Hitting SEPTA's website to see what the next train to leave is.
  • Finding my way around new parts of Philadelphia. (the nicer parts, that is Eye-wink )
  • Streaming portions of Anthrocon
  • Taking some "behind the scenes" video of the setup in the days before Anthrocon, or during setup at the convention itself.

Have your own suggestions for how I should use the glasses? Let me know in the comments!

Average: 2.4 (7 votes)
Your rating: None

The Cira Centre on a Cloudy Day

I took some pictures of the Cira Centre while walking around Philadelphia today:

The Cira Centre The Cira Centre
The Cira Centre The Cira Centre

The Regional Rail section of 30th Street Station can be seen in the foreground.

Anyway, like... favorite... let me know what you think in the comments!

Average: 3.4 (7 votes)
Your rating: None

Running Dorsai Thing 2013: Lessons Learned

Dinner on Saturday Evening

Each year, The Dorsai Irregulars have their own annual convention known as "Dorsai Thing". It is a weekend long event, held in a different city each year, where we all gather, socialize, and hold our semi-annual business meeting. This year, it was held in Phildelphia, and it fell on my shoulders to run the event, dubbed "Liberty Thing" While I've worked at plenty of conventions before, this was the first one where I was effectively the Con Chair. This was an entirely new thing for me, and I wanted to write a blog post about some of the things I learned during the experience.

Don't Procrastinate!

It's easy to let things slide when you're several months out from the convention. But remember, you never know when life is going to throw you a curveball that eats up your spare time. In my case, a job change combined with my bank failing suddenly began to eat up a lot of my free time, and it left me feeling under pressure to get things done. The lesson here is to try and get things out of the way early.

Hotels: Get a Small Room Block

A room block is a guarantee of room nights that you will give the hotel during the convention. If the hotel doesn't sell enough room nights to fill the block, they will often reserve the right to come after you for the difference in revenue. This sounds harsh, but remember that if a hotel does not rent a room for a given night, that's lost revenue which it will never be able to regain. Hotels really don't like empty rooms. Smiling

In my case, I contracted a room block of 40 nights. We ended up renting nearly twice as many rooms. The hotel really liked that!

Average: 2.6 (5 votes)
Your rating: None

Setting up IPv6 on Linode with nginx

So Linode has gotten onto the IPv6 bandwagon. In addition to each Linode getting an IPv6 address, they will also gladly assign an entire /64 netblock to your specific node on request. That gives you all sorts of flexibility for bringing up IPv6 services. And this blog post is going to be all about how to make use of multiple IPv6 addresses.

As of this writing, everything I mention will work on a Linode running Ubuntu 12.04 LTS and Nginx 1.1. There's no reason that these instructions shouldn't work on other distros or versions of Ubuntu with some modifications.

Setting An IPv6 Address

Warning: You're playing around with networking settings. If something goes wrong, your Linode could find itself unreachable from the network and you'll need to reboot it. And that's not fun. I strongly recommend proceeding through the LISH console so that you can undo any mistakes you make.

To start with, send a support request to Linde and get your /64 assigned. For the purposes of this post, let's assume you get the subnet 2600:3000:4000:1000/64. That will give you a range of 2^64 addresses from 2600:3000:4000:1000:0:0:0:0 to 2600:3000:4000:1000:ffff:ffff:ffff:ffff. That should be plenty of addresses to work with. Eye-wink

To bring up one of those addresses, run this command:

ip -6 addr add 2600:3000:4000:1000::100/64 dev eth0

The address should now show up when you run ip -6 addr show eth0, and you should be able to use the ping6 utility to ping 2600:3000:4000:1000::100. If ping6 is not working, then something is wrong.

$ ip -6 addr show eth0
3: eth0:  mtu 1500 qlen 1000
    inet6 2600:3000:4000:1000::100/64 scope global 
       valid_lft forever preferred_lft forever

Average: 2.6 (15 votes)
Your rating: None

R.I.P. Casey

Casey Sitting on a Chair
The guilty cat, himself!

We know you killed those mice, dude.

My parents' cat Casey passed away early last week. He managed to live to be 20. That's not so bad for a cat that grew up on the streets and was eventually rescued by Forgotten Felines and Fidos. My parents adopted him when he was just a few years old, and gave him a loving home for the rest of his life.

That's not to say that Casey wasn't a jerk from time to time. Later in life, he took on the "grumpy old man" persona, and would eat pretty much anything he could get his paws on, just because he could. He'd even catch mice. That cat just did not care.

Anyway, here's a video that we shot sometime last year--in which Casey tries to steal food while we are seated at the table:

Average: 2.8 (8 votes)
Your rating: None

Scaling Anthrocon's Website to Handle 1,400 Simultaneous Connections


The Challenge

When hotel reservations open, that is the single busiest time of the year for Anthrocon's webserver. In fact, it even caused us performance problems last year. That was not so good.

So this year, I decided to try something different. Instead of leaving the regular website up and running, which involves using Drupal, I instead decided to replace the entire page with a relatively static "countdown" page, which displayed a countdown timer and automatically started displaying the hotel link at 11 AM on the opening day.

First, some stats for the Anthrocon website:

  • Peak bandwidth: 1.6 Megabits/sec
  • Peak connections: 1,400 concurrent connections

And now some status for Passkey, who handled most of the traffic:

  • Peak bandwidth: 190 Megabits/sec
  • Peak connections: 4,000 concurrent connections

Lightening the Load on the Webserver

Average: 3 (11 votes)
Your rating: None

Web Development on Port 80 and 443 in Vagrant

In this post I'm going to talk about using port 80 and 443 for web development on a Mac running 1 or more virtual machines under Vagrant.

Why use port 80 and port 443?

Port 80 is the standard port for HTTP and 443 is the standard port for HTTPS. While other ports can be used, they're non-standard, and some (if not most) webapps make assumptions about those port numbers. You can really get burned on matters of HTTP redirection, such as when a form is submitted or when redirection from HTTP to HTTPS happens. While applications can certainly be built to take the port numbers into account, many are not.

Unfortunately, it's not a simple matter of telling configuring your Vagrant instance to listen on ports 80 and 443. Any port under 1024 requires the program to be running as root. And running an app as the root user is generall y never a good idea.

Can I forward ports from the Vagrant instance?

Why yes, you can! It's as simple as putting these lines in your Vagrantfile:

config.vm.forward_port 80, 8080
config.vm.forward_port 443, 8443

Wait, those are ports 8080 and 8443!

Yes, they are. Remember what I said about needing to be the root user? But, under Mac OS/X we can forward ports 80 and 443 to ports 8080 and 8433. The trick is to use the ipfw utility which comes with OS/X and is used to manage its built in firewall.

Here's a "quick and dirty" way to do it from the command line:

sudo ipfw add 100 fwd,8080 tcp from any to me 80
sudo ipfw add 101 fwd,8443 tcp from any to me 443
Average: 4.3 (20 votes)
Your rating: None

A Foggy Night in Ardmore

It was a bit foggy last night, and I couldn't sleep last night. So I did the only sensible thing. I wandered around town and took pictures.

The Apple Store in Suburban Square Suburban Square Parking Lot (not really full) Under the Anderson Ave Bridge Lancaster Ave at 3 AM Lancaster Ave at 3 AM Foggy Wawa

There's a few more such pictures. The full set can be found on Flickr:


Average: 3.6 (14 votes)
Your rating: None