Welcome to dmuth.org!

Welcome to my website and blog! It has been been around in one form or another for well over a decade. It’s changed purposes a few times, and at this point is now a mostly tech blog where I share things I’ve learned or made. Feel free to have a look around, or consider checking out some of the more popular posts I’ve made over the years:

“Who disturbs my nap?”

Thanks for visiting, and feel free to reach out if you have any comments or just want to chat!

HOWTO: Safely use “git rebase -i”

Git is a very powerful revision control system used in software development, and at this point, is effectively the industry standard. One of the things that makes Git so powerful is that there are all sorts of low-level operations that developers can do in it. This includes everything from tagging releases, to having an insane number of branches, to painless merging of said branches, to rewriting history.

“I am serious. And don’t call my Shirley.”

If you’re new to Git, your brain probably threw an exception when reaching the end of the paragraph as you thought to yourself, “Wait WHAT? Why would you want to REWRITE HISTORY?” Good question! Normally, you don’t want to mess history in a revision control system. But sometimes you may want to or even need to. A few possible reasons for rewriting history include:

  • Feature Branch “B” is a branch of Feature Branch “A” and you want to merge B’s changes to master, but not A’s.
  • Squashing all the commits in a feature branch to a single commit, after having previously pushed those commits.
  • A developer made a commit with something that doesn’t belong in Git, such as PII or credentials.
  • You want to “clean up” the commit history a little.

Whether any of the above are fully valid reasons or not is something up to you and your dev team.

That said, rebasing is not something you want to experiment with for the first time on your production repo. It’s just a bad idea. So I built a playground/lab where anyone can experiment with git rebasing in an isolated and local environment, which can be stood up in seconds. Here’s how to get started:

git clone git@github.com:dmuth/git-rebase-i-playground.git
cd git-rebase-i-playground
./init.sh
Continue reading “HOWTO: Safely use “git rebase -i””

Great Palace Map from Zelda II: New Adventure of Link

I’ve been trying to keep myself amused during the COVID-19 shutdown, and as such have been trying a number of new things. One new thing I decided to look into is playing hacked NES ROMs. It turns out there are a great number of ROM hacks out there, with the website ROMhacking.net listing over 4,600 hacked ROMs as of this writing.

While looking into hacked ROMs, I quickly learned that ROM hacks fall into one of 3 main categories: “really good”, “really bad”, and “joke”. I spent some time looking around, and found one particular ROM which I found in the “really good” category: Zelda II: New Adventure of Link, created by someone named HollowShadow. So I downloaded OpenEmu, grabbed the ROM hack, bought a USB NES controller, and got to playing!

Being a big fan of Zelda 2, I had no problem playing through the vast majority of the ROM. It’s when I got to the final palace is when the game became quite difficult, and quickly got lost.

What I needed was a map, but because this was a ROM hack, there wasn’t a whole lot available to me in the way of resources.

So I decided to make a map.

Continue reading “Great Palace Map from Zelda II: New Adventure of Link”

Philadelphia Public Transit vs COVID-19

We’re well past a month into the COVID-19 shutdown, and I noticed that fewer and fewer trains were running on Regional Rail each day. I knew that SEPTA had decreased their service due to less riders, but I wondered just how strict the service cuts were. I also wondered if more or perhaps less trains were running on time.

Fortunately, I have several years worth of train data due to running SeptaStats.com, so I could answer these questions myself!

I started off by firing up Splunk Lab then went for a walk while it took 15 minutes or so to load the data up. I came back, and decided to see where we were:

“That’s one big Twinkie.”

That’s well north of 60 million data points. While I could crunch that data as-is, it would take longer for me to run my subsequent queries as well as look for trends in the data. I ended up writing a couple of scripts to summarize that data on a daily basis, so that I can get a bucketed breakdown of late trains for the entire day. We’ll get back to that bucketing later, because I first want to talk about train volumes.

I figured that with the perceived dropoff in service, I could look at how many distinct trains (identified by train number) each day. Sure enough, there was a drop off in train service levels:

“Symmetrical book stacking… just like the Philadelphia mass turbulence of 1947!”

Things started to get serious the week of Monday, March 16th. In fact, that was the last day of “normal” service on Regional Rail. Starting on Tuesday the 17th, the number of trains per weekday went from nearly 500 to about 362, with the weekends unaffected.

Continue reading “Philadelphia Public Transit vs COVID-19”

How to Watch Netflix with Friends during COVID-19

With this whole COVID-19 pandemic going on, we all have to socially isolate ourselves in our homes in order to flatten the curve of cases, and this is likely to continue for some time.

I’ve looked into ways to watch Netflix with friends online, and after trying several different “solutions”, I found one which works quite well: Netflix Party

In order to use Netflix Party, each person will need a desktop or laptop running the Google Chrome web browser. If you don’t have Chrome installed, ask your child to install it on your computer for you. If you don’t have any children, text your neighbor’s kid for installation instructions.

To install Netflix Party click “Install Netflix Party” from the Netflix Party website. This will take you to the plugin page on the Chrome Web Store. Click the “Add to Chrome” button to install it.

“I am serious. And don’t call me Shirley.”

You’ll be prompted to install the extension in Chrome. Do that.

“That rug really tied the room together, did it not?”
Continue reading “How to Watch Netflix with Friends during COVID-19”

SSH At Scale: CAs and Principals

If you manage Linux servers over the Internet, you use SSH to connect to them. SSH lets you have a remote shell on a host over an encrypted channel so that an attacker cannot watch what you are doing over the network. In this blog post, I’m going to talk about using SSH at scale across thousands of posts.

Phase 0: Passwords

When you get started with SSH for the first time, you likely won’t have keys set up and will instead use passwords to authenticate to your servers. It will look something like this:

ssh dmuth@cheetah.dmuth.org
dmuth@cheetah.dmuth.org's passsword: ********
$ 

You use SSH to connect to the server, type in your password, and you’re good to go. That’s fine for small scale, such as managing a single server, but it doesn’t come without downsides. Specifically, you won’t be able to easily use a tool such as Ansible nor do code checkins with Git.

And that’s actually a bigger problem than it sounds, because if you make it harder to use a tool, that tool will be used far less often. This can lead to things such as configuration drift due to Ansible being run less often, or giant code pushes happening once a day if Git is being run less. And giant code pushes are a particular problem, because if other engineers have written code, you’ll have to do a merge, and if a bug presents itself, you’ll now have to think back to what you did 8 hours ago, not 8 minutes ago. Having to type in a password every single time will also slow down the rate of deployment, which in turn slows down the rate of product releases. Not good.

Seriously, don’t use SSH with a password for any reason other than as a stepping step to using keys. And that brings us to…

Continue reading “SSH At Scale: CAs and Principals”

Staying Safe Online: A Guide for Beginners and Non-Technical People

Perhaps you’re worried about being doxxed, perhaps you’ve received some specific threats, maybe you just want to increase your security. No matter the reason, this article is for you! Below I will list a collection of good practices to keep you and your accounts safe online. I fully expect to update this post as things change in the future.

I have tried to put things in a logical order, with some later steps depending on earlier steps, and some things that may be considered “controversial” towards the end.

This post was last updated on Jan 2, 2020.


Passwords

Let’s start with passwords. I shouldn’t have to say this, but I will do so anyway: do not reuse passwords. Reusing passwords mean that if a single account provider is breached and your plaintext password is recovered, you now have additional accounts at risk of compromise. This has happened before.

Anyone in a hoodie is NOT to be trusted.

I recommend using a password manager such as LastPass to keep track of your passwords. While having your passwords stored in an app that uploads them somewhere increases your risk slightly, I feel it is outweighed by using a different password for each service. For passwords themselves, you can use random characters or a system such as Diceware to create long passwords that are easier to remember. While the latter is slightly less secure, a password that can be remembered is one less password to store into a password manager.

Continue reading “Staying Safe Online: A Guide for Beginners and Non-Technical People”

Removing Dependent Child Images of a Docker Container

As much as I love using Docker, one of the frustrations I have is when I try to remove an an image which other images are based on, only to get this error:

$ docker rmi b171179240df
Error response from daemon: conflict: unable to delete b171179240df (cannot be forced) - image has dependent child images
Docker Logo

I did some searches on Google, and most of the advice centered around the heavy-handed approach of removing all Docker images and basically starting over with a clean slate. That approach didn’t sit well with me because it doesn’t strike me as all that efficient, and also causes me to have to spend more time waiting for unrelated containers to rebuild.

That prompted me to write a script which, when provided with the ID of a container to remove, will recurse through all child containers and delete them first.

Quick and Dirty Usage

bash <(curl -s https://raw.githubusercontent.com/dmuth/docker-remove-dependent-child-images/master/docker-remove-image) IMAGE_ID

That will delete the image IMAGE_ID, and all child images.

No muss, no fuss, no awkward explanations to senior engineers.

When run successfully, the output will look something like this:

The source and testing scripts can be found over on GitHub at https://github.com/dmuth/docker-remove-dependent-child-images.

Does this script save you some time? Let me know in the comments!

Doing Rollups of AWS S3 Server Access Logs

If you are storing files in Amazon S3, you absolutely positively should enable AWS S3 Access Logging. This will cause every single access in a bucket to be written to a logfile in another S3 bucket, and is super useful for tracking down bucket usage, especially if you have any publicly hosted content in your buckets.

But there’s a problem–AWS goes absolutely bonkers when it comes to writing logs in S3. Multiple files will be written per minute, each with as few as one event in them. It comes out looking like this:

2019-09-14 13:26:38        835 s3/www.pa-furry.org/2019-09-14-17-26-37-5B75705EA0D67AF7
2019-09-14 13:26:46        333 s3/www.pa-furry.org/2019-09-14-17-26-45-C8553CA61B663D7A
2019-09-14 13:26:55        333 s3/www.pa-furry.org/2019-09-14-17-26-54-F613777CE621F257
2019-09-14 13:26:56        333 s3/www.pa-furry.org/2019-09-14-17-26-55-99D355F57F3FABA9

At that rate, you will easily wind up with 10s of thousands of logfiles per day. Yikes.

Dealing With So Many Logfiles

Wouldn’t it be nice if there was a way to perform rollup on those files so they could be condensed into fewer bigger files?

Well, I wrote an app for that. Here’s how to get started: first step is that you’re going to need to clone that repo and install Serverless:

git clone git@github.com:dmuth/aws-s3-server-access-logging-rollup.git
npm install -g serverless

Serverless is an app which lets you deploy applications on AWS and other cloud providers without actually spinning up virtual servers. In our case, we’ll use Serverless to create a Lambda function which executes periodically and performs rollup of logfiles.

So once you have the code, here’s how to deploy it:

cp serverless.yml.exmaple serverless.xml
vim serverless.xml # Vim is Best Editor
serverless deploy # Deploy the app. This will take some time.
Continue reading “Doing Rollups of AWS S3 Server Access Logs”

Saving and Restoring Your development/ Directory

If you’re like me, you write a fair bit of a code, which means you have to interact with many Git repositories. If you’re also like me, chances are you have them in a directory called development/ or similar. It might even have some nested directories, something like this:

./allaboutcheetahs.info
./diceware
./docker/check-disk-space
./docker/health-check
./node/circuitbreaker-demo
./node/neural-network
./s3/bucket-sizes
./s3/disk-usage
./snowdrift
./ssh-to
Code
Code you may write someday.

So that’s cool, but let’s say that you get a new machine and you want replicate your development/ directory structure onto it? One way is to check out everything by hand, but that’s laborious and time consuming. A second way is to keep backups–and you should absolutely do this–but aside from challenges of restoring a single directory out of an entire archive, what if that backup doesn’t have the latest commits in it?

I can now offer a third way. I recently wrote a couple of scripts available on GitHub that can be used to extract Git remote from each repo in an entire directory stucture, and save those remotes and the directories they belong in to a file. Given the above example, it might look something like this:

./allaboutcheetahs.info git@github.com:dmuth/dmuth.github.io.git
./diceware      git@github.com:dmuth/diceware.git
./docker/check-disk-space       git@github.com:dmuth/docker-check-disk-usage.git
./docker/health-check   git@github.com:dmuth/docker-health-check.git
./node/circuitbreaker-demo      git@github.com:dmuth/another-circuit-breaker.git
./node/neural-network   git@github.com:dmuth/neural-network.git
./s3/bucket-sizes       git@gitlab.com:dmuth/s3-bucket-sizes.git
./s3/disk-usage git@github.com:dmuth/s3-disk-usage.git
./snowdrift     git@github.com:Comcast/snowdrift.git
./ssh-to        git@github.com:Comcast/ssh-to.git
Continue reading “Saving and Restoring Your development/ Directory”