If you manage Linux servers over the Internet, you use SSH to connect to them. SSH lets you have a remote shell on a host over an encrypted channel so that an attacker cannot watch what you are doing over the network. In this blog post, I’m going to talk about using SSH at scale across thousands of posts.
Phase 0: Passwords
When you get started with SSH for the first time, you likely won’t have keys set up and will instead use passwords to authenticate to your servers. It will look something like this:
You use SSH to connect to the server, type in your password, and you’re good to go. That’s fine for small scale, such as managing a single server, but it doesn’t come without downsides. Specifically, you won’t be able to easily use a tool such as Ansible nor do code checkins with Git.
And that’s actually a bigger problem than it sounds, because if you make it harder to use a tool, that tool will be used far less often. This can lead to things such as configuration drift due to Ansible being run less often, or giant code pushes happening once a day if Git is being run less. And giant code pushes are a particular problem, because if other engineers have written code, you’ll have to do a merge, and if a bug presents itself, you’ll now have to think back to what you did 8 hours ago, not 8 minutes ago. Having to type in a password every single time will also slow down the rate of deployment, which in turn slows down the rate of product releases. Not good.
Seriously, don’t use SSH with a password for any reason other than as a stepping step to using keys. And that brings us to…
Perhaps you’re worried about being doxxed, perhaps you’ve received some specific threats, maybe you just want to increase your security. No matter the reason, this article is for you! Below I will list a collection of good practices to keep you and your accounts safe online. I fully expect to update this post as things change in the future.
I have tried to put things in a logical order, with some later steps depending on earlier steps, and some things that may be considered “controversial” towards the end.
This post was last updated on Oct 27, 2022.
Let’s start with passwords. I shouldn’t have to say this, but I will do so anyway: do not reuse passwords. Reusing passwords means that if a single account provider is breached and your plaintext password is recovered, you now have additional accounts at risk of compromise. This has happened before.
I recommend using a password manager such as 1Password to keep track of your passwords. While having your passwords stored in an app that uploads them somewhere increases your risk slightly, I feel it is outweighed by using a different password for each service. For passwords themselves, you can use random characters or a system such as Diceware to create long passwords that are easier to remember. While the latter is slightly less secure, a password that can be remembered is one less password to store into a password manager.
As a service to the Philly tech community (and because folks asked), I took notes at tonight’s presentation, called “Security Practices for DevOps Teams”. It was presented by Chris Merrick, VP of Engineering at RJMetrics.
Security is a “cursed role”
…in the sense that if you’re doing a really good job as a security engineer, no one knows you exist.
It isn’t sexy
It’s hard to quantify
It’s never done
As DevOps engineers, we are all de facto security engineers
Some tips to avoid ending up like this [Picture of a dismembered C3PO]
Obscurity is not Security
“A secret endpoint on your website is not security”
“Don’t rely on randomness to secure things”
Do not give more privileges than are needed
If you talk to an insecure system, you’re at risk
Breaking into datacenters
Any vector that comes through an application you developed
Applications you didn’t write
Phishing, social engineering
Authentication is who you are
Authorization is what you can access
Don’t access production directory
Good news: this is our job anyways
Don’t spread private keys around
Don’t put in your Dropbox
Don’t let it leave the machine you generated it on
Use SSH agent forwarding
ssh -A you@remote
Don’t use shared accounts
Be able to revoke access quickly
Time yourself. Go.
We use Amazon OpsWorks to help us achieve these goals
Chef+AWS, with some neat tricks: simple autoscaling, application deployment, and SSH user management
“Logs are your lifeline”
When you get into a high pressure security investigation, you start with your logs
Capture all authentication events, privilege, escalations, and state changes.
From your Os and all running applications
Make sure you can trust your logs
Remember – they’re your lifeline
Have a retention policy
We keep 30 days “hot”, 90 days “cold”
Logging – ELK
We use ELK for hot log searching
Kibana creates logs and lets you monitor your application in real time
Keep unencrypted secrets out of code
Otherwise, a MongoLab exploit becomes your exploit
Don’t keep old code around
Make deployment and rollback easy
More good news: this is our job anyways
When dealing with a security issue, the last thing we need a “hard last step” in order to get the fix out
Don’t use your root account, ever.
Set a long password and lock it away
Set a strong password policy and require MFA
Don’t create API keys where API access isn’t needed
Same goes for a console password
Use Managed Policies
To make management easier
Use Roles to gran taccess to other systems
No need to deploy keys, auto-rotates
IAM Policy Pro Tips
Don’t use explicit DENY policies
Keep in mind that everything is denied by default
Don’t assume your custom policy is correct just because it saves – the interface only confirms the JSON is valid
Use the policy simulator
Know Thy Enemy
People are out there scanning for AWS keys – treat your private key like a private SSH key