Doug's Home On The Web

As a service to the Philly tech community (and because folks asked), I took notes at tonight’s presentation, called “Security Practices for DevOps Teams”. It was presented by Chris Merrick, VP of Engineering at RJMetrics.

Security is a “cursed role”

• …in the sense that if you’re doing a really good job as a security engineer, no one knows you exist.
  • It isn’t sexy
  • It’s hard to quantify
  • It’s never done

As DevOps engineers, we are all de facto security engineers

Some tips to avoid ending up like this [Picture of a dismembered C3PO]

• Security Principles
  • Obscurity is not Security
    • “A secret endpoint on your website is not security”
    • “Don’t rely on randomness to secure things”
  • Least Privilege
    • Do not give more privileges than are needed
  • Weakest Link
    • If you talk to an insecure system, you’re at risk
  • Inevitability

Security Types

• Physical
  • Stealing laptops
  • Breaking into datacenters
• Application
  • Any vector that comes through an application you developed
    • XSS
• Network*
• Systems*
  • Applications you didn’t write
• Human
  • Phishing, social engineering

Server Auth

• Reminder:
  • Authentication is who you are
  • Authorization is what you can access
• Don’t access production directory
  • Good news: this is our job anyways
• Don’t spread private keys around
  • Don’t put in your Dropbox
  • Don’t let it leave the machine you generated it on
  • Use SSH agent forwarding
    • ssh-add
    • ssh -A you@remote
    • ssh-add -l
• Don’t use shared accounts
  • Especially root
• Be able to revoke access quickly
  • Time yourself. Go.
• We use Amazon OpsWorks to help us achieve these goals
  • Chef+AWS, with some neat tricks: simple autoscaling, application deployment, and SSH user management

Logging

• “Logs are your lifeline”
• When you get into a high pressure security investigation, you start with your logs
• Capture all authentication events, privilege, escalations, and state changes.
  • From your Os and all running applications
• Make sure you can trust your logs
  • Remember – they’re your lifeline
• Have a retention policy
  • We keep 30 days “hot”, 90 days “cold”
• Logging – ELK
  • We use ELK for hot log searching
  • Kibana creates logs and lets you monitor your application in real time

Deployment

• Keep unencrypted secrets out of code
  • Otherwise, a MongoLab exploit becomes your exploit
• Don’t keep old code around
• Make deployment and rollback easy
  • More good news: this is our job anyways
  • When dealing with a security issue, the last thing we need a “hard last step” in order to get the fix out
• IAM
  • Don’t use your root account, ever.
    • Set a long password and lock it away
• Set a strong password policy and require MFA
• Don’t create API keys where API access isn’t needed
  • Same goes for a console password
• Use Managed Policies
  • To make management easier
• Use Roles to gran taccess to other systems
  • No need to deploy keys, auto-rotates
• IAM Policy Pro Tips
  • Don’t use explicit DENY policies
    • Keep in mind that everything is denied by default
  • Don’t assume your custom policy is correct just because it saves – the interface only confirms the JSON is valid
  • Use the policy simulator
• Know Thy Enemy
  • People are out there scanning for AWS keys – treat your private key like a private SSH key

Bonus Tips

• Set up a security page
  • https://rjmetrics.com/security
• Sign up for the US-CERT email list, and the security notification list for your OS
• Other resources
  • OWASP – owasp.org
  • SecLists.org
  • Common Weakness Enumeration – cwe.mitre.org

Conclusion

• What else should we be doing to keep our work secure? [Picture of C3PO in a field full of flowers]