Philadelphia Public Transit vs COVID-19

We’re well past a month into the COVID-19 shutdown, and I noticed that fewer and fewer trains were running on Regional Rail each day. I knew that SEPTA had decreased their service due to less riders, but I wondered just how strict the service cuts were. I also wondered if more or perhaps less trains were running on time.

Fortunately, I have several years worth of train data due to running SeptaStats.com, so I could answer these questions myself!

I started off by firing up Splunk Lab then went for a walk while it took 15 minutes or so to load the data up. I came back, and decided to see where we were:

“That’s one big Twinkie.”

That’s well north of 60 million data points. While I could crunch that data as-is, it would take longer for me to run my subsequent queries as well as look for trends in the data. I ended up writing a couple of scripts to summarize that data on a daily basis, so that I can get a bucketed breakdown of late trains for the entire day. We’ll get back to that bucketing later, because I first want to talk about train volumes.

I figured that with the perceived dropoff in service, I could look at how many distinct trains (identified by train number) each day. Sure enough, there was a drop off in train service levels:

“Symmetrical book stacking… just like the Philadelphia mass turbulence of 1947!”

Things started to get serious the week of Monday, March 16th. In fact, that was the last day of “normal” service on Regional Rail. Starting on Tuesday the 17th, the number of trains per weekday went from nearly 500 to about 362, with the weekends unaffected.

Continue reading “Philadelphia Public Transit vs COVID-19”

Examining 12 Months of SEPTA API Queries

With the release of SEPTA’s new app, I’ve suddenly been flooded with questions about their API. People wanted to know how stable it was.

Well, I don’t work for SEPTA, which means I don’t have insight into their operations, but I can perform some analytics based on what I have, which is approximately 18 months of Regional Rail train data, read every minute by SEPTA Stats.

Overall Stats

This is all of the data that I have in Septa Stats currently:

  • Events Since Inception: 26,924,887 events
  • First Event: Mar 1, 2016 12:00:01 AM
  • Last Event: Nov 16, 2017 10:33:53 PM

That’s way more events than minutes in that timeframe, and the reason for that is each API query is split into a separate event for each train. So if an API call returns status for 20 trains, that gets split into 20 different events. This is done because Splunk has a much easier time working with JSON that isn’t a giant array. 🙂

Continue reading “Examining 12 Months of SEPTA API Queries”

Introducing the SEPTA Regional Rail System Dashboard!

…and 2 new API endpionts, too. But more on those later.

I’m proud to say that there is now a dashboard for the entire Regional Rail system. It is present on both the front page and the “SEPTA System Stats” page:

This new dashboard makes it straightforward to determine the status of the entire Regional Rail system at a glance.

Continue reading “Introducing the SEPTA Regional Rail System Dashboard!”

Notes from February 2015 Philly DevOps Meetup: Security Practices for DevOps Teams

As a service to the Philly tech community (and because folks asked), I took notes at tonight’s presentation, called “Security Practices for DevOps Teams”. It was presented by Chris Merrick, VP of Engineering at RJMetrics.

Security is a “cursed role”

  • …in the sense that if you’re doing a really good job as a security engineer, no one knows you exist.
    • It isn’t sexy
    • It’s hard to quantify
    • It’s never done

As DevOps engineers, we are all de facto security engineers

Some tips to avoid ending up like this [Picture of a dismembered C3PO]

Sense. This picture makes none. 


  • Security Principles
    • Obscurity is not Security
      • “A secret endpoint on your website is not security”
      • “Don’t rely on randomness to secure things”
    • Least Privilege
      • Do not give more privileges than are needed
    • Weakest Link
      • If you talk to an insecure system, you’re at risk
    • Inevitability

Security Types

  • Physical
    • Stealing laptops
    • Breaking into datacenters
  • Application
    • Any vector that comes through an application you developed
      • XSS
  • Network*
  • Systems*
    • Applications you didn’t write
  • Human
    • Phishing, social engineering

Server Auth

  • Reminder:
    • Authentication is who you are
    • Authorization is what you can access
  • Don’t access production directory
    • Good news: this is our job anyways
  • Don’t spread private keys around
    • Don’t put in your Dropbox
    • Don’t let it leave the machine you generated it on
    • Use SSH agent forwarding
      • ssh-add
      • ssh -A you@remote
      • ssh-add -l
  • Don’t use shared accounts
    • Especially root
  • Be able to revoke access quickly
    • Time yourself. Go.
  • We use Amazon OpsWorks to help us achieve these goals
    • Chef+AWS, with some neat tricks: simple autoscaling, application deployment, and SSH user management

Logging

  • “Logs are your lifeline”
  • When you get into a high pressure security investigation, you start with your logs
  • Capture all authentication events, privilege, escalations, and state changes.
    • From your Os and all running applications
  • Make sure you can trust your logs
    • Remember – they’re your lifeline
  • Have a retention policy
    • We keep 30 days “hot”, 90 days “cold”
  • Logging – ELK
    • We use ELK for hot log searching
    • Kibana creates logs and lets you monitor your application in real time

Deployment

  • Keep unencrypted secrets out of code
    • Otherwise, a MongoLab exploit becomes your exploit
  • Don’t keep old code around
  • Make deployment and rollback easy
    • More good news: this is our job anyways
    • When dealing with a security issue, the last thing we need a “hard last step” in order to get the fix out
  • IAM
    • Don’t use your root account, ever.
      • Set a long password and lock it away
  • Set a strong password policy and require MFA
  • Don’t create API keys where API access isn’t needed
    • Same goes for a console password
  • Use Managed Policies
    • To make management easier
  • Use Roles to gran taccess to other systems
    • No need to deploy keys, auto-rotates
  • IAM Policy Pro Tips
    • Don’t use explicit DENY policies
      • Keep in mind that everything is denied by default
    • Don’t assume your custom policy is correct just because it saves – the interface only confirms the JSON is valid
    • Use the policy simulator
  • Know Thy Enemy
    • People are out there scanning for AWS keys – treat your private key like a private SSH key

Bonus Tips

  • Set up a security page
  • Sign up for the US-CERT email list, and the security notification list for your OS
  • Other resources
    • OWASP – owasp.org
    • SecLists.org
    • Common Weakness Enumeration – cwe.mitre.org

Conclusion

  • What else should we be doing to keep our work secure? [Picture of C3PO in a field full of flowers]

Notes from “Scaling on AWS for the First 10 Million Users”

Earlier tonight, I had the pleasure of attending a presentation from Chris Munns of Amazon at the offices of First Round Capital about scaling your software on AWS past the first 10 million users. I already had some experience with AWS, but I learned quite a few new things about how to leverage AWS, so I decided to write up my notes in a blog post for future reference, and as a service to other members of the Philadelphia tech community.

Without further preamble, here are my notes from the presentation:

Continue reading “Notes from “Scaling on AWS for the First 10 Million Users””

Running Dorsai Thing 2013: Lessons Learned

The Liberty Bell

Each year, The Dorsai Irregulars have their own annual convention known as “Dorsai Thing”. It is a weekend long event, held in a different city each year, where we all gather, socialize, and hold our semi-annual business meeting. This year, it was held in Phildelphia, and it fell on my shoulders to run the event, dubbed “Liberty Thing” While I’ve worked at plenty of conventions before, this was the first one where I was effectively the Con Chair. This was an entirely new thing for me, and I wanted to write a blog post about some of the things I learned during the experience.

Continue reading “Running Dorsai Thing 2013: Lessons Learned”