Doug's Home On The Web "Now monochrome monitor compatible!"

Screenshot Gallery (5 pictures)

I'm a huge fan of Drupal's Fivestar module. I use it on all of the sites I run. That said, I always wanted to see some reporting and stats features with the module. It would be nice to track voting activity so that I can spot instances of abuse, or see if there are any users who consistently receive low ratings. So I decided to write a separate module which did just that.

Here's what Fivestarstats does:

• Gets total vote stats, including the average rating
• Tells what IP addresses:
  • Are top voters
  • Are top 1-star voters (to catch abuse)
  • Have the lowest vote average. (generally mean people)
• Tells what users:
  • Have the highest average rating
  • Have the lowest average rating
  • Have had the most 1-star votes cast against them (to catch abuse)

So where can it be found? On GitHub, naturally:

https://github.com/dmuth/fivestarstats

Have fun!

So I was troubleshooting Anthrocon's website the other night, trying to figure out why pages were taking a few seconds to load, but used to be much quicker. While digging around, I saw that the filecache directory was over a Gigabyte in size. So I cleared the cache, and saw that it was still several megs in size, which struck me as odd.

Looking in the cache/ directory, I saw that the file that held the cached copy of the system variables was over 5 Megabytes in size. Definitely not normal. My next step was to check the variables table, and look what I found:

mysql> SELECT name, LENGTH(value) AS len FROM variable WHERE name LIKE 'captcha_placement%';
+-----------------------------+---------+
| name                        | len     |
+-----------------------------+---------+
| captcha_placement_map_cache | 5155086 |
+-----------------------------+---------+

It seems that for reasons I don't understand, the "captcha_placement_map_cache" variable from the CATPCHA module grows without bound. In our case, it was well over 5 Megs. This means that on every page load, a 5 Megabyte file was being read into memory and deseralized. While the filesystem cache kept the disk from getting hammered, the CPU still had to do the same work over and over.

The good news is that I could delete that variable with no ill effects, and the entire website suddenly became much more responsive. Here's a CPU graph:

I'm unclear as to what's happening in the captcha module. I think my long-term solution will be to modify my DDT module to just delete that variable once it exceeds 10 K. That will keep this from ever bothering us again.

Screenshot Gallery (5 pictures)

I'm pleased to announce my module called DDT, which stands for "Doug's Drupal Tools". (Clever, huh?)

I wrote this module to add some auditing and other functionality to the Drupal-powered sites that I run. Here's a list of its main features:

• A command-line script for backing up an entire Drupal installation. (bin/backup.php)
• Auditing of changed Drupal-wide variables between cron runs. (To determine if another site admin changed anything)
• Auditing of this module being enabled/disabled (to see if admins are trying to bypass it)
• Auditing of changes made to published/promoted/sticky/comments status on any node. (see what posts other admins are removing)
• Auditing of changes made to comments (see if other admins are removing comments)
• Auditing of changes made to any user's active/blocked status (see if other admins are blocking users)
• Integration with Drupal's Flag module. This module logs whenever a node or comment is flagged or unflagged.
• Integration with Drupal's Privatemsg module. This module logs whenever a private message is created or read.

But wait, there's more! Due to some incidents of spam and other network abuse, I added in some anti-abuse tools:

• See what IPs a user or a list of users came from
• See what users came from a given IP (helpful for spotting sockpuppets)
• Search chats from the Chat Room module to investigate and verify claims of abuse, harassment, and/or spam.

So where can you get a copy? Right now, it's on GitHub:

https://github.com/dmuth/Dougs-Drupal-Tools

Share and enjoy!

Screenshot from my DDT module.

Screenshot from my DDT module.

Screenshot from my DDT module.

Screenshot from my DDT module.

Screenshot from my DDT module.

It's that time of the year again, to clean out Anthrocon's ride share and room share forums.

I dislike deleting old content, because that just causes lots of bad links from Google (and possibly other sites). Not to mention that my logs fill up with 404s unnecessarily. Instead, this SQL will do the job nicely.

CREATE TABLE temp_nids (nid INT(10) UNSIGNED);

INSERT INTO temp_nids (
   SELECT DISTINCT nid FROM term_node WHERE nid IN (
      SELECT n.nid FROM node AS n
      LEFT JOIN term_node AS tn 
         ON n.nid=tn.nid
      LEFT JOIN term_data AS td 
         ON tn.tid=td.tid
      WHERE 
      (
         td.name LIKE 'Room Share%' 
         OR td.name LIKE 'Ride Share%'
      )
      AND FROM_UNIXTIME(n.created) < '2011-07%'
   )
)

DELETE FROM term_node WHERE nid IN (SELECT nid FROM temp_nids);

The innermost SELECT statement will return the posts made from those forums before last year's convention. It is then wrapped up in a SELECT DISTINCT just to make extra sure that we have valid NIDs. That's probably not strictly necessary, but there aren't any major performance issues with smaller databases, either. :-)

All of those nids are inserted into the temp_nids table, which is then used by the DELETE statement at the end to remove all terms from those nodes, effectively "detaching" those posts from the forums they were originally made in.

End result: the posts aren't removed from the site, just from the forums.

As folks noticed on Thursday morning when we opened hotels, the website displayed errors for 5-10 minutes starting at 9 AM EST due to the rush of traffic that came in.  This was due to a few factors, which I'd like to go into in technical detail below.

First, some raw stats:

Peak bandwidth: approx 3.6 Megabits/sec

Peak connections: 1,060 concurrent connections

Number of users logged into the site: 100+

Here are some graphs that show just how big those numbers are, compared to normal traffic levels:

For those who saw last year's post about the load on the webserver when we opened up hotels, this year's traffic was about twice what last year's traffic was.  I didn't see that coming.

As is plainly visible in the traffic graph, the machine that this website runs on is capabale of much higher bandwidth throughput.  So, what happened?

In a word: caching.  Or rather, the lack thereof in certain cases.