Data Circle

I get a ridiculous amount of spam (500+ emails/day), most of which is caught by Spamassasin. However, this takes its toll on my poor virtual machine, which has most of its 250MB of RAM eaten up by MySQL as it is. As a result, I’ve started to be more aggressive with the rules I apply to incoming mail, which include:

1. Reject anything from an invalid hostname.
2. Reject emails where the sender address is a non-existant domain.
3. Reject where the recipient address is a non-existant domain.
4. Reject where the domain is hosted on this server but the email address is invalid (e.g. iveabigone@roguestudents.com).
5. Check against Spamhaus RBL.
6. Greylist.

Naturally, open relaying is disabled too. I’ve also updated the DNS for all of my domains so that they only have one MX record, as secondary servers are prone to abuse and can’t usually check whether an individual address is deliverable until they pass the message on to the primary server. Greylisting has proved to be particularly effective—personally I’m not a big fan of this solution as it introduces unnecessary delays, but the trade off between mail arriving a bit later and the amount of time I have to spend dealing with spam means that enabling it is now a no-brainer.

I could increase the level of checks by rejecting mail from servers without a reverse DNS record or which don’t identify themselves correctly in HELO/EHLO, but unfortunately that impacts too much on legitimate mail from people who can’t configure servers properly.

Since implementing these checks, my spam has fallen from 500 messages per day to about 10 per day, a reduction of 98%, even though my mail server is still receiving the same number of incoming connections. Furthermore, I’m no longer seeing “out of memory” error messages on my virtual machines, which is a great relief as this usually meant downtime until I noticed the problem, followed by a reboot.

Yes, I know I could just set my MX records to Google and let Gmail do all the work for me, but I don’t like the idea of a company in the US who I have absolutely no control over holding all of my emails and performing data mining on them.

MySociety has an interesting post about growing pains of WhatDoTheyKnow, a popular site for tracking Freedom of Information requests. It’s well worth a read if you’re involved with building any applications with Ruby on Rails which need to scale to thousands of visitors per day and you don’t want your servers to turn into a pile of molten metal. I did find this quote particularly depressing though:

“Ruby’s memory manager never returns memory to the operating system”

That has certainly put me off using Ruby on Rails for the time being…

The stable version of Debian, codename Lenny, has been updated today, with security fixes for a number of core packages including base-files and perl. To upgrade on most systems, you’ll simply need to run the following commands as root:

aptitude update
aptitude safe-upgrade

For those who haven’t used aptitude before, it’s the slightly improved version of apt-get, and the recommended tool to use by the Debian Project.

If you’re running a version of WordPress prior to 2.8.4, or have only upgraded in the last few days, you should stop what you are doing and head over to this post on the WordPress development blog: How to Keep WordPress Secure. It discusses a worm which is currently doing the rounds, attempting to exploit older versions of WordPress by creating a “hidden” admin user. Incidentally, all the methods suggested to find the hidden user seem unnecessarily complicated – if you have a tool which allows access to your database (e.g. phpMyAdmin), just check the wp_users table manually.

Subscribing to the development blog is a must if you use WordPress, unless you are a WordPress.com user, where everything is taken care of for you.

Mapumental is a cool new service brought to you by the brainboxes at mySociety with some help from 4ip. Basically it allows you to see exactly where in the country you can afford to live (based on “average” house prices) and still commute to a given place of work.

Unfortunately you need an invite to get access to the interactive site, so you’ll have to make do with the teaser video for now. However, I can confirm that the system is one of the coolest and potentially most useful sites I’ve seen in ages. Hopefully there will be some improvements to come, some of the things I’d like to see include:

1. Ability to set a start time which isn’t 9:00am, for those of us who have to be in work earlier, work shifts or are students with lectures at varying times each day.
2. Rent data as well as house prices, as the latter doesn’t help much if you can’t afford to buy or the banks won’t give you the credit you need. Not entirely sure how this would work as house prices are kept at the Land Registry but there is no equivalent for rents.

Does such a thing exist, or am I still dreaming? Firefox is currently eating up 15% of my CPU and 20% of my RAM according to the top command, and I only have ten tabs open. I have just tried Dillo, but it doesn’t seem to support floating objects in CSS, rendering almost every 2 column stylesheet layout—such as the one I use on most of my hand-crafted sites—useless.

Perhaps once I have finished my thesis I will dedicate some time to hunting down memory leaks/usage in Firefox, or just write my own standards-enforcing browser from scratch…

Every day there are a number of tech stories which I come across that are interesting but I don’t have time to write full commentaries on them all. To get around this, I’ve created a Twitter account where I will ‘tweet’ (i.e. post to the updates page) links to tech stories, with a short headline for each. You can follow the updates on Twitter directly, subscribe to the RSS feed for datacircle or take a look at the sidebar on this site, which displays the last four tweets.

Have you ever typed twitter.co.uk into your browser instead of twitter.com? If so, you may have been surprised to find yourself on a completely different site, rather than being transparently redirected to the one you thought you were going to. The reason for this is that the domain is not owned by the Twitter company, but an individual named Steve Crawford, who is currently inundated by emails from the site as people enter something@twitter.co.uk as their email address and then start being ‘followed’ by other users—causing Twitter to send a new email each time.

This is partly Twitter’s fault, as they do not make any attempt to verify the email address you supply when registering. Not only does this hammer the wrong user/mail server with emails if an incorrect address is provided, but it also means that the real account owner cannot reset their password either. On the other hand, I have slightly less sympathy for Steve as he must have switched on a catchall address if he is actually receiving all these emails, which is asking for trouble under any circumstances from spammers who email all common names and words at every domain they can find. Of course, turning off catchall would still hammer his mail server and be rather annoying, but at least the emails would be rejected.

One does wonder why the people behind Twitter didn’t register the name with other extensions—the UK in particular is a big online market and .co.uk is probably one of the most common country domains, possibly because it doesn’t impose any residential restrictions on registrants. However, they haven’t figured out how to make money from the site yet, and openly admit to holding off implementing such features, so I don’t expect them to be clued-up on the topic of registering domain names under different suffixes.

Further information

• Twitter.co.uk – the tale of a forgotten domain

I have been using the latest version of Ubuntu for a couple of weeks now, and so far it is proving to be a fairly robust and reliable system—certainly good enough for me to do my job, write development code and work on my thesis on a daily basis.

The only problems I have encountered so far are:

• Corrupt tracker index problem – I can’t believe this made it into the final release as it seems to have affected so many people it should have picked up in testing. There is a proposed solution and a workaround, but unfortunately everyone seems to be using the latter (can’t blame them – having a working machine is generally more important than testing bugs for most people). If I get a bit of time over the weekend I might install a separate instance under VMWare and see if I can do some testing.
• Freeciv doesn’t have any sound, unless you install the freeciv-sound package—and even then you’ll be lucky to get music. I don’t think this is a Jaunty-specific problem though.
• Although the function buttons on my laptop still work, I no longer get any visual feedback when using them—before upgrading when I altered the volume a bar would appear on screen showing me the current setting. This isn’t a major problem as the functionality still works, but having no visual feedback is a bit annoying.

Other than those issues though—some of which I hope to look into and file bugs for—everything seems to be running fine. Now if only I could get Alien Crossfire to play under Wine…

There is currently a petition on the Number 10 website asking the Prime Minister to save Bletchley Park. For anyone who hasn’t heard of Bletchley, the Wikipedia article on the subject has some good starting information—basically this was the place where some of the defining events of the Second World War took place, including the cracking of the ciphers generated by the Engima machine. I’ve yet to make it to Bletchley Park myself, so I would be extremely disappointed to see it go due to a lack of funding.

If you want to take more direct action, the Bletchley Park Trust welcomes donations and has a Friends of Bletchley Park scheme which you can join.

N.B. You can only sign the petition if you are British citizen or resident, though this includes expats and people living in British Overseas Territories or Crown Dependencies.