Security threats Toolkit

How spam may feed the thinking machine

Rupert Goodwins ZDNet.co.uk

Published: 24 Aug 2004 16:50 BST

By now, the whole business resembles a planetwide reverse Turing test. Instead of human arbiters deciding whether their interlocutor is man or machine, uncountable thousands of filtering robots anxiously scan gigabytes of chatter to fish out the spawn of their evil cousins. It turns out that the only way to be sure whether something is spam is to look at it like a human, with all our knowledge of context, language, meaning and intent. In short, you must be truly intelligent to do the job. Suddenly, the mildly moribund field of AI has a real job to do: saving the world.

Evidence of this can be found as far afield as the University of Melbourne, where programmers Matthew Sullivan and Guy Di Mattina, together with mathematics lecturer Dr Kevin Gates, have stapled a Support Vector Machine to an email firewall to get a claimed rate of 90 emails a second with one error every 25,000 messages. Support Vector Machines are fearsome mathematical constructs that have only just escaped from the lab. As far as I can make out, they seek non-linear hyperplanes in Hilbert space using Lagrangian transforms - check http://www.kernel-machines.org/ if you don't believe me.

Whatever the details, a SVM looks at data in lots of ways at once - it extends the variables in the data into many dimensions -- and then learns which characteristics mark out members of one set from another. The eponymous support vectors are the dividing lines between the two sets: once the machine has established these, filtering is a matter of finding out which side of the lines the messages fall. Performance is predictable and prone to optimisation: in short, this is one of the most powerful methods of handling real-world data within a computer that has yet been developed.

With spam estimated to be costing tens of billions of dollars worldwide each year, the motivation to develop really effective filtering is intense -- and that's before the fact that whoever defeats the Spam Monster will be crowned God-Emperor for Life and forever be preceded by dusky maidens and/or oiled hunks (delete according to taste) casting rose petals in their path. If the evil of spam leads to a renaissance of well-funded research into fundamental knowledge systems -- nothing else will do -- it could be the final kick we need to create truly intelligent machines. What they say when they find out they're being fed a diet of pure rubbish will be another matter: we'd better get our excuses ready, and fast.