The existing tools for testing spam filters evaluate a filter
instance by simply feeding it with a stream of emails, possibly
also providing a feedback to the filter about the correctness of
the detection. In such a scenario the evaluated filter is
disconnected from the network of email servers, filters, and
users, which makes the approach inappropriate for testing many of
the filters that exploit some of the information about spam
bulkiness, users' actions and social relations among the users.
Corresponding evaluation results might be wrong, because the
information that is normally used by the filter is missing,
incomplete or inappropriate. In this paper we present a tool for
testing spam filters in a very realistic scenario. Our tool
consists of a set of Python scripts for unix/linux environment.
The tool takes as inputs the filter to be tested and an affordable
set of interconnected machines (e.g., PlanetLab machines, or
locally created virtual machines). When started from a central
place, the tool uses the provided machines to build a network of
real email servers, installs instances of the filter, deploys and
runs simulated email users and spammers, and computes the
detection results statistic. Email servers are implemented using
Postfix, a standard linux email server. Only per-email-server
filters are currently supported, whereas per-email-client filters
testing would require additional tool development. The size of the
created emailing network is constrained only by the number of
available PlanetLab or virtual machines. The run time is much
shorter then the simulated system time, due to a time scaling
mechanism. Testing a new filter is as simple as installing one
copy of it in a real emailing network, which unifies the jobs of a
new filter development, testing and prototyping. As a usage
example, we test the SpamAssassin filter.