ProxSMTP is a flexible tool that allows you to reject, change or log email based on arbitrary critera. It accepts SMTP connections and forwards the SMTP commands and responses to another SMTP server. The 'DATA' email body is intercepted and filtered before forwarding.

You need to be able to write the filtering scripts that integrate it with your particular needs. If you're looking for something that does virus filtering, take a look at ClamSMTP which behaves similarly and uses a similar code base.

I wrote this with the Postfix mail server in mind. Here's how to configure it as a Postfix Content Filter.

ProxSMTP can also be used as a transparent proxy to filter an entire network's SMTP traffic at the router.