Technology
From the Trenches

Spam-proofing Leopard Server

Posted on Monday, April 13th, 2009 at 14:14 MST

This article will show you how to dramatically improve the effectiveness of the anti-spam system that is built into Mac OSX Server 10.5. After the upgrade, at least 99.9% of spam should be prevented from reaching your user’s e-mail Inboxes.

Spam

This process requires that the Developer Tools are installed on the system. As always, backup your server before making any changes.

There are six steps to a spam-free Mac OS X Server:

  1. Modify the Postfix configuration.
  2. Upgrade SpamAssassin to the latest version.
  3. Install and configure Vipul’s Razor.
  4. Test SpamAssassin and razor.
  5. Teach SpamAssassin by automatically growing a Bayesian database.
  6. Filter the tagged spam out of user’s Inboxes.

1. Enable Postfix Anti-Spam Measures

Leopard server’s SMTP server, Postfix, comes with numerous anti-spam features. Here are a few simple modifications you can make that will stop a lot of spam in its tracks, before it enters your server and consumes any resources.

First, make a backup copy of /etc/postfix/main.cf. Then open main.cf with your favorite text editor, and change or add the following lines. To ensure that you never block your own servers or hosts, change the ‘mynetworks’ line to include the IP addresses and/or networks that you trust, in a comma-separated list:

mynetworks = 127.0.0.0/8, 10.0.1.0/24, 24.68.38.115

Enable SMTP authentication:

smtpd_sasl_auth_enable = yes
smtpd_use_pw_server = yes

If you plan to use SMTP authentication, you should enable SSL (TLS) on Postfix, so that user credentials don’t go over the network in the clear. How to set up SSL with Postfix is covered in the Mail Service Administration manual starting on page 19.

This line in main.cf will allow authenticated and trusted users to send, and will reject mail from invalid sources, typically used by spammers:

smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_unknown_recipient_domain, reject_unknown_sender_domain, reject_invalid_hostname

The last addition exploits an amazing community resource, the ZEN Spamhaus real-time black hole list. It is basically a free, community-updated database of known spammer’s IP addresses. Before you add it, you may want to review the information on the Spamhaus website.

Add this to your main.cf:

smtpd_client_restrictions = permit_sasl_authenticated, reject_rbl_client zen.spamhaus.org

Restart postfix by issuing the command:

$ sudo postfix stop

Launchd will automatically re-launch postfix, and the new settings will take effect. You should watch the mail logs for a while, to ensure that everything is operating as expected:

$ tail -f /var/log/mail.log

Make a backup copy of your custom main.cf file, since future updates from Apple may over-write it.

2. Update SpamAssassin

OSX Server comes with SpamAssassin, the most popular open-source anti-spam software in use today. It is very effective if configured properly and kept up-to-date. As of the time of this writing, OSX Server 10.5.6 has SpamAssassin version 3.2.1, which was released on June 11, 2007. You should update SpamAssassin to the latest version, 3.2.5 at the time of this writing.

There are several ways to update SpamAssassin. Since it is a perl program, the best way to update is with the CPAN system, which can automatically update dependencies. To upgrade SpamAssassin using CPAN, run:

$ sudo -s
# export FTP_PASSIVE=1
# perl -MCPAN -e shell
cpan> install Mail::SPF
cpan> install IP::Country
cpan> install Net::Ident
cpan> install Mail::DomainKeys
cpan> install Mail::DKIM
cpan> install Encode::Detect
cpan> install Mail::SpamAssassin

If you have not previously configured CPAN, it will ask a series of questions to set it up. The configuration and use of CPAN is beyond the scope of this article, but I would highly recommend going through with it, since it makes the upkeep and installation of perl software much easier. Hint: Answer “follow” to the question, “Policy on building prerequisites”. Another hint: install wget before configuring CPAN, which will be used in case FTP doesn’t work for you.

Another way to upgrade SpamAssassin is to download the source code directly and compile it yourself. As you will see, this is a lot more work than setting up CPAN.

$ sudo -s
# mkdir /usr/local/src ; cd /usr/local/src
# curl -O http://east.unified.net/apache/spamassassin/source/Mail-SpamAssassin-3.2.5.tar.bz2
# tar xjvf Mail-SpamAssassin-3.2.5.tar.bz2
# cd cd Mail-SpamAssassin-3.2.5
# perl Makefile.PL

optional module missing: Mail::SPF
optional module missing: IP::Country
optional module missing: Razor2
optional module missing: Net::Ident
optional module missing: Mail::DomainKeys
optional module missing: Mail::DKIM
optional module missing: Encode::Detect

warning: some functionality may not be available,
please read the above report before continuing!

There are a lot of missing modules! For an effective setup, you will need to locate the source code for each of them, download, and install them. CPAN isn’t sounding too bad now, is it?

After you upgrade, confirm that spamassassin is the latest version:

$ spamassassin –version
SpamAssassin version 3.2.5
running on Perl version 5.8.8

Note that –version has two dashes. You can use -V as well.

3. Install Razor Agents

Vipul’s Razor is a distributed, collaborative, spam detection and filtering network.” It is also a perl program, and requires several perl modules. Fire up CPAN again, and install the following:

# export FTP_PASSIVE=1
# perl -MCPAN -e shell
cpan> install Time::HiRes
cpan> install Digest::SHA1
cpan> install MIME::Base64
cpan> install Getopt::Long
cpan> install URI::Escape
cpan> install File::Spec

It’s OK if CPAN tells you that some of these packages are already up-to-date. All of these modules are included in the razor-agents-sdk package, and are required for the main razor-agent program.

(There is a slight discrepancy in the documentation: this page says that the File::Copy module is required, and it also says that the SDK includes everything that is required. The discrepancy is that File::Copy is not in the SDK, despite its being in the list of modules that are supposedly required. The SDK includes the File::Spec module instead of File::Copy. I installed File::Spec (via CPAN) instead, because File::Copy requires a newer version of perl than comes with OSX Server, and I don’t want to upgrade perl. Razor appears to be working fine with File::Spec, so I’ll leave it at that.)

Once the modules are installed, download razor-agents, decompress, compile and install.

# mkdir /usr/local/src ; cd /usr/local/src
# wget http://prdownloads.sourceforge.net/razor/razor-agents-2.84.tar.bz2?download
# tar xjvf razor-agents-2.84.tar.bz2
# cd razor-agents-2.84
# perl Makefile.PL
# make
# make install

As per the razor installation instructions, you now need to create configuration files, and register an account with the razor system.

Mac OSX Server runs SpamAssassin (and thus Razor) as the user ‘_amavisd’, and uses the folder /var/amavis to hold anti-virus and anti-spam data. Since user _amavisd has no shell (/usr/bin/false) and has by default a different home directory (/var/virusmails), we should create the razor config files as a different user and then copy them to the /var/amavis folder:

$ razor-admin -create
$ razor-admin -register
$ sudo mv ~/.razor /var/amavis
$ sudo chown -R _amavisd:_amavisd /var/amavis

Open the /var/amavis/.razor/razor-agent.conf file with your text editor, and change defaults if you wish. I changed the logfile to /var/log/razor-agent.log.

4. Test SpamAssassin

Now that SpamAssassin has been upgraded, and Razor has been installed and configured, we need to test to make sure that it is working. To do this, you’ll need a copy of a spam email. I’ll leave that part up to you. I have some in an mbox file named ‘Junk’. To test, run spamassassin from the command line in debug mode. You can also learn a lot about your installation this way.

$ spamassassin -t -D < Junk
[1458] dbg: logger: adding facilities: all
[1458] dbg: logger: logging level is DBG
[1458] dbg: generic: SpamAssassin version 3.2.5

[1458] dbg: config: using “/etc/mail/spamassassin” for site rules pre files
[1458] dbg: config: read file /etc/mail/spamassassin/init.pre
[1458] dbg: config: read file /etc/mail/spamassassin/v310.pre
[1458] dbg: config: read file /etc/mail/spamassassin/v312.pre
[1458] dbg: config: read file /etc/mail/spamassassin/v320.pre
[1458] dbg: config: using “/usr/local/share/spamassassin” for sys rules pre files
[1458] dbg: config: using “/usr/local/share/spamassassin” for default rules dir
[1458] dbg: config: read file /usr/local/share/spamassassin/10_default_prefs.cf

[1458] dbg: config: using “/etc/mail/spamassassin” for site rules dir
[1458] dbg: config: read file /etc/mail/spamassassin/local.cf

[1458] dbg: plugin: loading Mail::SpamAssassin::Plugin::URIDNSBL from @INC
[1458] dbg: plugin: loading Mail::SpamAssassin::Plugin::Hashcash from @INC
[1458] dbg: plugin: loading Mail::SpamAssassin::Plugin::SPF from @INC
[1458] dbg: plugin: loading Mail::SpamAssassin::Plugin::Pyzor from @INC
[1458] dbg: pyzor: network tests on, attempting Pyzor
[1458] dbg: plugin: loading Mail::SpamAssassin::Plugin::Razor2 from @INC
[1458] dbg: razor2: razor2 is available, version 2.84

As you can see from the last line, above, Razor is running.

The debug output also lets you know that SpamAssassin is using the config files in /etc/mail/spamassassin, and the ones in /usr/local/share/spamassassin. Feel free to change the config files, but note that in Mac OSX Server, spamassassin is run from amavisd-new, which overrides some SpamAssassin settings. The settings that take precedence over SpamAssassin’s settings are in the file /etc/amavisd.conf:

$sa_spam_subject_tag = ‘***JUNK MAIL*** ‘;

# add spam info headers if at, or above that level
$sa_tag_level_deflt = -5.0;

# add ’spam detected’ headers at that level
$sa_tag2_level_deflt = 2.0;

# triggers spam evasive actions
$sa_kill_level_deflt = 6.0;

# spam level beyond which a DSN is not sent
$sa_dsn_cutoff_level = 7.0;

# spam level beyond which quarantine is off
$sa_quarantine_cutoff_level = 8.0;

Many options in /etc/amavisd.conf are set by the Server Admin GUI. Run perldoc Mail::SpamAssassin::Conf for documentation on SpamAssassin’s configuration variables. This page is also a good reference.

5. Teach SpamAssassin

To a large extent, SpamAssassin depends on a Bayesian database of spam and “ham” (legit email) for it’s effectiveness. A good way to accomplish this with the minimal amount of work from your users is to use IMAP, and have the server automatically learn from your user’s mail folders.

Mail.app and Thunderbird mail clients both have a “Junk” button that users can press if they get spam in their inbox. Configure the mail clients to move junk mail to an IMAP folder named “Junk”. Also, ask your users to put some examples of legitimate email into a folder named “NotJunk”, to help teach the system to recognize legitimate email.

Apple provided a script for teaching SpamAssassin: /etc/mail/spamassassin/learn_junk_mail. I modified it to seek out and learn from the IMAP folders “Junk” and “NotJunk” for each user on the sytem. Here it is.

Copy the script to your /etc/periodic/daily folder to have it run automatically every night at 3:15am. If you want to have it run more frequently, you should set up a launchd plist file for it.

If you are not using IMAP, Apple has provided a way that works with POP accounts. It is described in the Mail Service Administration manual on page 49.

6. Filter out the Junk

In Server Admin -> Mail -> Settings -> Filters, there is an option for “Attach subject tag”. I set mine to “***JUNK MAIL***”, so that every spam’s subject field is renamed with that prefix. This allows users to set up their own rules to handle junk mail the way that they see fit.

You can take this a step further, and have the tagged spam automatically delivered to an IMAP folder other than the Inbox. This would be demanded by users who check their mail on mobile devices such as the iPhone or Blackberry. In Mac OSX server, server-side mail filtering is accomplished using sieve scripts.

I covered setting up sieve in this article. Once sieve is up and running, sieve scripts will need to be installed for every user on the system. The script would be something like this:

require “fileinto”;

if header :contains “X-Spam-Flag” “YES” { fileinto “CaughtSpam”; }
if header :contains “Subject” “JUNK MAIL” { fileinto “CaughtSpam”; }

That will filter tagged spam into an IMAP folder named “CaughtSpam”, if the folder exists. You will have to tell your mail users to create the folder, and review it for false positives. Any false positives should be copied into a “NotJunk” folder.

I wish there were a way to implement this on a site-wide, instead of per-user basis. I briefly looked into using procmail, but the way that Mac OSX Server is set up, using amavisd-new, I don’t see how it’s possible. Apparently Apple played with the idea of using procmail before they decided to go with Amavisd. Amavisd passes mail to Cyrus, which uses it’s own lmtpd process to deliver mail to the IMAP store. If anyone has figured out a way of using both procmail and amavisd with cyrus, or has another way of filtering tagged spam into alternate IMAP mailboxes, I would love to hear from you.

14 Responses to “Spam-proofing Leopard Server”

  1. Spam-proofing Leopard Server • Netmojo Systems | Mac Affinity Says:

    [...] the rest here: Spam-proofing Leopard Server • Netmojo Systems Ads by GoogleReliable Backup – BRU – 30 day demo – googlepages.com – Call: [...]

  2. Svein Arild Says:

    Hey
    Just found this guide, and after reading it a couple of times, I tried it out on my own mail server.
    Everything seems to work fine, I got all the response from my server like the ones you write for.

    But yours custom learn_junk_mail script seems to be missing on the server, can you see after that or send it to me if possible?
    I can understand a script by reading it, but cant write my own yet, so looking at the original script from Apple dont help me mutch :(

    I hope the same moves can be done on a 10.6 server to fight spam too, maybe you get us an update then 10.6 is out?

    Again, thanks :)
    Best regards
    Svein
    Norway

  3. Brent Says:

    Thanks Svein! Fixed.

  4. Mike Friedman Says:

    Also, the command to tell the version of spamassassin is:

    spamassassin -v (not -version)

    Thanks for this tutorial. It was VERY helpful.

  5. Chris Wheeler Says:

    Thanks for the informative article. Most seems to be working but the learn_junk_mail script doesn’t seem to work as-is. If I remove the “su -l _amavisd” from the various command lines it works fine and we get an increase in the nspam number in our database but if I put them back in the script does nothing. The result is the same if I “su -l ANYUSER” for any command at the terminal – no output. Any suggestions?

  6. Mike Friedman Says:

    Whoops. Typo…should be spamassassin -V

  7. Brent Says:

    Mike: ’spamassassin –version’ (with two dashes) works as well. -V is shorter though, thanks. :)

    Chris: thanks for pointing that out. I had actually modified the script and didn’t update it here on the blog. I just copied over the learnspam script that I’m actually using (see the "Here it is" link), and the ’su -l _amavisd’ is not there. ‘/usr/sbin/chown -R _amavisd $SADB’ is added to ensure proper permissions. Here’s the diff [new script] [old script]:

    19c19
    < LEARN_CMD="$SA_LEARN -u _amavisd --dbpath $SADB --no-sync"
    ---
    > LEARN_CMD=”$SA_LEARN -u _amavisd –showdots –dbpath $SADB –no-sync”
    51c51
    < #echo "Learning spam from $user"
    ---
    > echo “Learning spam from $user”
    55,56c55
    < cat $file | $LEARN_CMD --spam >> /var/log/amavis.log 2>&1
    < /usr/sbin/chown -R _amavisd $SADB
    ---
    > cat $file | su -l _amavisd $LEARN_CMD –spam
    64c63
    < #echo "Learning ham from $user"
    ---
    > echo “Learning ham from $user”
    68,69c67
    < cat $file | $LEARN_CMD --ham >> /var/log/amavis.log 2>&1
    < /usr/sbin/chown -R _amavisd $SADB
    ---
    > cat $file | su -l _amavisd $LEARN_CMD –ham
    77,78c75
    < $SA_LEARN -u _amavisd --dbpath $SADB --sync
    < /usr/sbin/chown -R _amavisd $SADB
    ---
    > su -l _amavisd $SA_LEARN -u _amavisd –dbpath $SADB –sync
  8. Svein Arild Says:

    Hi again :)

    I yust found out that SORBS having problems and maybe wil close anyday soon.
    I changed the line “reject_rbl_client dnsbl.sorbs.net” with “reject_rbl_client zen.spamhaus.org” and having wery good resoults so far :)

    Regards
    Svein Arild
    Norway

  9. Chris Wheeler Says:

    Thanks for the confirmation on the script. I’ve noticed that messages are being marked as Spam but not as I’ve set in Server Admin. In Server Admin it says messages will be marked “***THIS MESSAGE MAY BE SPAM***” (no quotes) but messages are being marked {SPAM?} which I believe is the default for Spam Assassin. Any ideas?

  10. Rich Conaway Says:

    Anyone know of any Mac specific tutorials for getting CPAN set up correctly? I just get a ton of errors when trying to update Spam Assassin.

  11. Brent Says:

    @Chris: that’s odd. What’s in /etc/amavisd.conf? /etc/mail/spamassassin/local.cf?

  12. Brent Says:

    @Rich: What are the errors you’re getting, and what do you do to produce them?

    CPAN isn’t a Mac specific system…

  13. Brent Says:

    @Svein Arild: Thanks! It’s too bad SORBS has closed. I updated my servers & the article.

  14. gctwnl Says:

    After getting as far as the install Mail::SpamAssassin, it turns out spamasassin is still the old version and the new one is installed in /usr/local/bin

    bash-3.2# spamassassin –version
    spamassassin: spamassassin script is v3.002001, but using modules v3.003001
    bash-3.2# /usr/local/bin/spamassassin –version
    SpamAssassin version 3.3.1
    running on Perl version 5.8.8

    (I have had this experience with CPAN every time I tried it, always something goes awry if I try to use it)

    This is the end of the install:

    /usr/bin/perl build/preprocessor -Mvars -DVERSION=”3.003001″ -DPREFIX=”/usr/local” -DDEF_RULES_DIR=”/usr/local/share/spamassassin” -DLOCAL_RULES_DIR=”/etc/mail/spamassassin” -DLOCAL_STATE_DIR=”/var/lib/spamassassin” -DINSTALLSITELIB=”/Library/Perl/5.8.8″ -DCONTACT_ADDRESS=”postmaster@rna.nl” -m644 -Irules -O/usr/local/share/spamassassin user_prefs.template languages sa-update-pubkey.txt
    chmod 755 /usr/local/share/spamassassin
    JMASON/Mail-SpamAssassin-3.3.1.tar.gz
    /usr/bin/make install — OK

    And this is somehwre halfway:

    Installing /Library/Perl/5.8.8/Mail/SpamAssassin/Util/TieOneStringHash.pm
    Installing /usr/local/share/man/man1/sa-compile.1

    Argh! Have I killed my system?

Leave a Reply

Live Comment Preview: