MUSC SPAM Report

Last Updated: March 4th, 2005

Current Spam Problem

In today's email environment, Spam is increasingly becoming more and more of a problem, both to users and administrators.  Looking at MUSC statistics relating to Spam a dramatic increase can be illustrated just in the past ten months.  Averaging over the past month, during business days, the following is indicative of the current situation:

From the above statistics we can derive that 71% of email that is sent to MUSC during a business day is spam.  On the weekend, the percentage grows to 95%.  Comparing this to last year, the result is;

or 62%, making around a 22,000 message increase in 10 months.  The current statistical analysis is comparable with the global epidemic Spam has become, and there is no indication that the current growth will not continue.

Without any user intervention, MUSC currently blocks or deletes an average of 62,463 messages daily, or, roughly, 67% of the total spam count.  The threshold has been fine tuned over time to provide the least amount of false positives or legitimate mail that appears to be spam.  While this setting helps, it is far from satisfactory from a user standpoint.

User Options for Reducing Spam

For a little background, included are some general safeguards for minimizing the amount of spam "lists" on which an email address is included, as well as minimizing the amount Spam you have to sift.

IMAP Users

If you use the MUSC IMAP server (cyrus), via Econtrol you can greatly increase the server identification of spam messages and filter those messages automatically.  To filter messages the server identifies as spam, perform the following steps in order:

1. Create a sub folder to hold the spam messages to be filtered on your IMAP account.  The way to make a new email folder on your IMAP account depends on which client you are using.  This folder name is Case Sensitive.  Remember this name.  It is not recommended that filtered spam be deleted as real mail may be lost.  In this example, junk will be used.
2. Go to Econtrol and select the Procmail button, then Edit.
3. If you have never used Procmail before, put the below text in the Edit Procmail box, check Enable Rule(s) and click the Save button.

   USERID=$LOGNAME
   LOGFILE=/dev/null
   
   :0
   * ^X-MUSC-MailScanner-SpamScore.*ss
   | procmail-deliver $USERID.junk
   
   :0
   | procmail-deliver $USERID
   

   Make sure you replace junk with the exact name of the folder you created in the first step.
4. If you already use Procmail to filter mail, insert the above bold lines before the lines below it.  This will put any spam that does not match your other rules in the new folder.

After saving, any messages identified as spam by our email gateway will be delivered into your newly created sub-folder.

GroupWise Users

At this time, unfortunately, there are no GroupWise specific instructions to take advantage of server side spam filtering above what the global settings provide.

Future Enhancements

Many current projects are currently underway that will directly or indirectly improve Spam filtering from both a user and server perspective.  It is hoped that the new improvements will address most, if not all, issues many users have with the current infrastructure.

1. IMAP (cyrus) Server upgrade
   The current IMAP server has not been upgraded for some time due to many issues behind the scenes.  Currently, we are very close to a major upgrade of the IMAP system.  (slated to happen by the end of March)  For Spam, the server upgrade will add Sieve server-side filtering, allowing simplified, user configurable email filtering on the server.  Some mail clients support sieve natively (including Mulberry) meaning you would not have to edit your filters from a web page. 
2. Econtrol rewrite
   The current Email Control Center interface is in the process of being completely rewritten from the ground up.  At first, after the new interface is in place, only minimal improvements will be implemented.  With the modernized package, however, adding new functionality to the service will be much easier, and, among the first goals, we will be adding a more user friendly interface to procmail.  Also, we plan to offer web friendly spam filter preferences and plan to extend this functionality to GroupWise users.  Once we get it up and running we will be taking suggestions for improvements and changes to allow it to become a valuable tool in your email arsenal.  Right now, we are concentrating on keeping its deployment on track to be around the decommissioning of the old IMAP server some time after the new server becomes primary.  For a preliminary mock-up of what the new interface will look like, visit this tentative design layout
3. Greylisting
   Greylisting is currently being evaluated for implementation on our email infrastructure.  In simple terms, it is a way of greatly reducing the amount of spam that can come from one source by slowing down their ability to send, and rejecting invalid mail servers.  Once the redundant systems are in place (see below), testing will take place with minimal end-user impact.  If the implementation is effective enough to greatly reduce the amount of spam without causing appreciable delay to legitimate external mail servers, greylisting will be the latest addition to our front line of defense. 
4. Mail Server Redundancy
   Many aspects of our email infrastructure do not have any active redundant systems for load balancing and / or failover.  Because of this and the steady increase of spam traffic, our external gateway, in particular, is routinely overloaded.  It is planned over the long-term to remedy this condition to improve performance, maximize uptime, and provide transparent upgrade paths for existing systems. 
5. LDAP Routing
   Centralizing email server configuration files will provide benefits to MUSC too numerous to mention.  As it relates to spam, today's localized, individual configurations of Sendmail on our more and more robust, yet complex email system result in the individual servers only knowing their specific task.  By putting all configuration information in one central and redundant source, the external buffer will be better equipped to make decisions on incoming mail.  Right now, for example, MUSC's mail hub is the only email server that knows all valid MUSC email addresses.  This means that any external email to an MUSC domain must traverse through our entire email system twice for every email to an invalid user.  By adding this information to a secure LDAP server accessable only at the server level, the external server can reject messages to invalid senders before they even get to disk on any system.  This alone should decrease traffic delays due to spam processing tremendously.
6. Blacklist Search Blog
   By popular request, a web tool allowing anyone with an MNA to track down possible blacklist issues with valid senders is being developed to, hopefully, simplify troubleshooting at the end-user level.

Conclusion

This document attempts to shed some light on the difficult problem we all face with today's hostile Spam war.  As well, it aims to both educate and provide what, we hope, you will see as a light of growing intensity at the end of a long, growing tunnel.  If you feel any of this document is in error, or have any questions or concerns, please let me know and I will try to provide assistance in a timely fashion.  Thank you.

Paul Arrington

Email Me