LogSat Software's SpamFilter for ISPs

How it works
SpamFilter Settings
Quarantine Database
Antivirus Plugin
Network Configuration Samples
Log Analysis & Statistics
Appendix A
Purchase

We created SpamFilter ISP in desperation to implement what our SMTP server lacks: Spam Filtering.

SpamFilter ISP's implementation is very simple. Its use is designed for ISPs and companies running their SMTP servers, not end-users. SpamFilter is designed to be your incoming SMTP server. It receives all emails addressed to your domain(s). If the sender's IP is not listed in one or multiple DNS-based block lists, if Bayesian statistical analysis of the email suggests it is not spam, and the email content passes your keyword tests, the email is then forwarded to your main SMTP server. End-users have web access to their own quarantined emails. Please look at the technical details for more information.

You can download SpamFilter free of charge here. SpamFilter ISP runs on Windows systems. The product is fully functional, it has no expiration dates, no nag boxes. You may evaluate as long as you wish. If you do decide that it is helping you substantially reducing the spam, we ask that you purchase a license for it's use.

How it works

Basic operation for SpamFilter ISP is as follows. SpamFilter is configured to handle your primary MX record for incoming emails (see DNS details here). SpamFilter can be configured to listen on a specific IP, multiple IPs or all IPs bound to the NIC card. See Configuration Section for more details.

What emails can be blocked (blacklisted)

• MAPS Servers - SpamFilter checks the IP address initiating the connection. If it is listed in one of its many DNS blacklists the connection is refused.
• SURBL Servers - SpamFilter scans the content of emails for any HTTP links and URLs. Every link found is then tested against one of the many SURBL DNS blacklists available. If present, the connection is refused.
• Image Filter - SpamFilter is able to scan actual images embedded in emails for spam content.
• SFDB Filter - All the thousands of SpamFilter installations in the world are networked together to create a huge database of spammer's IP addresses. SpamFilter uses this database, updated in real-time, to block spam.
• Local IP Blacklist - If the remote IP address matches an entry in your local IP blacklist file, the email is rejected.
• Local Domain Blacklist - If the domain portion in the sender's email address is in your local domain blacklist file, the email is also rejected.
• Local Country Blacklist - The sender's Country is tested to see if it is in your list of undesired countries. If so, the email is refused. This product includes GeoIP data created by MaxMind, available from http://maxmind.com.
• Local FROM EMail Blacklist - The sender's email address is checked against your local list of blacklisted email addresses. If present, it is rejected.
• Local TO EMail Blacklist - The recipient's email address is checked against your local list of blacklisted email addresses. If present, it is rejected.
• Attachment Blocking -  You can check emails for specific attachments or attachment extensions. If found, the email is rejected.
• Keyword Content Filtering -  You can check email content and subject for specific keyword and/or phrases. If found, the email is rejected.
• Bayesian statistical DNA fingerprinting -  SpamFilter ISP features statistical DNA fingerprinting of incoming emails. This filter is self-learning, continuously analyzing your incoming traffic to improve its accuracy with time.
• SPF - Sender Policy Framework - SPF fights email address forgery and makes it easier to identify spams, worms, and viruses. Domain owners identify sending mail servers in DNS. SpamFilter ISP verifies the envelope sender address against this information, and can distinguish legitimate mail from spam before any message data is transmitted.
• Honeypot Emails -  You can have a list of "honeypot" email addresses. Any email sent to an address in the list will cause the sender's IP to be blacklisted.
• Automatic spammer's IP caching - SpamFilter can automatically blacklist in a temporary cache the IP address of repeat spammers.
• Additional anti-spam tests
  • SpamFilter can check to see if the recipient address has a % sign in it. Many SMTP servers are susceptible to being tricked into relaying with this.
  • Connections can be rejected if the remote server does not have a reverse DNS PTR entry.
  • Connections can be rejected if the remote server does not have a valid MX record in their DNS.
  • Refuse connections if the remote server attempts more than n RCPT TOs in a single connection or if there are too many spaces in the subject line.

What emails are allowed to be delivered (whitelisted)

• Allowed domains - If the IP passes the DNS tests SpamFilter then checks the recipient domain. If the domain is listed as a local domain, then the recipient is accepted. This is done to prevent spammers to use SpamFilter to relay. It is very important to add your own domains to the local domain list. If not, you will not be able to receive email.
• Excluded IPs - If an IP is blacklisted, but you really need to be able to receive email from that domain anyways, the domain can be added in an exclude list as to allow it to bypass the blacklist rules.
• Excluded Domains - If an IP is blacklisted but you still wish to receive email from them, the IP can be added to an IP exclude list to allow it to bypass the blacklist rules.
• Unfiltered Emails - If you have users who do not want to receive filtered emails, they can be accommodated by adding them to a pass-list. Emails addressed to them will bypass all of SpamFilter rules.
• Excluded FROM Emails - If you want a sender's email address to be excluded from all filtering rules, you can add it to an exclude list.
• Authorized TO Emails - If you want SpamFilter to only deliver emails to specific addresses in your domain(s), you can manage such a list here. Please not that if such a list is present, SpamFilter will not deliver email to an address unless it is present in such a list. Use with care.
• Keyword whitelisting -  You can provide your customers with specific keywords that, if found in the body or subject of emails, will bypass all filtering rules. 
• User Authentication - SpamFilter supports sender's authentication via Active Directory, LDAP, and Unix-style password files.

SpamFilter Settings

Configuration tab

• SMTP listener socket

  • IP(s) - SpamFilter can be configured to listen on a specific IP, multiple IPs (separate the IPs with a comma in the IP input box), or all IPs (leave the IP field blank).

  • Port -  The port on which SpamFilter will listen on. 25 is the standard SMTP port.

  • SSL Port - SpamFilter supports SSL with SMTP. The standard SSL port is 465, however SSL is disabled by default by setting the port to 0. Please see the SSL certificates section for more information about using your own certificates.

  • FQDN - This is the Fully Qualified Domain Name that SpamFilter will output in the welcome banner sent to initiating connections.

• Destination Server

  • Name - The name or IP address of the default server where your email will be forwarded to. Please note that this default value can be overridden for individual domains if you need to have multiple routes for your domains. Please see the section on Local Domains for more info.

  • Port -  The port on which email will be forwarded to. Having a port other that 25 can be useful in situations where you only have access to a single IP. If you configure your destination MTA server to listen on a port other than 25, you can have both SpamFilter and your MTA co-exist on a single IP address.
    Note - if the destination server is unavailable, emails are saved to the queue directory. Redelivery of items in the queue is attempted every 60 minutes, and also every time SpamFilter is started.

• DNS Server - Enter the IP of your DNS server here.

• Other Options

  • Max concurrent incoming SMTP connections - You can limit the maximum number of concurrent incoming connections here.

  • Max concurrent incoming SMTP connections from single IP- You can further limit the maximum number of concurrent connections originating from an single IP address with this setting.

  • Max Recipients in single session - Use this setting to limit how many RCPT TO commands can be issued in a single session.

  • Min MAPS matches needed to reject msgs - Sometimes MAPS blacklists can be too strict and list legitimate domains in their blacklists. You can reduce the number of false positive by requiring that more than one single blacklist match is found before rejecting a connection.

  • Max Email Size - Incoming emails can be blocked if they exceed a certain size.

  • Process queue every n minutes - Use this setting to control how often SpamFilter attempts to redeliver the items on hold in the queue directory.

  • Max number of spaces in subject line - Many spam messages contain large number of spaces and tabs, they can be filtered here.

  • Image filter threshold - Adjusts the threshold of the image filter. graphics embedded in emails are scanned for spam content. The higher the threshold, the more aggressive the filter.

  • Bayesian Filter Threshold - Use this slider to control the accuracy of the statistical filter. Incoming emails are assigned a probability of being Spam, ranging from 0% (most likely a valid email) to 100% (most likely Spam). Any emails that have a probability of being spam above the value you set will be rejected. Typical threshold values are in the 99.9% range.

  • Days to quarantine rejected emails - Normally SpamFilter will reject an email if it considered as spam. You can optionally choose to receive and archive those emails rather than having them lost. The remote server will still receive an error stating that the email was rejected, but you will keep a copy in the quarantine directory for this amount of days. This will allow you to force delivery of legitimate email which could have been filtered. If you enter a 0 in this field quarantine is disabled and email is rejected immediately.

  • Allow % in address - SpamFilter can then optionally check to see if the recipient address has a % sign in it. Many SMTP servers are susceptible to being tricked into relaying mail with this. Ex. if you are isp.com, then a spammer could try to use joe%yahoo.com@isp.com to relay mail to joe@yahoo.com if your server is vulnerable.

  • Logging - Check this box to enable logging in the log directory.

  • Remember Stats - Check this box to save the email statistics when shutting down SpamFilter.

  • Disable Connections Grid - The Connections tab will show you in real-time what the various connections on your servers are and what they are doing. If you have a busy site with 500 concurrent connections this list can get pretty crowded and unwanted....

  • Auto-check for new build - If checked SpamFilter will connect with our website to see if a new version is available. SpamFilter will issue a simple GET request to http://logsat.com/spamfilter/version.htm to retrieve the version number. Absolutely no data will be sent to us! 

  • Tag Spam & Deliver - Allows to tag spam by adding the header "X-SF-SPAM:Y" to email classified as spam. The email is then forwarded to the destination SMTP server. This allows administrators to handle spam as they wish on the back-end.

  • Tag Spam in Subject & Deliver - Allows to tag spam by prefixing the word SPAM: in the subject line of emails classified as spam. The email is then forwarded to the destination SMTP server. This allows administrators to handle spam as they wish on the back-end.

  • Enable Cached IP Blocking - If an IP address sends more than a certain number of spam emails (3 by default) during a certain time interval (10 minutes by default), then it can be temporarily banned (blacklisted). All further connections from that IP address will be immediately rejected without allowing the sender to transmit any data. This should greatly reduce the load on the server. A banned IP address will be automatically removed from this temporary blacklist after a defined time interval (60 minutes by default). To prevent specific IPs to be added to this list, they can be added to DoNotAddIPToHoneypot SpamFilter.ini option.

  • Reject if no reverse DNS - SpamFilter can be configured to reject emails if the remote server does not have a valid reverse DNS PTR entry.

  • Reject if Empty "Mail From" - If this option is checked SpamFilter will reject all emails with an empty "Mail From" field. Please note that this setting will delete legitimate email, as in email receipt notifications and some error emails.

  • Reject if "Mail From" = "Mail To" - Reject all emails where the sender's email is the same as the recipient's email. Note that this causes problems with users who send emails to themselves using EBay's web interface for example.

  • Reject if "From Domain" = "To Domain" - SpamFilter can reject all email where the sender's domain is the same as the recipient's domain. Usually your users will not go thru SpamFilter when sending emails to themselves, Spammers often use this technique

Black / White Lists tab

• Blacklists

  • MAPS Blacklist servers - SpamFilter checks the IP address initiating the connection. If it is listed in one of its many DNS blacklists the connection is refused. SpamFilter can reject connections based on a configurable minimum number of matches. A ",true" after an RBL entry means their DNS is expecting the IP to be reversed, i.e. to test a connection from 1.2.3.4 they expect 4.3.2.1.bl.spamcop.net

  • SURBL Blacklist servers - SpamFilter scans the content of emails for any HTTP links and URLs. Every link found is then tested against one of the many SURBL DNS blacklists available. If present, the connection is refused.

  • Blacklisted IPs - You can keep a file with additional IPs that you want to blacklist by entering the filename below. If the file does not exist it will be created. The file is reloaded every minute. List individual IP addresses on each line. Use an ending .0 for a Class C wildcard (i.e. 192.12.45.0 to block 192.12.45.1 --> 192.12.45.255). The contents of the file will be loaded in the memo box, allowing you to make changes to the file.

  • Blacklisted Domains - You can keep a file with additional Domains that you want to blacklist (based on the MAIL FROM field) by entering them below below. Enter one domain per line, wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx). If the file does not exist it will be created. The file is reloaded every minute. The contents of the file will be loaded in the memo box, allowing you to make changes to the file. This list supports the :NULL option to send emails in a black hole. If an entry is in the form domain1.com:NULL it will cause all emails from domain1.com to be accepted and then sent to NULL right away. Such emails will not cause NDRs, they will not be quarantined, they will not be seen by the users. If an entry is in the form domain1.com:NoNDR such emails will not cause NDRs as in the DoNotSendNDROnQuarantine parameter in the ini file. This list supports the :Honeypot option, which will cause the sender's IP address to be automatically blacklisted in the future.

  • Blacklisted FROM Emails - If you want to block any particular email addresses, enter them here, one email per line. Wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx). This list supports the :NULL option to send emails in a black hole.If an entry is in the form user1@domain1.com:NULL it will cause all emails from user1@domain1.com to be accepted and then sent to NULL right away. Such emails will not cause NDRs, they will not be quarantined, they will not be seen by the users. If an entry is in the form domain1.com:NoNDR such emails will not cause NDRs as in the DoNotSendNDROnQuarantine parameter in the ini file. This list supports the :Honeypot option, which will cause the sender's IP address to be automatically blacklisted in the future.

  • Blacklisted TO Emails - If you want to block any particular destination addresses, enter them here, one email per line. Wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx).  If an entry is in the form user1@domain1.com:NULL it will cause all emails to user1@domain1.com to be accepted and then sent to NULL right away. Such emails will not cause NDRs, they will not be quarantined, they will not be seen by the users. If an entry is in the form domain1.com:NoNDR such emails will not cause NDRs as in the DoNotSendNDROnQuarantine parameter in the ini file. This list supports the :Honeypot option, which will cause the sender's IP address to be automatically blacklisted in the future.

  • Country Filters - SpamFilter checks the what country incoming connections are coming from. The current number of connections for each country can be updated by clicking on the Update Stats Now  button. Columns can be sorted by clicking on the column header. This will help you in sorting countries and hits so you can determine if there are any countries you do not wish to receive email from.

  • Honeypot Emails -  You can have a list of "honeypot" email addresses. Any email sent to an address in the list will cause the sender's IP to be blacklisted. The IP address will be added to the file HoneypotBlockedIPs.txt, which contains the list of blocked IPs automatically added by this filter. This filter is typically used by adding non-existent email accounts to it that you know should never receive mail. If they do, then the email is likely spam, so the remote IP will be blacklisted automatically.
  • Attachment Blocking -  You can block emails that have unwanted attachments. You can keep a file with banned attachments here. check emails for specific attachments or attachment extensions. If the attachment is found, the email is rejected. Wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx). This list supports the :NULL option to send emails in a black hole. If an entry is in the form filename:NULL it will cause all emails with the filename attachment to be accepted and then sent to NULL right away. Such emails will not cause NDRs, they will not be quarantined, they will not be seen by the users. If an entry is in the form domain1.com:NoNDR such emails will not cause NDRs as in the DoNotSendNDROnQuarantine parameter in the ini file. This list supports the :Honeypot option, which will cause the sender's IP address to be automatically blacklisted in the future.

  • Keywords Filter - You can check email content and subject header for specific keyword and/or phrases. If found, the email is rejected. You can also use Regular Expressions (RegEx). If the keyword file does not exist it will be created. The file is reloaded every minute. The contents of the file will be loaded in the memo box, allowing you to make changes to the file. This list supports the ::NULL option to send emails in a black hole. If an entry is in the form keyword::NULL it will cause all emails to be accepted and then sent to NULL right away. Such emails will not cause NDRs, they will not be quarantined, they will not be seen by the users. If an entry is in the form keyword::NoNDR such emails will not cause NDRs as in the DoNotSendNDROnQuarantine parameter in the ini file. This list supports the ::Honeypot option, which will cause the sender's IP address to be automatically blacklisted in the future.  Please note that unlike in other cases, with the keyword list you must enter the ":" symbol twice to specify the extra tag.
    The keyword rules are as follows:

  • Keyword(s) are entered on separate lines
  • If all keyword(s) on a single line are found in the message, it is then rejected.
  • Separate keywords on a single line by using commas.
  • The subject of an email can be scanned for keywords by prefixing the keyword list with Subject:
   

  Sample keyword entries:	Sample email content and effects:
  mortgage,click here,mailing free,mailing,list unsubscribe Subject:viagra	.... low mortgage, click here to be removed from our mailing ...	rejected	matches all keywords in 1st line
  	.... low mortgage, click over here to be removed from our mailing ...	accepted	click over here is no match for click here
  	.... low mortgage, click over here to unsubscribe from our mailing ...	rejected	matches single keyword on 3rd line

• Whitelists

  • Local Domains - SpamFilter will only deliver email to the domains listed here. Wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx). If the domain in the  RCPT TO email address is listed as a local domain, then the recipient is accepted. This is done to prevent spammers to use SpamFilter to relay email to third party email addresses/servers. It is very important to add your own domains to the local domain list. If not, you will not be able to receive email. If you need to have any domain listed here forward its destination email to a different server than the default destination server, you can specify so here. You can override the default destination server by appending the forwarding mail server and port to any domain in this list. The syntax should be as follows:
    
    DomainName:DestinationServer:DestinationPort - ex. logsat.com:mail.netwide.net:25
    

  • Excluded Domains / IPs - You can keep a file containing a list of  any "MAIL FROM" domains or any IPs from which you want to receive email if they would be blocked by any of your blacklist rules. Enter as many IPs or domains as you wish, one per line. Wildcards (* and ?, same rules as DOS wildcards) are allowed. To exclude a whole class C, enter it as 209.20.21.*. If the file does not exist it will be created. The file is reloaded every minute.

  • Unfiltered Emails - Any local email address listed here will cause SpamFilter to bypass all blacklist rules for it. If you have any users who do not want to have their email filtered, enter them here. Wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx). This list supports the :TAG option to bypass the default "pass all" rule for entries on this list. If an entry is in the form user@domain1.com:TAGSUBJECT it will cause all emails sent to user@domain1.com to be accepted and then delivered to that user no matter what. However emails that are classified as spam by the various filters will have the prefix "SPAM:" added to the subject line. If an entry is in the form user@domain1.com:TAG it will cause all emails sent to user@domain1.com to be accepted and then delivered to that user no matter what. However emails that are classified as spam by the various filters will have the header "X-SF-SPAM:Y" added to them.

  • Excluded FROM Emails - You can keep a file containing a list of sender's email address to be excluded from all filtering rules. Enter one email address per line, wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx). If the file does not exist it will be created. The file is reloaded every minute. The contents of the file will be loaded in the memo box, allowing you to make changes to it.
  • Authorized TO Emails - You can keep a file containing a list of authorized recipients. If you want SpamFilter to only deliver emails to specific addresses in your domain(s), you can manage such a list here. Enter one email address per line, wildcards (* and ?, same rules as DOS wildcards) are allowed. You can also use Regular Expressions (RegEx). If the file does not exist it will be created. The file is reloaded every minute. Please not that if such a list is present, SpamFilter will not deliver email to an address unless it is present in such a list. Use with care. Delete the filename form the edit box to disable the list.
  • Keywords Filter - You can check email content and subject header for specific keyword and/or phrases. If found, the email is allowed through the filters. Useful if you want to allow certain customers to send you email without having to place them all in a email address whitelist. The same syntax rules as the blacklist keywords apply. 
  • User Authentication - SpamFilter is able to authenticate senders via LDAP, Active Directory, and a Unix-style password file. If a user is successfully authenticated, they will be whitelisted and will be able to use SpamFilter to relay emails. This will allow users to use SpamFilter's server as their "Outgoing SMTP Server" in their email client configuration. SSL support is available to encrypt the login credentials. Please see the SSL certificates section for more information about using your own certificates.

Bayesian Statistical Filtering

The new v2 release of SpamFilter ISP features statistical DNA fingerprinting of incoming emails. The statistical analysis is performed using Bayesian rules. Tokens within incoming emails are scanned and categorized in a corpus file. The content of all new incoming email is fingerprinted and checked against the historical data. If there is a high statistical probability that the email is spam, it is rejected. 

The statistical engine kicks in after 5,000 non-spam and 5,000 spam emails have been received (values customizable by editing the SpamFilter.ini file). This is done to build a valid statistical base to use before emails are rejected. During this period of time, it is critical to avoid false positives. If a good email is quarantined, forcing it's redelivery either thru the web interface or the SpamFilter GUI will "teach" SpamFilter that the fingerprint in that email is a "good" one, and the statistical DNA database will adapt itself to it. It is very important initially to check the quarantine often to force delivery of legitimate email that has been blocked by the "regular" filtering rules.

A slider is used to control the accuracy of the statistical filter. Incoming emails are assigned a probability of being Spam, ranging from 0% (most likely a valid email) to 100% (most likely Spam). Any emails that have a probability of being spam above the value you set will be rejected. Typical threshold values are in the 99.9% range.

SPF - Sender Policy Framework

SPF is an open source standard that is emerging as a solution to prevent spammers from using fake email addresses. The following description was taken from the official SPF website at http://spf.pobox.com:

Domains use public records (DNS) to direct requests for different services (web, email, etc.) to the machines that perform those services. All domains already publish email (MX) records to tell the world what machines receive mail for the domain.
SPF works by domains publishing "reverse MX" records to tell the world what machines send mail from the domain. When receiving a message from a domain, the recipient can check those records to make sure mail is coming from where it should be coming from.
With SPF, those "reverse MX" records are easy to publish: one line in DNS is all it takes. Suppose a spammer forges a hotmail.com address and tries to spam you.

He connects from somewhere other than hotmail.

When his message is sent, you see MAIL FROM: <forged_address@hotmail.com>, but you don't have to take his word for it. You can ask Hotmail if the IP address comes from their network.

(In this example) Hotmail publishes an SPF record. That record tells you (your computer) how to find out if the sending machine is allowed to send mail from Hotmail.

If Hotmail says they recognize the sending machine, it passes, and you can assume the sender is who they say they are. If the message fails SPF tests, it's a forgery. That's how you can tell it's probably a spammer.

SpamFilter ISP looks up SPF DNS records for all incoming emails. If an SPF record exists, the query results can be any one of the following:

• Pass: the message meets the domain's definition of legitimacy.
• Neutral : the message does not meet a domain's definition of
  legitimacy, but the SPF client MUST proceed as if a domain did not
  publish SPF data. Likely used by domains in transition phase
  who are beginning to adopt SPF.
• Softfail : the message does not meet a domain's strict
  definition of legitimacy, but the domain cannot confidently state
  that the message is a forgery.
• Fail : the message does not meet a domain's definition of
  legitimacy.

If the result is "Pass" the email will pass the SPF filter. Behavior for all the other failing results can be customized by the administrators in the SpamFilter GUI by adjusting the settings in the Settings - SPF Filter tab.

SFDB Filter - SpamFilter Distributed Blacklist

The SFDB filter uses a very powerful resource to stop spam:
The entire global SpamFilter ISP user community.

Anytime an IP address is added to SpamFilter's local IP blacklist cache, the SFDB filter updates our Distributed Blacklist centralized database. This allows the SFDB filter to have access to a huge repository of spammer's IPs, updated in real-time by all the SpamFilter ISP users in the world. IP addresses from the database are automatically aged and removed from the database within 24 hours if they receive no further reports.

The SFDB filter detects spam by checking IP addresses against the SFDB database. The "network reliability" level tells SpamFilter how many different users must have reported a specific IP in order to classify it as spam.

The SFDB filter will reject IPs if there are *currently* in the database more that a minimum number of reports for a certain IP. This threshold is the "Network Reliability". Enter 0 to disable the SFDB filter.

In the free version of SpamFilter, the SFDB filter will only query the SFDB database, reporting of new IPs is disabled. Furthermore, in the free version, the SFDB filter is limited to a 15-day trial period.

Filters per-Domain

Most filters can be enabled and disabled individually for each domain. This allows for further customization of the filtering settings for each email domain that SpamFilter handles.

Regular Expressions

Every single LINE of text in every single local list that is currently able to support wildcards will be treated with the usual wildcard rules. But if any one of those lines of text starts with an open parenthesis "(", and end with a closing parenthesis ")", then that one single line will be tested using regex rules. Please see additional help file for regular expression syntax.

Customized Items

Most rejection notices to the remote servers can be customized. In the error string you can embed the following connection-specific parameters:

• %IP% - The IP address of the remote server connecting to SpamFilter

• %Domain% - The MAIL FROM domain name of the incoming email attemp

• %EMailTo% - The recipient of the incoming email attempt

• %EMailFrom% - The sender's email address

If you want to reset a field to its default value you can delete its reference in the SpamFilter.ini file and restart SpamFilter.

Quarantine Database

The quarantine in SpamFilter v1.2 is completely database driven. The old file-based quarantine is no longer functional.

SpamFilter will continue to function if a database is not configured, but emails will not be quarantined in this case. A database connection can be added using the SpamFilter GUI while the program runs. The connection can be defined either by using an Universal Data Link File (.UDL) or by specifying a complete connection string within SpamFilter. We recommend using an UDL file (a sample file is included) since it can be used by the webserver as well without exposing the database password in the web files.

Currently the following database platforms have been tested:

• Microsoft Access (MDAC 2.8 is required)
• Microsoft SQL Server 7 and up
• Oracle 8.x and up
• MySQL

An empty MS Access database is included in the distribution files. SQL scripts to create the necessary tables for Oracle, MSSQL and MySQL are included. 

This release allows the creation of the SpamFilter tables from the GUI, using a 3 step process, without needing to manually run the sql scripts against the database. Note that to do so, the database user account defined in the connection string must have the proper DB rights to be able to create tables (usually dbOwner). Do not forget to check the option that stores the password in the connection string or in the UDL file!

The three tables used for the quarantine and the privileges needed for them are:

• tblMsgs (contains the message body)
  • DB user needs SELECT, INSERT, UPDATE, DELETE rights
• tblQuarantine (contains all details about the email except the content)
  • DB user needs SELECT, INSERT, UPDATE, DELETE rights
• tblRejectCodes (contains the current 13 reject categories)
  • DB user needs SELECT rights

Using the SpamFilter GUI administrators are able to view and force delivery of any email in the quarantine area.

Web interface – We provide sample ASP and PHP pages that interface with the quarantine database. The webserver does not need to communicate in any way with the server where SpamFilter is installed on. The web server instead does need to connect to the database server. We recommend using an UDL file for the database connection in the ASP/PHP code, as you are able to place this file in a secure location on the webserver, outside the public web area, making it harder for intruders to gain access to it. The UDL file can contain the database password, so you will not have to store it in the web pages. The database connection is defined in the db_Connect web page

The important part in using the web interface is choosing a way to authenticate users. We provide a tblLogins table in the database that can store a list of Email addresses and passwords. Our sample authenticate.asp and authenticate.php pages perform authentication based on that table. You can choose your own authentication schema and create your own pages to authenticate in other ways. At the end, ensure that the authenticated email address will be stored in a session variable.  The ListSpam and ResolveSpam pages list and deliver the emails belonging to the address stored in the session variable.

For the ASP pages only (PHP version will follow soon) we created a sample Register.asp page where users can self-register for the quarantine access. Users can enter their email address, it will be stored in the tblLogins table, a random password will be generated and emailed to them. With it they will be able to then access their quarantine area.

Starting from SpamFilter ISP v 2.1, when an end user forces the delivery of a quarantined email to his mailbox, the sender of that email will be whitelisted so that the number of false-positives (good emails wrongly classified as spam) is reduced. The list of user-created entries is stored in the file AutoWhiteListForceDelivery.txt. The whitelisting is on a per-user basis, meaning that a sender is whitelisted only when he sends emails to that specific recipient. This will prevent a user mistakenly whitelisting a spammer, who could then send spam to all of your users.

Note that in the free version of SpamFilter, the web interface will not deliver emails to the recipients!

The tblQuarantine has a Deliver field and an Expire field with default values of 0. Changing the Deliver field to 1 will cause SpamFilter to deliver that email within 10 seconds. Changing the Expire field to 1 will cause SpamFilter to erase that email from the database within 1 hour. The web pages simply update these two fields to deliver and delete the emails.

Antivirus Plugin

Starting from version 2.5, SpamFilter ISP includes support for an anti-virus plug-in. LogSat Software has partnered with Norman to provide optional antivirus protection for email traffic.

The antivirus plug-in is be available for purchase separately from SpamFilter ISP and is an optional component. Unlike SpamFilter ISP's licenses, the antivirus plug-in is offered as a subscription service with a yearly subscription fee of $400.

An Activation Code is required to enable the antivirus plug-in. A 15-day trial Activation Code can be obtained from SpamFilter's GUI on the Settings - Antivirus tab. The code is required only for the antivirus plug-in activation. SpamFilter (both retail and free versions), will continue to work and stop spam even without the antivirus plug-in. Please note that only one request for a trial code will be honored per installation.

Technical notes - SpamFilter can run with or without the antivirus plug-in. When SpamFilter starts, it will check for the plug-in files. If they are found, antivirus support will automatically be enabled. We recommend installing the antivirus plug-in after installing SpamFilter. Restart SpamFilter after installing the plug-in to activate it.

The antivirus plug-in is a set of 3 DLL's that are to be installed in the SpamFilter directory: dwnse.dll, ncl.dll and nselapi.dll. In addition to those files, the Norman scan engine needs to be present. If a Norman product is not already installed on the server, the installer adds the necessary files (including the virus definitions) in the SpamFilter\nse directory.

There are 2 additional DLL's that are placed in the SpamFilter directory by the install program: libeay32.dll and ssleay32.dll. These DLLs are used by the antivirus plug-in. If performing a manual install please make sure you copy these files to the SpamFilter directory.

Network Configurations Samples

In order to use SpamFilter properly, you need to configure your SMTP setups in such a way as to have SpamFilter handle all of your incoming email.

Log Analysis & Statistics

SpamFilter ISP log files can be parsed by Sawmill, an excellent log analysis tool. Sawmill generates reports of email traffic by IP, domain, country, sender and recipient, action taken on messages and much more. In the SpamFilter\Database directory you will find the Sawmill plug-in file SpamFilterISP. If your copy of Sawmill 6.5 or higher does not recognize SpamFilter ISP's log format, simply copy that file in the Sawmill\LogAnalysisInfo\LogFormats directory to allow it to read SpamFilter ISP logs.

SSL Certificates

SpamFilter requires the following certificates to be placed in SpamFilter's root directory when accepting SSL connections via SMTP or thru the internal web interface:
root.pem: Certificate Authority (CA) certificate
cert.pem: x.509 certificate, signed by CA
key.pem: x.509 private key

Included with SpamFilter's distribution are sample certificates issued LogSat Software.  While they are fully functional and allow encrypted communications, they are signed by our internal Certificate Authority (CA). As this CA is not trusted by browsers and mail client software, using them will often cause security warnings in client software.
Administrators can purchase SSL certificates from commercial entities like Verisign or Thawte, which will eliminate any security warnings. Administrators can also select to issue their own certificates, using a CA they trust. In these cases, the certificates will have to be converted to .pem format if they are issued in other formats. The OpenSSL utility from www.openssl.org can be used for both of these purposes. Pre-compiled binaries for OpenSSL for Windows can usually be found at www.openssl.org/related/binaries.html.

Creating and self-signing your own certificate

To create and self-sign a certificate using OpenSSL, you can issue, from a MSDOS command prompt, the following command:

openssl req -new -x509 -keyout key.pem -out cert.pem -nodes -days 3650

When following the prompts to generate the certificate, please note that when asked for the "Common Name", you should enter the DNS name of the server you are installing the certificate on, ex: mail.logsat.com.

Now that you created and signed your own certificate, you will have the files key.pem and cert.pem. SpamFilter also needs the CA certificate. As you signed your own certificate, the CA is certificate will be the same as your public certificate. So simply copy/paste the file cert.pem to root.pem. Place all three files (key.pem, cert.pem and root.pem) in the SpamFilter directory and restart SpamFilter to activate the certificates.

Exporting an existing commercial SSL certificate

If you already purchased a commercial SSL certificate, you need to export it into the .pem format used by SpamFilter. This procedure assumes that you have already received your key and certificate pair from some Certificate Authority (like Verisign or Thawte) and that you have installed them in Microsoft Internet Explorer in the Personal Certificates Store.

• Export Certificate
  • Select the certificate and export it as a .pfx file (Personal Exchange Format). You may optionally protect it with a password.
• Convert .pfx to .pem using OpenSSL:
  • Issue the following command from a MSDOS prompt:
    
    openssl.exe pkcs12 –in <your file>.pfx –out <your file>.pem
    
    Openssl.exe will prompt you for a password. Enter it if you used one, or leave it blank if you did not specify one. It will also prompt you for a new password for the .pem file. This is optional, but if you protect it with a password you will need to enter the password in the SpamFilter.ini file (SSLCertificatePassword setting). Use the "-nodes" option in the above command to avoid specifying a new password.
• Split the .pem file into the private and public key files
  • If you examine the new .pem file jsut created with a text editor, you will notice that it consists of two parts. The two parts are the private key and the certificate (public key). There is also some addition information included. SpamFilter requires that this information be separated into separate files.
  • Key.pem
    • Create a blank text file with Notepad, name it key.pem, and paste everything between and including these two statements:
      
      -----BEGIN RSA PRIVATE KEY-----
      
      -----END RSA PRIVATE KEY-----
  • Cert.pem
    • Create a blank text file with Notepad, name it cert.pem, and paste everything between and including these two statements:
      
      -----BEGIN CERTIFICATE-----
      
      -----END CERTIFICATE-----
  • Root.pem
    • The final file that SpamFilter requires is the Certificate Authority certificate file. You can obtain this from the Internet Explorer in Trusted Root Certificate Authority dialog. Select the Authority that issued your certificate and export it in Base64 (cer) format. This format is also the same as PEM format so after export simply rename the file to root.pem.
       

Appendix A

The following custom parameters in the [server settings] section of the SpamFilter.ini file can be used to change default behaviors.

;Set this to 1 if you want to disable EHLO extensions
DisableEHLO=0

;Any emails whose text portion exceeds this number of KB will not be scanned for keywords and Bayes
;Higher values *may* catch more spam but will cause higher load on processor
MaxMsgSizeForKeywordScan=64

;Set FilterBase64html to 1 if you want to block any emails with Content-Transfer-Encoding=base64 and Content-Type=text/html or text/plain
FilterBase64html=0

;Set RequireHELOBeforeMAILFROM to 0 if you do not want to require remote servers to issue a HELO or EHLO command before sending the email
RequireHELOBeforeMAILFROM=1

;Controls the minimum number of good and spam emails that must be received before the Bayesian filter kicks in
MinEmailsForBayesKickIn=5000

;by default SpamFilter will not allow any IP to relay thru it except for 127.0.0.1 (localhost). Change DoNotTrustSelfByDefault to 1 if you do not want localhost to be able to relay
DoNotTrustSelfByDefault=0

;Remove any stale token in the corpus db.dat file that did not appear in incoming emails for the past n days
CleanUpCorpusIntervalDays=7

;Force disconnect of sessions after they have remained connected for this long
IdleDisconnectMinutesTimeout=15

;Force disconnect of sessions if a command has not been received within the last nn seconds
ReadTimeout=60

;Timeout when delivering emails to the destination SMTP server (in seconds)
ReadTimeoutOutgoing=60

;if turned on, this will cause tokens in incoming emails being logged to screen with relevant probabilities
ShowBayesianTokens=0

;Set TagSPAMAndQuarantine=1 if you want to prefix every quarantine subject line with the prefix specified in SPAMTagPrefix ini parameter
TagSPAMAndQuarantine=0

;This SPAMTagPrefix will be prefixed to all subject lines marked for "mark as SPAM and deliver" along with the ation specified by TagSPAMAndQuarantine
SPAMTagPrefix=SPAM:

;Setting DoNotSendNDROnQuarantine to 1 will prevent generation of NDR when email are quarantined by causing SpamFilter *not* to send an error code when quarantining emails
DoNotSendNDROnQuarantine=0

;If turned on, the threads that save to disk and load into memory the bayes corpus tokens will have increased priority
BoostBayesPriority=1

;if TrailingSQLSemiColon is set to 1 SpamFilter will add a ";" to the end of SQL statements. Disable only to help solve problems with some databases.
TrailingSQLSemiColon=1

;If turned on, any quarantined (false positives) emails that the end user force-delivers will cause the sender to be automatically whitelisted
AutoWhiteListForceDeliveryEnabled=1

;if EnableBadMailDir is set to 1, this will cause all emails that generate a "server error" when forwarded to your destination SMTP server will be saved in a "BadMailDir" for troubleshooting
EnableBadMailDir=0

;if ScanReceivedHeaders is set to 1 SpamFilter will add the "Received:" headers to the text examined for keywords and statistical Bayesian searches.
ScanReceivedHeaders=1

;Number of hours SpamFilter will retry to deliver messages in queue to your destination SMTP server if it was unreacheable. Enter 0 to try forever until back online.
ExpireRetryQueueHours=0

;Path to logfile directory
LogFilePath=

;Optional destination SMTP server where to forward SPAM emails only. Good emails are still forwarded to main SMTP server
DestSMTPServerForSPAM=

;The frequency in seconds for which the quarantine table is scanned to check for emails pending delivery - includes web-access password registration emails
QuarantineToDeliverCheckInterval=5

;By default the activity logfile is saved to disk every 60 seconds. Set RealtimeDiskLogging=1 to save the log every time it is updated
RealtimeDiskLogging=0

;Add any IPs (separated by commas - no wildcards) that you do not wish to be automatically added to the Honeypot IP blacklist or the IP Blacklist Cache
DoNotAddIPToHoneypot=

;An alternate server for sending NDR (non-delivery) notification emails can be used. Leave the "NotificationSMTPServer" value blank to use the default destination SMTP server
NotificationSMTPServer=
NotificationSMTPServerPort=25

;Set EnableDbgLogs=1 to enable separate detailed logging for troubleshooting purposes
EnableDbgLogs=

;The timeout in milliseconds for all DNS-related queries.
DNSTimeout=5000

;If an IP sends more than this number of spams in a certain period of time then it is temporarily banned (blacklisted)
IPCacheLimboCountTrigger=3

;If an IP sends more than a certain number of spams during this number of minutes then it is temporarily banned (blacklisted)
IPCacheLimboTimeTrigger=10

;If an IP address was banned because it sent too many spams in a certain time interval, it will be un-banned after this number of minutes
IPCacheBlacklistDuration=60

;You can force the antivirus plugin to block emails if they contain password protected archives that cannot be tested for viruses by setting this to 1
BlockArchivesWithPassword=0

;By default SpamFilter will only perform DNS lookups when the reverse DNS filter is enable. Change value to 1 to always perform a reverse lookup on connecting IPs
AlwaysDoReverseDNSLookups=0

;Specifies how often the logfiles are rotated (Min=1, Max=24). The default is 24 (rotates at midnight). A value of 1 means every hour at the hour, value of 2 means at 2am, 4am, 6am etc...
RotateLogsEveryNNhours=24

;Change DoNotStartWithoutAV to 1 if you do not want SpamFilter to start/run if there is an error with the Antivirus plugin.
DoNotStartWithoutAV=0

;Determines if SpamFilter should hold in the queue emails that were rejected by the destination SMTP server with an error in the 4xy range
QueueIfDestinationError400=true

;Determines if SpamFilter should hold in the queue emails that were rejected by the destination SMTP server with an error in the 5xy range
QueueIfDestinationError500=false

;Image filter threshold. Higher values indicate a more aggressive filter. 0 disables the filter. Min=0, Max=15
SpamImageThreshold=10

;Image filter color sensitivity. Used internally to detect color shades
SpamImageColorSensitivity=20

;Images embedded in email's html having a width smaller than this will not be scanned. Useful to bypass signatures and logos
SpamImageMinWidth=300

;Images embedded in email's html having a height smaller than this will not be scanned. Useful to bypass signatures and logos
SpamImageMinHeight=300

;Determines the number of points that will be scanned in a image to process it for spam
SpamImageSamplingPoints=200

;To reduce false positives, emails with multiple inline images can bypass the image filter by setting this value to 1
SpamImagePassMultiImage=1

;Set this to 0 to prevent queued emails to be spooled to memory, and force spooling to disk. While less efficient, spooling to disk helps allow existing antivirus software to detect and block some infected email files
SpoolQueueFilesToMemory=1

;If the private key of the SSL certificate is protected by a password, enter is here
SSLCertificatePassword=

;Some older email clients have a bug that requires them to see "AUTH=LOGIN" in the EHLo response rather than "AUTH LOGIN". Set this to 1 to add the incorrect syntax to the EHLO output
AddIncorrectAUTHLOGINEHLOEntry=0

;Use this option to customize the X-Server header SpamFilter adds to emails:
XServerHeader=LogSat Software SMTP Server
 

Purchase

SpamFilter can be licensed for $600 / production server. You may freely use it for development/evaluation purposes.

You may contact us at sales@logsat.com or visit our website at www.logsat.com/SpamFilter for any questions