| Re: Custom SpamAssassin scores and rules [message #122151 is a reply to message #122149] |
Thu, 18 June 2015 22:02   |
MarkK
Messages: 342
Registered: April 2007
|
|
|
|
|
You should be able to just copy the file there, the server does not need to be stopped at all to do that. If you are not able to write the file there, then you do not have the correct permissions to do so for some reason.
|
|
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #122724 is a reply to message #122723] |
Thu, 09 July 2015 13:08 |
yukiomishima
Messages: 116
Registered: July 2006
|
|
|
|
markK
a couple of quick questions if you could be so kind:
- whilst installing the rules has made a difference (so i am pretty sure i have loaded it in the correct location (we are running OSX))...is there anywhere within the kerio admin interface where i can check to see that the custom .cf file has been loaded and is being used
- one of the partners here is complaining that periodically we are having messages tagged incorrectly.. even from senders that have been sending to us for years... i don't suppose there is any way to prevent this sort of thing from happening... (we have a HUGE custom white/black list... but for those that are not on it yet... would be nice if there was a facility to have them added.. somewhat auto... say... if i have sent an email to someone >50 times... add them to a whitelist)
speaking of the custom white/black list... it looks like there is a facility for sorting by clicking on the various column headers... but... when i do... nothing changes... am i missing something?
huge thanks again for all your help
yukioMishima
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #122729 is a reply to message #122724] |
Thu, 09 July 2015 18:31 |
MarkK
Messages: 342
Registered: April 2007
|
|
|
|
yukiomishima
If you have put the custom .cf with all of the other ones, restart SA by unselect and reselect in Admin, then it will start using that set of rules. Only way that I know of seeing if it is using the rules is to look at the email's spam headers. Of course, you can send a spam from your personal email to a Kerio account that will hit one of the rules. Send a message that has "auto warranty" as the body of the message, and in the spam headers you should see a hit for lcl_BODY_30 that has added 3.0 points to the spam score.
As for the false tagging, take a look at the mail headers spam scoring, see what is getting the score up high enough to be tagged, and make the needed adjustments. That can mean either adding the sender (either their specific email address or the email address domain) to a custom allow rule, or adjusting the builtin or the custom SA spam scoring. As I had mentioned, these rules were put together for my employers industry and the spams we were receiving. Your experience may vary.
If you can take the time, look at the headers for some of the emails from your white / black list members and see what you need to do to have SA discover and properly score the emails. I also do have some entries in the custom rules to allow and block some specific email addresses and domains, but I wouldn't say it is huge.
As for sorting by column, that is annoying, but I believe that they have disabled that because the custom rules are processed in the specific order that they are shown. You can use the Search box to filter out the stuff that doesn't match what you are looking for.
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #122734 is a reply to message #122729] |
Thu, 09 July 2015 20:04 |
yukiomishima
Messages: 116
Registered: July 2006
|
|
|
|
markK
i did indeed pop the file into the same location that i had put the previous ones in
i was asking about the admin interface to see if there was a log entry etc that shows which .cf files were loaded
i sent a test email as suggested.. and recieved the header response as you described.. so all good (which i knew it was as spam has seriously decreased since installing the initial .cf file that you provided)
as for the headers... there is lots of info in there... and it is not super clear which info i need to utilise when writing either custom whitelist entries.. or SA rules
on the whole.. i have been using address / domain for 99% of my custome white/blacklist rules... but am not really sure if that is the best option
HUGE HUGE thanks again for all... all of your help and insight... as well as the .cf file has made an incredible difference
yukioMishima
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #122737 is a reply to message #114168] |
Thu, 09 July 2015 20:52 |
MarkK
Messages: 342
Registered: April 2007
|
|
|
|
In the email headers, to see what Spam Assassin rules and scores that have been hit is this:
X-Spam-Status: Yes, hits=7.0 required=5.0
tests=DNSBL_MULTI.URIBL.COM: 1.00,BAYES_99: 4.07,FUZZY_AMBIEN: 0.552,
HTML_MESSAGE: 0.5,MIME_HTML_ONLY: 0.4,T_REMOTE_IMAGE: 0.5,
UNPARSEABLE_RELAY: 0.001,TOTAL_SCORE: 7.023,autolearn=no
X-Spam-Flag: YES
X-Spam-Level: *******
This is from a spam that we received, scoring 7.023. Lets look at the hits. If you want specific meanings for the rules, you can either search google by the rule name, or search through the rule .cf files and read the brief explanation there.
DNSBL_MULTI.URIBL.COM: 1.00,
I have this black list turned on, but after so many hits to it, they start to just deny everything. I should either subscribe or turn it off.
BAYES_99: 4.07,
This is the Bayes filter rating this email very high.
FUZZY_AMBIEN: 0.552,
If your ambien is fuzzy, I think you have other problems besides getting spam. This rule hits when it appears that the word ambien was trying to be hid some how in the email. A rating of .552 seems low to me for something like this, so I would probably set a custom score to 1.5. That should allow for a good score on spams that hit this rule, but not automatically high enough for a possible valid email that contains the word ambien.
HTML_MESSAGE: 0.5,
MIME_HTML_ONLY: 0.4,
When I went through and start setting custom scores, I saw these in a lot on the spams. It was set to the detect-only value of like .001. I set it to .5 / .4 so that on valid emails it would not be a false trigger, but on spams to at least add some weight.
T_REMOTE_IMAGE: 0.5,
More of the above. This means there is a link to a remote image out on the internet. Not only spam behavior, but valid behavior as well. Wanted a little bit of weight from it.
UNPARSEABLE_RELAY: 0.001,
Here is on of the alert-only rules built in to Spam Assassin. it would take thousands of rule hits at .001 to my 5.0 mark it as spam threshold.
This rules means that Kerio was not able to trace back to the mail relay servers used. Unfortunately, that does not signal 100% spam. Not all valid emails and email servers behave the way that they should.
The SA for this rule can be found here
http://wiki.apache.org/spamassassin/Rules/UNPARSEABLE_RELAY
SpamAssassin Rule: UNPARSEABLE_RELAY
Standard description: Informational: message has unparseable relay lines
Explanation
The Received: lines from the email are analyzed to determine the relay path. This rule matches mail that contains one or more Received: lines that cannot be parsed to extract this information.
Note that this is an "informational" rule -- in other words, it is not intended to differentiate spam from nonspam, and should not have a significant score.
I would probably raise this rule up to .75 or 1.0, since it should not have a significant score. Then listen for users that might complain about valid emails being marked. You could always search through the Kerio mail store for .em files that contain UNPARSEABLE_RELAY and see if it is mostly spams or goods emails as well.
|
|
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #124332 is a reply to message #114168] |
Tue, 22 September 2015 04:29 |
sjwanta
Messages: 45
Registered: April 2012
|
|
|
|
Thank you to everyone who has contributed, this thread is a huge help. Does anyone know why Kerio's default local.cf file contains so many 0 scores? For example:
score __RCVD_IN_NJABL 0
score RCVD_IN_NJABL_RELAY 0
score RCVD_IN_NJABL_SPAM 0
score RCVD_IN_NJABL_MULTI 0
score RCVD_IN_NJABL_CGI 0
score RCVD_IN_NJABL_PROXY 0
score __RCVD_IN_SORBS 0
score RCVD_IN_SORBS_HTTP 0
score RCVD_IN_SORBS_SOCKS 0
score RCVD_IN_SORBS_MISC 0
score RCVD_IN_SORBS_SMTP 0
score RCVD_IN_SORBS_WEB 0
score RCVD_IN_SORBS_BLOCK 0
score RCVD_IN_SORBS_ZOMBIE 0 |
|
|
|
| Re: Custom SpamAssassin scores and rules [message #124334 is a reply to message #124332] |
Tue, 22 September 2015 07:08 |
MarkK
Messages: 342
Registered: April 2007
|
|
|
|
Well, here is my answer. I don't know and it doesn't really matter. The reason I say that is I won't worry about what is in the local.cf file. Though that is the typical file that you would make custom changes to, with Kerio that file is overwritten with every upgrade. Because of that, you should leave it alone, and create your own custom named .CF file.
Also, in all of the spam headers that I have read, I have never seen any of those get a hit. Even though they score a 0, they would still show in the spam headers as a hit. So to me personal, those are unused items.
Spend some time with the spams you get, and create scores for those. That has work for 2 different SA installs that I manage.
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #124354 is a reply to message #124334] |
Tue, 22 September 2015 16:06 |
Jonn
Messages: 23
Registered: June 2015
|
|
|
|
Well it does matter. I asked why to find out why I was getting so many spam messages in my our inboxes.
The spamassassin (SA) that is in KC is very hacked up. They removed a lot of modules from the basic build and built their own. Most of the items set to zero in local.cf file have been removed. I try and change them to something else you will likely cause SA to crash during spam filtering.
Kerio basically expected that you create custom rules for everything. This was really stupid and has caused a lot users problems. I custom build SA for our old mail server and we had 99% of spam going into our junk email folders. The second big problem with what they did was SA has not been updated. The version they are using has been discontinued and is no longer supported. It is not even getting security patches. I asked if they are ever going to upgrade SA and I was told maybe in a future release.
So basically you have 2 options. 1 - create hundreds of custom rules in KC or 2 - use a 3rd party spam filter to tag email. Kerio support told me to use a 3rd party spam service. 
The way I fixed this problem was setup another server to filter spam with and then forward the emails on to KC. I used mailcleaner but any goo spam filter that adds headers can be used and KC can use this to move messages to the junk email box.
I posted the complete setup in another thread if anyone is interested.
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #124372 is a reply to message #124354] |
Tue, 22 September 2015 20:40 |
MarkK
Messages: 342
Registered: April 2007
|
|
|
|
Can't argue that Kerio has not fully implemented all of Spam Assassin, but what is there can be very effective. One of the advantages of using a third party spam filter (paid or not) is that you have someone else updating the rules, hopefully.
Personally, I have had very good success with what is built in. I got sick of the spams coming through to my end users, edited some of the spam assassin scores that were getting hits, still didn't like the amount of spams coming through, and looking at the individual spams started to see the patterns in them. So I looked up how to write some simple rules of my own, and spam results went from disappointing to extremely good. My current spam stats (10 months worth) are:
803904 Messages Rec'd
720744 Messages Chk'd
134207 Spams detected (tagged)
492573 Spams detected (rejected)
2754 Messages marked by users as spam
319 Messages marked by users as not spam
Those are levels that we can live with. I'm not shooting to catch ALL spams, for fear of starting to catch too many good emails.
The current custom SA rule file I am using is attached. Put it in the .MailServer\plugins\spamserver\spamassassin\rules folder,
go in to Admin Panel > Configuration > Spam Filter > SpamAssassin tab
Uncheck the box "Check every incoming message in Spam URI Realtime Blocklist (SURBL) database" and click APPLY
Check the box "Check every incoming message in Spam URI Realtime Blocklist (SURBL) database" and click APPLY
The new rules are now being used, all without having to restart your server.
Don't like the results? Remove the file and do the steps above again.
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #124380 is a reply to message #124372] |
Tue, 22 September 2015 22:16 |
sjwanta
Messages: 45
Registered: April 2012
|
|
|
|
MarkK:
Thanks for your sharing your rules and your earlier tutorial. It has been a huge help in my on-going spam battle. I agree with Jonn that Kerio has a long way to go in keeping their SA implementation current, and that is one of their top underperforming features. But, MarkK's rules are helpful, and certainly bridge at least some of the gap between full and current SA installs and the version Kerio builds into Connect.
In the spirit of sharing, attached are my tweaks to MarkK's rules that have been working so-so for me. The two biggest differences (apart from deleting a rule and a few scoring tweaks) are the addition of score RDNS_NONE 1.8 and even more so score UNPARSEABLE_RELAY 2.5. The largest volume of spam that I have been seeing lately triggers the UNPARSEABLE_RELAY test, but Kerio's internal score (0.001) is too low to significantly affect the spam rating. Custom SA rules are tricky in Kerio because we don't have access to SA's rule validation tools.
I am also considering building a separate SA server to run on my VMWare cluster alongside Kerio and relay messages through the SA server to Kerio.
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #124385 is a reply to message #124380] |
Tue, 22 September 2015 23:31 |
MarkK
Messages: 342
Registered: April 2007
|
|
|
|
This is where everyone's mileage may vary with the rules. Apparently my rule modifications have given me better results than others have had. It would be nice if Kerio kept SA up to date, or gave us the ability to use something like ASSP spam filter (SourceForge Anti-Spam SMTP Proxy Server) as a built in replacement. In the past, I have installed the latest version of Spam Assassin base rules in Kerio, but when you update Kerio, they get written over by the older Kerio implemented rules.
The rules you marked, RDNS_NONE 1.8 and even more so score UNPARSEABLE_RELAY 2.5, for me were something I would see on good emails and didn't want to give them too high of a score for fear of mistagging good emails.
Kerio doesn't provide all of the SA tools normally supplied, but we do have full access to the rule files used. Search the .cf for the rule name and you will find the match rule. Problem is that they are sometimes hard to understand if you don't know regex formulas.
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #125895 is a reply to message #114168] |
Mon, 23 November 2015 11:49 |
gseum
Messages: 18
Registered: June 2015
Location: Germany
|
|
|
|
Hello Markk,
thanks for your rules! Working very good - nearly perfekt.
I use the version of zMyRules-20151031.cf
I have seen a rule, that is problematic, because your are giving some common (european) domain-endings (some of countries) a high spam factor:
header lcl_DOMAINS_50 From =~ /(\.adult>|\.asia>|\.bid>|\.careers>|\.cash>|\.chat>|\.cheap >|\.click>|\.club>|\.click>|\.cn>|\.date>|\.dating>|\.de>|\.deals>|\.diet>|\.discount>|\.dk>|\.download>|\.eu>|\.exposed>|\.fi>|\.fyi>|\.gay>|\.help>|\.hk>|\.in>|\.in\.net>|\.info>|\.li>|\.lv>|\.link>|\.me>|\.party >|\.poker>|\.porn>|\.pw>|\.racing>|\.review>|\.ro>|\.ru>|\.sale>|\.science>|\.sex>|\.sexy>|\.singles>|\.sucks>|\.tattoo >|\.tk>|\.th>|\.top>|\.tw>|\.ua>|\.uk>|\.uno>|\.us>|\.uy>|\.webcam>|\.whoswho>|\.win>|\.work >|\.wtf>|\.xxx>|\.xyz>)/i
I will remove these entries for my use - especially the de, eu, info and uk endings are "normal" here in europe... 
Thanks!!!
Greetings from Germany
gseum
|
|
|
|
| Re: Custom SpamAssassin scores and rules [message #125903 is a reply to message #125895] |
Mon, 23 November 2015 15:36 |
MarkK
Messages: 342
Registered: April 2007
|
|
|
|
You are welcome. Looking at and changing the rules for your particular install is exactly what you needed to do. I'm in the U.S., and we are a small local company, so those domain extensions have proven to be spam to me, where they are the norm to you.
Just monitor what spam slips through and decided if you need to modify / add to the rules to make it work even better for you.
|
|
|
|
) 1 Vote
Members
Search
Help
Register
Login
Home
