CastleCops, Internet Crime Fighters
Need help? Click here to register for free! Absolutely zero advertisements on this site!

$9736.22 of $21422.68
left sidedonated so farneed $11686.46 donated to reach our goalright side, our goal
Help CastleCops serve the community on new servers, Donate Here to reach our goal.

Donation/Premium
spacer
block bottom
Security Central
spacer
· Home
· PIRT/Fried Phish
· MIRT
· SIRT
· Deutsch
· Wiki
· Newsletter
· O16/ActiveX
· CLSID List
· Contest2007
· Downloads
· Feedback (send)
· Forums
· HijackThis
· Hijacktrend
· LSPs
· My Downloads
· O18
· O20
· O21
· O22
· O23
· O9
· Premium
· Private Messages
· Proxomitron
· Reviews
· Search
· StartupList
· Stories Archive
· Submit News
· WsIRT
· Your Account
· Acceptable Use Policy
block bottom
Survey
spacer
Was 2007 a good year?

Yes it was a wonderful year
Yes, but there is always room for improvement
Status quo
It was a challenge
Other (leave comment)



Results
Polls

Votes: 937
Comments: 25
block bottom
spacer spacer

The Unofficial Homeland Security Filter
Goto page 1, 2, 3  Next
 
Post new topic   Reply to topic       All -> FavForums -> Mailwasher - Troubleshooting / General [del.icio.us!] [digg it!] [reddit!]
View previous topic :: View next topic  
Author Message
dallas7

Sergeant
Sergeant


Joined: Oct 11, 2003
Posts: 79


PostPosted: Wed Oct 29, 2003 7:25 pm    Post subject: The Unofficial Homeland Security Filter
Reply with quote

My previous anti-spam tool was NovaSoft's SpamKiller and in there was a screen in which country codes could be selected for filtering. After checking off all countries except the USA, I was impressed at how much spam was filtered out.

Not having a similar option choice in MailWasher, I created the following filter based on Gary Partain's model:

[enabled],"Foreign Routing","Foreign Routing",33023,OR,Delete,EntireHeader,containsRE,(?m)^Received:.+ (\.af|\.al|\.dz|\.as|\.ad|\.ao|\.ai|\.ag|\.ar|\.am|\.aw|\.au|\.at|\.az|\.bs| \.bh|\.bd|\.bb|\.by|\.be|\.bz|\.bj|\.bm|\.bt|\.bo|\.ba|\.bw|\.bv|\.br|\.io| \.bn|\.bg|\.bf|\.bi|\.kh|\.cm|\.ca|\.cv|\.ky|\.cf|\.td|\.cl|\.cn|\.cx|\.cc|\.co| \.km|\.cg|\.cd|\.ck\.cr|\.ci|\.hr|\.cu|\.cy|\.cz|\.dk|\.dj|\.dm|\.do|\.tp| \.ec|\.eg|\.sv|\.gq|\.er|\.ee|\.et|\.fk|\.fo|\.fj|\.fi|\.fr|\.fx|\.gf|\.pf|\.tf| \.ga|\.gm|\.ge|\.de|\.gh|\.gi|\.gr|\.gl|\.gd|\.gp|\.gu|\.gt|\.gg|\.gn|\.gw| \.gy|\.ht|\.hm|\.va|\.hn|\.hk|\.hu\.is|\.in|\.id|\.int|\.ir|\.iq|\.im|\.il|\.it| \.jm|\.jp|\.je|\.jo|\.kz|\.ke|\.ki|\.kp|\.kr|\.kw|\.kg|\.la|\.lv|\.lb|\.ls|\.lr| \.ly|\.li|\.lt|\.lu|\.mo|\.mk|\.mg|\.mw|\.my|\.mv|\.ml|\.mt|\.mh|\.mq| \.mr|\.mu|\.yt|\.mx|\.fm|\.md|\.mc|\.mn|\.ms|\.ma|\.mz|\.mm|\.na| \.nr|\.np|\.nl\.an|\.nc|\.nz|\.ni|\.ne|\.ng|\.nu|\.nf|\.mp|\.no|\.om|\.pk| \.pw|\.pa|\.pg|\.py|\.pe|\.ph|\.pn|\.pl|\.pt|\.pr|\.qa|\.re|\.ro|\.ru|\.rw| \.kn|\.lc|\.vc|\.ws|\.sm|\.st|\.sa|\.sn|\.sc|\.sl|\.sg|\.sk|\.si|\.sb|\.so|\.za| \.gs|\.es|\.lk|\.sh|\.pm|\.sd|\.sr|\.sj\.sz|\.se|\.ch|\.sy|\.tw|\.tj|\.tz|\.th| \.tg|\.tk|\.to|\.tt|\.tn|\.tr|\.tm|\.tc|\.tv|\.ug|\.ua|\.ae|\.gb|\.uk|\.uy|\.uz| \.vu|\.ve|\.vn|\.vg|\.wf|\.eh|\.ye|\yu|\.zm|\.zw)[^\.\w]

OCT 31: SPACES WERE ADDED TO ALLOW WORD WRAP WITHIN THIS FORUM. USE THE FILTER(S) BELOW AS MINE WAS CORRECTED AND OTHERS HAVE BEEN SUGGESTED AS ALTERNATIVES.

This one filters everything except the USA and the Antarctic. I'd hate to miss the first spam routed through the Antartic!

You can find a country list in http://www.w5hq.com/MailWasher/MailWasherFilters.txt
and remove any countries you need unfiltered.

This filter along with a few good DNS blacklist servers in MWP catch over 95% of the 150-200 spams I get every day.



Last edited by dallas7 on Fri Oct 31, 2003 7:08 pm, edited 2 times in total
Back to top
View users profile Send private message
Jarron

Lieutenant
Lieutenant
Premium Member

Joined: Oct 16, 2003
Posts: 163

Premium

PostPosted: Wed Oct 29, 2003 9:25 pm    Post subject:
Reply with quote

Dallas,

I had just downloaded the country codes from Gary's filters and was fixin' to work with them when I happened to find your post. Talk about timing, you saved me a lot of work! Thanks!

Working with your filter I found where you seem to have left the "|" between hu\is, ck\cr, nl\an, and sj\sz.

Here's the correction I made (I hope they're corrections, I don't quite trust myself working with filters).

[enabled],"Foreign Routing","Foreign Routing",33023,OR,Delete,EntireHeader,containsRE,(?m)^Received:.+
(\.af|\.al|\.dz|\.as|\.ad|\.ao|\.ai|\.ag|\.ar|\.am|\.aw|\.au|\.at|\.az|\.bs|
\.bh|\.bd|\.bb|\.by|\.be|\.bz|\.bj|\.bm|\.bt|\.bo|\.ba|\.bw|\.bv|\.br|\.io|
\.bn|\.bg|\.bf|\.bi|\.kh|\.cm|\.ca|\.cv|\.ky|\.cf|\.td|\.cl|\.cn|\.cx|\.cc|\.co|
\.km|\.cg|\.cd|\.ck|\.cr|\.ci|\.hr|\.cu|\.cy|\.cz|\.dk|\.dj|\.dm|\.do|\.tp|
\.ec|\.eg|\.sv|\.gq|\.er|\.ee|\.et|\.fk|\.fo|\.fj|\.fi|\.fr|\.fx|\.gf|\.pf|\.tf|
\.ga|\.gm|\.ge|\.de|\.gh|\.gi|\.gr|\.gl|\.gd|\.gp|\.gu|\.gt|\.gg|\.gn|\.gw|
\.gy|\.ht|\.hm|\.va|\.hn|\.hk|\.hu|\.is|\.in|\.id|\.int|\.ir|\.iq|\.im|\.il|\.it|
\.jm|\.jp|\.je|\.jo|\.kz|\.ke|\.ki|\.kp|\.kr|\.kw|\.kg|\.la|\.lv|\.lb|\.ls|\.lr|
\.ly|\.li|\.lt|\.lu|\.mo|\.mk|\.mg|\.mw|\.my|\.mv|\.ml|\.mt|\.mh|\.mq|
\.mr|\.mu|\.yt|\.mx|\.fm|\.md|\.mc|\.mn|\.ms|\.ma|\.mz|\.mm|\.na|
\.nr|\.np|\.nl|\.an|\.nc|\.nz|\.ni|\.ne|\.ng|\.nu|\.nf|\.mp|\.no|\.om|\.pk|
\.pw|\.pa|\.pg|\.py|\.pe|\.ph|\.pn|\.pl|\.pt|\.pr|\.qa|\.re|\.ro|\.ru|\.rw|
\.kn|\.lc|\.vc|\.ws|\.sm|\.st|\.sa|\.sn|\.sc|\.sl|\.sg|\.sk|\.si|\.sb|\.so|\.za|
\.gs|\.es|\.lk|\.sh|\.pm|\.sd|\.sr|\.sj|\.sz|\.se|\.ch|\.sy|\.tw|\.tj|\.tz|\.th|
\.tg|\.tk|\.to|\.tt|\.tn|\.tr|\.tm|\.tc|\.tv|\.ug|\.ua|\.ae|\.gb|\.uk|\.uy|\.uz|
\.vu|\.ve|\.vn|\.vg|\.wf|\.eh|\.ye|\yu|\.zm|\.zw)[^\.\w]

Thanks again,
Jarron

Edited for forum formatting purposes. Remove CR's before pasting into filter list (text should be one line). Not responsible for false positives or other problems; if you lose out on $1,000,000.00 because you didn't get your letter from Nigeria it's not my fault.



Last edited by Jarron on Fri Oct 31, 2003 7:17 am, edited 4 times in total
Back to top
View users profile Send private message
dallas7

Sergeant
Sergeant


Joined: Oct 11, 2003
Posts: 79


PostPosted: Wed Oct 29, 2003 11:41 pm    Post subject:
Reply with quote

It worked. I was wondering how I could get this filter error corrected and I thought, "Post it in the forum!" Wink

Really, thanks for catching those. Looks like you got all of 'em. Must've happened those four times my eyeballs popped out when I was working on it.

I hope you find it as effective as I do.

Back to top
View users profile Send private message
TalonTSi

Corporal
Corporal


Joined: Mar 16, 2003
Posts: 55
Location: Canada

PostPosted: Thu Oct 30, 2003 12:11 am    Post subject:
Reply with quote

Instead of a huge filter listing all the country codes you want to exclude, could you not instead add a blacklist entry like *@*.?? then write a filter to override the blacklist for the countries you'd like to receive email from (ie *@*.us or whatever)?


_________________
--Darren.
Back to top
View users profile Send private message Visit posters website
dallas7

Sergeant
Sergeant


Joined: Oct 11, 2003
Posts: 79


PostPosted: Thu Oct 30, 2003 1:12 am    Post subject:
Reply with quote

Quote:
Instead of a huge filter listing all the country codes you want to exclude, could you not instead...


I have no clue. I know nothing of regular expressions and have no interest in learning about them.

Like I said above, I just expanded on Mr. Partain's filter.

If there's a better way to do this, it'll never pop out of my head!

Back to top
View users profile Send private message
IP: 66.44.*.*

Guest






PostPosted: Thu Oct 30, 2003 1:26 am    Post subject:
Reply with quote

I was looking at your regex and I just wanted to make a couple suggestions.

The first iterator in the expression might be better as a non-greedy. Also, it might actually help to specify the "from" that is normally in the line.

Instead of:
^Received:.+(\.af|\.al|\.dz|\.as|\.ad|\.ao|\.ai|\.ag|\.ar|\.am....

change it to:

^Received: from.+?(\.af|\.al|\.dz|\.as|\.ad|\.ao|\.ai|\.ag|\.ar|\.am....


You might be able to speed it up, and shorten the expression if you remove the "\." from before each countrycode and place it before the Mega-OR statement.

Instead of:

^Received: from.+?(\.af|\.al|\.dz|\.as|\.ad|\.ao|\.ai|\.ag|\.ar|\.am....


change it to:

^Received: from.+?(af|al|dz|as|ad|ao|ai|ag|ar|am....

Once that is done, you might be able to shorten it some more by grouping all the first charactors together:

Change:

^Received: from.+?(a[defgilmnorstuwz]|b[abdefghijmnorstvwyz]|c[acdfghiklmnoruvxyz]|d[ejkmoz]|e[ceghrst]|f[ijkmorx]|g[abdefghilmqrstuwy]|h[kmnrtu]|i[dlmnoqrst]|j[emop]|k[eghimnprwyz]|l[abcikrstuvy]|m[acdghklmnopqrstuvwxyz]|n[acefgilopruz]|om|p[aefghklmnrtwy]|qa|r[eouw]|s[abcdeghijklmnortvyz]|t[cdfghjkmnoprtvwz]|u[agkyz]|v[acegnu]|ws|y[etu]|z[amw])[^\.\w]


Finally, at the end of the regex the terminator seems to be malformed.

[^\.\w] is telling the Regex interpreter that any charactor that follows the two letter country designator other than "\", . , or an alphanumeric charactor will make the result true.

It might be a good practice to get into now to skip the "\" before the dot when enclosed in the square brackets "[]"

Change it to read [^.\w]

Back to top
dallas7

Sergeant
Sergeant


Joined: Oct 11, 2003
Posts: 79


PostPosted: Thu Oct 30, 2003 4:51 am    Post subject:
Reply with quote

Shocked

Back to top
View users profile Send private message
Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16506

Forums Admin Moderators MVP Premium SRT Team CC Committee Team F@H

PostPosted: Thu Oct 30, 2003 6:46 am    Post subject:
Reply with quote

Homeland Security indeed. The term "isolationist" comes to mind......

BTW Dallas, would it be possible to edit your posts and insert a space every once in a while in your rather longish lines? These forums need some sort of autowrap after 200 or so characters.......

Back to top
View users profile Send private message
IP: 210.49.*.*

Guest






PostPosted: Thu Oct 30, 2003 9:03 am    Post subject:
Reply with quote

Dallas, I just emailed you the solution to do exactly the same thing, but 1/10 the size.

You may not have received it though, I'm in Australia. Oh well.

Back to top
TimeGhost

Captain
Captain


Joined: Apr 11, 2003
Posts: 747
Location: USA
Team F@H

PostPosted: Thu Oct 30, 2003 3:19 pm    Post subject:
Reply with quote

I agree with the suggestion to use non-greedy wildcard.
Also I think the expression should be terminated with \s.
Otherwise, you'll get false positives on servers with dashes and numbers in them:

Received: from mx.af-lala.com

So just replace [^.\w] with \s.

BTW, I appreciate the sarcastic title of the post.
Even if it wasn't intended to be so.

Back to top
View users profile Send private message
IP: 66.44.*.*

Guest






PostPosted: Thu Oct 30, 2003 5:17 pm    Post subject:
Reply with quote

TimeGhost,

I had considered that, but was concerned that somethimes there is no space between the end of the domain and a bracket.

This is usually indicative of a malformed line, but if you write the regex to insist on "\s" as a termintor you will miss those malformed Recieved lines.

Might as well use the filter to catch a few forgeries here and there. What do you think??

Back to top
IP: 66.44.*.*

Guest






PostPosted: Thu Oct 30, 2003 5:23 pm    Post subject:
Reply with quote

TimeGhost wrote:
Received: from mx.af-lala.com


oooops....missed that on your post before.


I see exactly what you mean now....

perhaps this would work:

[^.\w-] The \w already takes care of numbers

Back to top
dallas7

Sergeant
Sergeant


Joined: Oct 11, 2003
Posts: 79


PostPosted: Thu Oct 30, 2003 7:45 pm    Post subject:
Reply with quote

Quote:
...I just emailed you the solution to do exactly the same thing,
but 1/10 the size. You may not have received it though, I'm in Australia.



Thank you for appealing to my isolationism by sending it directly to me.
I haven't received it as of yet (OCT 30, 1900 UTC). However, the
globalist in me requests you post it here for everyone.


Quote:
Oh well.



Tsk. I expect more determination from one of our buds Down Under!

Quote:
BTW, I appreciate the sarcastic title of the post.
Even if it wasn't intended to be so.



It was absolutley tongue-in-cheek. Had I wanted it to be offensive,
I would have hidden behind a guest login and stated the purpose as
effective at filtering out those damn globalists.

Quote:
Homeland Security indeed. The term "isolationist" comes to mind......



I was wondering how long it would take to see a comment about my
choice of subject wording! Actually, I think it's smack dab in the middle
between "isolationist" and "globalist." And that's all the attention I'm
gonna pay to this.

Quote:
...would it be possible to edit your posts and insert a space every once in a while...



Do you mean a line feed? And I didn't actually notice there was a
fomatting issue until I re-read my posting the next day. Preview reveals nothing amiss.
This could have happened since I composed in a text editor, since I run a spell checker in it, and then pasted into the edit
window here. Although I do this ALOT in several other forums without
any problems (I've got over 4,000 posts in one of my favorites). I don't
know why my 14th posting here got boned up; maybe it was the rather
long string of filter text. It won't happen again. I hope...

AND NOW BACK TO A DISCUSSION ABOUT THE SPAM FILTER:

I would prefer to leave Gary's filtering structure as is. First, it works.
Second, I didn't notice any performance hits. Third, should one day I
observe I'm filtering tons of really really cool email from Djibouti, I can
simply go in and zap "dj" and revel in the comfort that the rest of the filter will still work.
Recall, please, one of my previous posts: I'm regular
expression illiterate. I am pleased, though, my posting became such an
inspiration to all you great folks who aren't!



Last edited by dallas7 on Fri Oct 31, 2003 7:28 pm, edited 1 time in total
Back to top
View users profile Send private message
IP: 66.44.*.*

Guest






PostPosted: Thu Oct 30, 2003 10:38 pm    Post subject:
Reply with quote

Dallas7 wrote:
Do you mean a line feed? And I didn't actually notice there was a
fomatting issue until I re-read my posting the next day. Preview reveals nothing amiss.


A space will do.....

The word wrapping algorythm does not know where to wrap without some kind of 'whitespace'.

If you have a long line of text, a space every 50-60 charactors will be sufficient to trigger the word wrapping.

Back to top
dallas7

Guest
IP: 130.13.*.*






PostPosted: Thu Oct 30, 2003 11:51 pm    Post subject:
Reply with quote

Noted. Thanks for the heads-up on that.

Back to top
Display posts from previous:   
Post new topic   Reply to topic       All -> FavForums -> Mailwasher - Troubleshooting / General All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Quick Reply:
Username: 

Quote the last message
Attach signature (signatures can be changed in profile)
 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001 phpBB Group
spacer spacer