F.P. acknowledgement from Wizcrafts

This evening I received a legitimate email with a .pdf attachment containing course information from a new sender. It was flagged and marked for manual deletion by my base64 filter. The filesize made it obvious that it contained a large attachment, as did the body text, so I investigated why it got flagged in the first place. It turned out that I had failed to account for attachments in the base64 conditions! This is one of those old filters that I have been using forever. It needed fixed and got it today.

First, I removed the Delete action from the GUI options, then I created a Regular Expression to allow the content to be declared either "x" OR "y," whereas before it was only "y."

Furthermore, in order to see the line of code containing the terms "x" OR "y" I had to increase the scanning throttle by one more line, to 251! Normally this wouldn't be necessary as most emails with attachments don't have very much source code before the Content-Disposition lines, but this message was created using Microsoft Word 11 and you won't believe how much crap was added by the program! The visible body contains only 11 lines of text and several blank lines, but the Word program added 191 lines of garbage codes!

Anyway, the Base64 filter still detects Base64 encoded spam, but it is a little less likely to flag a legitimate attachment as such and will no longer pre-mark it for deletion. It will just be an information filter, which I have set to show "Base64" in orange, in my MWP, which is now set to scan 251 lines of code (minimum).

My updated MWP filters are available here

This is just a technical note.

If you choose to use one of my filters.txt by dragging it into your MailWasher Pro profile, be aware that after you run the program, then close it, all of my comments in the head of the file will be replaced with these statements from Firetrust:

// MailWasher Pro filter settings
//
// If you make changes to this file while MailWasher Pro is running,
// the changes will be overwritten when MailWasher Pro is closed.

I don't know of any workaround at this time. I do recall reading something about inserting permanent comments in filter rules, from the dark past, and will try to locate the sample, which must live somewhere.

In the last three days I have updated 6 filters to reduce false positives, or improve detection rates. The altered filters are as follows:

HTML Spam Tricks
Postcard Trojan Scam
RX Spam
Pills Spam
Misspelled Drugs
Stock Spam #3

I will continue to try to identify and fix rules that cause false positives, without totally crippling a filter. I am also identifying Regular Expressions that cause heavy CPU usage and am trying to touch them up for better processing, without losing their specific targeting (a challenge). Some of my Regular Expressions are real spam killers and tend to remain so across many variations in spam techniques. Some cut to the core of their techniques.

As spammers discard old, failing methods some of my filters will become redundant, and will be moved out of the current filters set, into the larger, master set, or into the bit bucket. However, they won't be far away if needed again!

FYI: Did you know that all image or attachment spam is based on what's known as base64 encoding? With that in mind, all I have to say to the spammers is: All your Base64 are belong to us! Make your time!

Since I last checked in, on Aug 16, I have been busy - quietly refining my MailWasher filters. Some filters have been streamlined to reduce the burden they placed on the program, while others were updated to detect current spam phrases and misspellings. Filters were added to detect certain PayPal phishing scams and a few filters have been (temporarily) deactivated.

Most interesting is the reappearance of the "Postcard" scams, sent from Storm Worm infected PCs. These had disappeared for a couple of weeks and are back with a few alterations meant to fool the readers into clicking on the links, which are no longer numeric, to the eye. However, reading the source code, or hovering over the link without clicking on it does reveal that they still point to numeric IPs on infected computers, which host the Storm Worm. Of all the spam I have seen in the past two months the Storm Worm is without a doubt the most serious threat to our security. The number of computers infected with this malware is astronomical and the threat they pose, as a Zombie Army, is incredible. We may be their next attack target. Because the authors of the Storm Worm are changing the subjects and body text every other day, and hiding the numeric links in phoney website wrappers, my rulesets to identify them have been reduced and refined.

My downloadable filters list the changes I have made over the last few days. Sometimes I revise a rule more than once in the same day, so look at the time stamps, or use ChangeDetection to monitor for updates.

cool set of filters

i use these for image spam..

[enabled],"DEL GIF IMAGE SPAM (RegExpr)","DEL GIF IMAGE SPAM (RegExpr)",16711680,AND,Delete,Automatic,Body,contains,"Content-Transfer-Encoding: base64",Body,containsRE,Content-Type:.+(gif)

replace gif for other types of atachments what spammers use..

That might auto-delete a lot of my good mail since folks send me stuff with gifs in it all the time.

When sharing filters it is much better to set them to mark but not auto-delete just in case someone has a problem with them, that way they won't have their good mail auto-deleted.

@spatieman - if you go back to the 1st post on the previous page and read forward, you will note a fair amount of discussion about the auto-delete option and concerns such as that raised by Stan. Auto-delete is a bad idea! Unless you know exactly what that filter is doing, and what type email it will match on, don't use it. I agree with Stan - haphazardly auto-deleting emails, like your filter would result in many legitimate emails being deleted just because it contains a gif. Half the newsletters I subscript to use gifs in their logos! Many free newsletters and announcements are paid for by advertisements which use gifs. It might be better to let the remaining spam tools pick up those emails - they seem to do a pretty good job.

_________________
Bill, AFE7Ret
Freedom is NOT Free!

It's been a couple of weeks since I last checked in, but I haven't been idle with my filters. It seems that spam tricks have gone full circle, having started as plain text ads, then base64, to obfuscated-broken text, then html tricks to break up words, then image spam, then attachments with image spam and now back to plain text again. Every few days there is something different in the subjects and body text, sometimes in plain English, other times obfuscated-broken text.

While it is becoming harder to keep up with these rapidly evolving spam tricks I am doing the best I can, leaving non-filtered spam to the DNS Blacklist, or, hopefully, the learning filter. Unfortunately, some of the latest tricks are even eluding the learning filter. Fortunately, I have identified most of those and created rules to detect them.

A good example of a couple of odd spam filters are the one word subject and the "DW" spammer. I won't go into details about the latter, but take my word for it that anything flagged by the DW filter is 99.99% likely Chinese spam. I placed it down stream to let the more specific filters do their thing. When those filters failed to ID the type of spam the DW caught it.

The Storm Trojan's authors are very tricky people, with an intimate knowledge of North American habits. Not bad for Ruskies! The newest Storm runs advertise a free NFL Tracker program for pro football fans to "track" their favorite teams and players. These scam messages are in plain text, with a link to a domain URL, not a numeric IP. This is accomplished by parking the domain on ESTDomains (Known spam house for among others - Russian spam gangs), loading up to 13 name servers, then using free DNS services to locate that Zombie computer, on a cable connection. If the victim is deceived into clicking on any of the links on destination web page, they infect themselves with the Storm Trojan. I have included a new filter to detect and flag these scams.

After much discussion in this thread it has been decided that the filters.txt that are for general use will only flag and/or mark spam for deletion. The MailWasher user will only have to sort by status, remove any false positives, then hit the Process button to delete the marked messages from the mail server. As you gain confidence in their accuracy you may wish to begin hiding known types of spam, for manual deletion upon Processing. Of course, my personal filters are set to Murder-Death-Kill, but I've been doing this for about 6 years now.

I'm having a bad day. I forgot to include a link to my MailWasher Pro filters page. I also post fairly often about the newest Storm Trojan tactics and my work on spam filters, on my Blog.

Is anybody else having problems copying and pasting my filters from my website? I compose them in NoteTab Pro and save as ASCII text. They open perfectly in Notepad. I don't know why there would be a need to resave them in UTF-8, but would appreciate any feedback on this matter.

A lot has been said about the nature of my custom MailWasher filters; how they work, how much resources they consume, how unsafe hiding or auto deleting can be and things of this nature. What hasn't been mentioned too often is the fact that, as spam types and tricks change, you need to readjust the order of the various filters to get a match on the newest types of spam as soon as possible. I haven't been automatically doing this with my online versions of my filters, but may start doing so soon. At any rate, whether I change the order of the online rules or not, you should move filters upward to match the types of spam you see most often at any given time. That may be monthly, weekly, or every other day. It really depends on how much the filters slow down your scanning of incoming messages.

If the named spam filters you are matching are way down the list, then MailWasher may have to go through many irrelevant rules until the right one is matched. This uses up more processing time than it would if the appropriate filter was moved up. What I try to do is have all named spam filters ahead of general spam types. General spam filters include the one word subject, the digits and consonants in the From address, HTML tricks and various blocked country rules.

Another thing I find is that sometimes the wrong filter will match a spam email, because of variations in the message. When that occurs I move the correct filter above the one that flagged it with the wrong name.

Admittedly, this is slicing hairs, but moving the most common named filters up the list does save processing time, if a spam or scam is matched.

I update my online MailWasher Pro filters quite often, so, if you are using them, be sure to check for updates every day or two. You can use Change Detection to monitor the files, for free, and notify you by email whenever they are updated (daily notices only). There is a button on the web page to sign up. You can also target the individual URLs of my filters, if you wish.