|
Donation/Premium |
|
 |
|
|
|
|
|
|
|
Survey |
|
|
|
|
|
|
|
|
|
|
|
| View previous topic :: View next topic |
| Author |
Message |
UserFriendly
Captain
Joined: Apr 18, 2003
Posts: 348
|
 Posted: Thu Nov 20, 2003 10:30 pm Post subject: Filter This |
|
|
Any ideas on how to filter this and others like it without getting false positives? I could filter on the Yahoo! bulk mail header, but that'll only work on my Yahoo! account.
-Begin Spam-
X-Apparently-To: userfriendlyuser@yahoo.co.uk via 217.12.12.55; Thu, 20 Nov 2003 21:59:15 +0000
X-YahooFilteredBulk: 61.97.237.10
Return-Path: <lourdes_s.lyonpo@toberemoved.net>
Received: from 61.97.237.10 (HELO toberemoved.com) (61.97.237.10)
by mta108.mail.ukl.yahoo.com with SMTP; Thu, 20 Nov 2003 21:59:09 +0000
Message-ID: <245601c3af6e$67ecd430$36391a64@toberemoved.com>
From: "Lourdes S. Lyon" <lourdes_s.lyonpo@toberemoved.net>
To: usegane@yahoo.co.uk, userfriendlyuser@yahoo.co.uk
Subject: You're a Sweetie. Meet Me
Date: Thu, 20 Nov 2003 14:00:52 +0000
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0BA4_DE4248A8.0231D432"
This is a multi-part message in MIME format.
------=_NextPart_000_0BA4_DE4248A8.0231D432
Content-Type: text/plain
Content-Transfer-Encoding: 8bit
------=_NextPart_000_0BA4_DE4248A8.0231D432
Content-Type: text/html
Content-Transfer-Encoding: base64
PGh0bWw+DQoNCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1U
eXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9aXNvLTg4NTktMSI+
DQo8dGl0bGU+U3VwZXIgQmlnIENvY2tzPC90aXRsZT4NCjwvaGVhZD4NCg0K
PGJvZHkgYmdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCIgbGluaz0i
IzAwMDBGRiIgdmxpbms9IiMwMDAwRkYiIGFsaW5rPSIjMDAwMEZGIj4NCg0K
PHA+PGZvbnQgZmFjZT0iVGltZXMgTmV3IFJvbWFuIiBzaXplPSI0IiBjb2xv
cj0iI0ZGMDAwMCI+PHN0cm9uZz48ZW0+TXkgZ2lybGZyaWVuZCB0b2xkIG1l
DQphYm91dCB0aGlzIHBhcnR5PC9lbT48L3N0cm9uZz48L2ZvbnQ+PGJyPg0K
PGZvbnQgZmFjZT0iVmVyZGFuYSIgc2l6ZT0iMyI+d2hlcmUgdGhlIGd1eXMg
aGFkIGh1Z2UgPC9mb250Pjxmb250IGZhY2U9IlZlcmRhbmEiIHNpemU9IjQi
Pm1hc3NpdmUNCmNvY2tzPC9mb250Pjxmb250IGZhY2U9IlZlcmRhbmEiIHNp
emU9IjMiPi48L2ZvbnQ+PGJyPg0KPHN0cm9uZz48Zm9udCBmYWNlPSJWZXJk
YW5hIiBzaXplPSIzIiBjb2xvcj0iIzgwMDA4MCI+U2hlIGJldCBtZSBJIGNv
dWxkbid0IGhhbmRsZSBpdC48L2ZvbnQ+PC9zdHJvbmc+PGJyPg0KPGZvbnQg
ZmFjZT0iVmVyZGFuYSIgc2l6ZT0iNSI+SSBCTEVXIHRoZSBjb21wZXRpdGlv
biBhd2F5LjwvZm9udD48L3A+DQoNCjxwPjxmb250IHNpemU9IjYiPlNlZSBX
aGF0IEkgRGlkLCA8YSBocmVmPSJodHRwOi8vMjE5LjE1My4yLjEzOC90eWwv
YmlnY29ja3MvIj5DbGljayBIZXJlPC9hPjwvZm9udD48L3A+DQoNCjxwPiZu
YnNwOzwvcD4NCg0KPHA+Jm5ic3A7PC9wPg0KDQo8cD4mbmJzcDs8L3A+DQoN
CjxwPiZuYnNwOzwvcD4NCg0KPHA+Jm5ic3A7PC9wPg0KDQo8cD4mbmJzcDs8
L3A+DQoNCjxwIGFsaWduPSJjZW50ZXIiPjxmb250IGZhY2U9IlZlcmRhbmEi
IHNpemU9IjIiPjxhDQpocmVmPSJodHRwOi8vMjE5LjE1My4yLjEzOC90eWwv
cmVtb3ZlbWUvaW5kZXgucGhwIg0Kb25Nb3VzZU92ZXI9IndpbmRvdy5zdGF0
dXM9J1JlbW92ZSBNZSc7IHJldHVybiB0cnVlIj5DbGljayBIZXJlIFRvIEJl
IFJlbW92ZWQ8L2E+PC9mb250PjwvcD4NCjwvYm9keT4NCjwvaHRtbD4NCg==
------=_NextPart_000_0BA4_DE4248A8.0231D432--
-End Spam-
-UserFriendly
|
|
| Back to top |
|
 |
gary
Lieutenant
Premium Member
Joined: Dec 22, 2002
Posts: 260
Location: Dallas/Ft. Worth, USA
|
| Posted: Thu Nov 20, 2003 11:04 pm Post subject: |
|
|
I tag anything that is base64 encoded, but you can still get false positives on that. Until MW has a way of allowing filters to access the decoded message body, this might be tough.
_________________
Gary
|
|
| Back to top |
|
|
AlphaCentauri
Guest
IP: 151.197.*.*
|
| Posted: Thu Nov 20, 2003 11:11 pm Post subject: |
|
|
I filter everything that is in MIME base 64. Very few legitimate emails use it. My interface window lists the reason each mail is filtered, so if it's base 64 I will glance to see if it's anyone I know, then go ahead and process it. '
Aren't worms concealed in this base 64 code, too?
|
|
| Back to top |
|
|
denn988
Guest
IP: 66.44.*.*
|
| Posted: Fri Nov 21, 2003 2:32 am Post subject: |
|
|
| AlphaCentauri wrote: | I filter everything that is in MIME base 64. Very few legitimate emails use it. My interface window lists the reason each mail is filtered, so if it's base 64 I will glance to see if it's anyone I know, then go ahead and process it. '
Aren't worms concealed in this base 64 code, too? |
As are bmps, gifs, jpgs, etc. Anything that is not a text file has to be encoded into a text format in order to be sent via e-mail. The preferred method is Base64.
The one thing that appears to make the example message above vulnerable to a filter strategy is the fact that the sender is encoding 'text/html' as Base64. That could be used in a filter because there is very little reason to encode text into base64 to send via e-mail, other than to try to obfuscate the message.
The filter for this would be in two parts...one for the header and one for the body. they would both contain the same Regex.
You might want to try this filter:
| Quote: |
[enabled],"TEXT BASE64","TEXT BASE64",16711680,OR,Delete,EntireHeader,containsRE,"^Content-Type: (text/html|text/plain)\s*?^Content-Transfer-Encoding: base64",Body,containsRE,"^Content-Type: (text/html|text/plain)\s*?^Content-Transfer-Encoding: base64" |
It should hit on anything that is sent as plain text or text/html AND is encoded in base64. I have not tested it, but have placed it into my filters for testing. Anyone else who would like to may feel free to test it for themselves.
I would also suggest (as I normally do)....do NOT AUTO-DELETE with this filter, or any other till you are sure that you will not get false positives using it.
|
|
| Back to top |
|
|
denn988
Guest
IP: 66.44.*.*
|
| Posted: Fri Nov 21, 2003 2:37 am Post subject: |
|
|
One more thing to consider....
This filter will only look as far into the body as MWP is set to download using the TOP command.
In a multipart message, it is possible that the base64 encoded text can be located past the initial downloaded portion of the message.
In other words....it may not trap a message that has base64 encoded text unless you choose to download significantly more than the first 20 lines.
|
|
| Back to top |
|
|
Ikeb
Special Response Team
Forums Admin
Joined: Apr 20, 2003
Posts: 16506
|
| Posted: Fri Nov 21, 2003 6:25 am Post subject: |
|
|
| denn988 wrote: |
| Quote: |
[enabled],"TEXT BASE64","TEXT BASE64",16711680,OR,Delete,EntireHeader,containsRE,"^Content-Type: (text/html|text/plain)\s*?^Content-Transfer-Encoding: base64",Body,containsRE,"^Content-Type: (text/html|text/plain)\s*?^Content-Transfer-Encoding: base64" |
|
Why limit the header encoding to just text?
Also doesn't the "\s*?" sequence accept only spaces between the two lines you're matching?
Thus wouldn't the following be "better"?
| Quote: | | [enabled],"BASE64 (TEXT only in Body)","BASE64 (TEXT only in Body)",16711680,OR,Delete,EntireHeader,containsRE,"^Content-Transfer-Encoding: base64",Body,containsRE,"^Content-Type: (text/html|text/plain).+^Content-Transfer-Encoding: base64" |
|
|
| Back to top |
|
|
UserFriendly
Captain
Joined: Apr 18, 2003
Posts: 348
|
| Posted: Fri Nov 21, 2003 8:49 am Post subject: |
|
|
| denn988 wrote: | | The one thing that appears to make the example message above vulnerable to a filter strategy is the fact that the sender is encoding 'text/html' as Base64. |
Sounds good. I've tested the filter and it definately catches these emails.
| Ikeb wrote: | | doesn't the "\s*?" sequence accept only spaces between the two lines you're matching? |
I think that's what you'd want - the dodgy bit of the email the filter is trying to catch is:
| Quote: | Content-Type: text/html
Content-Transfer-Encoding: base64 |
There's only whitespace between to two lines. If you set it to look for any characters between the lines it might catch some normal 'text/plain' followed by some legitimate base64 encoded attachments.
| denn988 wrote: | | it may not trap a message that has base64 encoded text unless you choose to download significantly more than the first 20 lines |
I'm using FirstAlert! so 200 minimum. I'm not too bothered about false negatives - the filter may produce false-positives, but it shouldn't be because not enough of the message was downloaded.
Thanks
-UserFriendly
|
|
| Back to top |
|
|
Ikeb
Special Response Team
Forums Admin
Joined: Apr 20, 2003
Posts: 16506
|
| Posted: Fri Nov 21, 2003 9:05 am Post subject: |
|
|
| UserFriendly wrote: | | Ikeb wrote: | | doesn't the "\s*?" sequence accept only spaces between the two lines you're matching? |
I think that's what you'd want - the dodgy bit of the email the filter is trying to catch is:
| Quote: | Content-Type: text/html
Content-Transfer-Encoding: base64 |
There's only whitespace between to two lines. If you set it to look for any characters between the lines it might catch some normal 'text/plain' followed by some legitimate base64 encoded attachments. |
Ah OK. The lines are always together, missed that. While I don't think it would trigger on just any 'text/plain' since the expression looks for a match of the complete line, the expression needs to prevent a match across a complete block of the message.
Just one question though? How does the "\s*?" expression get past the line feed?
|
|
| Back to top |
|
|
UserFriendly
Captain
Joined: Apr 18, 2003
Posts: 348
|
| Posted: Fri Nov 21, 2003 9:26 am Post subject: |
|
|
I wondered about that myself. I checked here http://anso.virtualave.net/RegExpE/tregexpr_syntax.htm: | Quote: | | \s any space (same as [ \t\n\r\f]) |
and | Quote: | \t tab (HT/TAB), same as \x09
\n newline (NL), same as \x0a
\r car.return (CR), same as \x0d
\f form feed (FF), same as \x0c |
So \s matches whitespace. Not just spaces.
-UserFriendly
|
|
| Back to top |
|
|
IP: 142.152.*.*
Guest
|
| Posted: Fri Nov 21, 2003 9:26 am Post subject: |
|
|
| denn988 wrote: | One more thing to consider....
This filter will only look as far into the body as MWP is set to download using the TOP command.
In a multipart message, it is possible that the base64 encoded text can be located past the initial downloaded portion of the message.
In other words....it may not trap a message that has base64 encoded text unless you choose to download significantly more than the first 20 lines. |
Just for reference, what does MWP consider a line?
Perry
|
|
| Back to top |
|
|
denn988
Guest
IP: 66.44.*.*
|
| Posted: Fri Nov 21, 2003 2:52 pm Post subject: |
|
|
| UserFriendly wrote: | So \s matches whitespace. Not just spaces.
|
You got it.....
| Perry wrote: | Just for reference, what does MWP consider a line?
|
anything upto a line separator....most likely CRLF..
Ikeb...
If you used th ".+" in place of the "\s*?' th e expression would read right on through to the next part in those cases where the transfer encoding was not immediate.
That could result in a part that was normal, unencoded 'text/html' that was followed at some point by a gif that was base64 encoded...being trapped.
It would also cause the filter to spend much more time per message than would normally be required.
As to the 'text/html' in the header....this filter is set up to only look for text that is base 64 encoded. You can place a filter right below it that looks for any base64 encoding in the header if you want.
|
|
| Back to top |
|
|
Ikeb
Special Response Team
Forums Admin
Joined: Apr 20, 2003
Posts: 16506
|
| Posted: Fri Nov 21, 2003 3:57 pm Post subject: |
|
|
| denn988 wrote: |
| Perry wrote: | Just for reference, what does MWP consider a line?
|
anything upto a line separator....most likely CRLF..
|
So a Base64 block is considered a single line then? How about attachments?
| denn988 wrote: | If you used th ".+" in place of the "\s*?' th e expression would read right on through to the next part in those cases where the transfer encoding was not immediate.
That could result in a part that was normal, unencoded 'text/html' that was followed at some point by a gif that was base64 encoded...being trapped. |
Yup. Thanks. Caught that from one of UserFriendly's posts.
| denn988 wrote: | | It would also cause the filter to spend much more time per message than would normally be required. |
As I learn this stuff, I'm beginning to understand some realities. A filter could take an inordinate amount of time if it finds lots of matches from the initial portion of a regex and then, each time, has to chunk through the rest of the message to verify if the rest of the regex matches!
| denn988 wrote: | | As to the 'text/html' in the header....this filter is set up to only look for text that is base 64 encoded. You can place a filter right below it that looks for any base64 encoding in the header if you want. |
Right. But why not just cut to the chase? As you pointed out in an earlier post, base64 in the header is as close one can get to a 100% guarantee that a SPAM has been IDed.
|
|
| Back to top |
|
|
denn988
Guest
IP: 66.44.*.*
|
| Posted: Fri Nov 21, 2003 4:39 pm Post subject: |
|
|
Ikeb,
A base64 block contains many CRLFs in it.
The next time you come across one, copy it into WORD...then goto:
TOOLS>>>OPTIONS>>>VIEW
Under "Formatting marks' place a check on the "Paragraph marks" box.
You will then see the paragraph marks (CRLF) in the base64 block.
Also
I could be wrong...but I believe I said that after looking through seven years worth of legitimate emails, I did not find any that sent the entire message in base64. Near 100%, but not guaranteed.
IANA- RESERVED would be a better example of 100%. Even then...it only indicates a forgery of the header.....but that should be sufficient.
|
|
| Back to top |
|
|
denn988
Guest
IP: 66.44.*.*
|
| Posted: Fri Nov 21, 2003 5:05 pm Post subject: |
|
|
Ikeb,
By the way....
I only have three filters set up to "Auto-Delete"
IANA_RESERVED is one.
Another filter that looks for forged return addresses from my own domain is another.
The last one is another very high (Above 30%) trap rate filter....but...that is one that I will not release. I consider that one to be my "ace in the hole' and releasing the strategy for that one could lead to it becoming inefective for me. Sorry 
|
|
| Back to top |
|
|
Ikeb
Special Response Team
Forums Admin
Joined: Apr 20, 2003
Posts: 16506
|
|
| Back to top |
|
|
|
|
|
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
|
Powered by phpBB © 2001 phpBB Group
|
Forum FAQ
Search
Usergroups
Profile
Login to check your private messages
Login
