CastleCops, Internet Crime Fighters
Need help? Click here to register for free! Absolutely zero advertisements on this site!

$9736.22 of $21422.68
left sidedonated so farneed $11686.46 donated to reach our goalright side, our goal
Help CastleCops serve the community on new servers, Donate Here to reach our goal.

Donation/Premium
spacer
block bottom
Security Central
spacer
· Home
· PIRT/Fried Phish
· MIRT
· SIRT
· Deutsch
· Wiki
· Newsletter
· O16/ActiveX
· CLSID List
· Contest2007
· Downloads
· Feedback (send)
· Forums
· HijackThis
· Hijacktrend
· LSPs
· My Downloads
· O18
· O20
· O21
· O22
· O23
· O9
· Premium
· Private Messages
· Proxomitron
· Reviews
· Search
· StartupList
· Stories Archive
· Submit News
· WsIRT
· Your Account
· Acceptable Use Policy
block bottom
Survey
spacer
Was 2007 a good year?

Yes it was a wonderful year
Yes, but there is always room for improvement
Status quo
It was a challenge
Other (leave comment)



Results
Polls

Votes: 941
Comments: 25
block bottom
spacer spacer

Body text in preview not triggering filters when it should
Goto page 1, 2, 3, 4  Next
 
Post new topic   Reply to topic       All -> FavForums -> Mailwasher - Troubleshooting / General [del.icio.us!] [digg it!] [reddit!]
View previous topic :: View next topic  
Author Message
Whisperer

Sergeant
Sergeant


Joined: Mar 29, 2003
Posts: 134
Location: USA

PostPosted: Fri Dec 12, 2003 8:51 am    Post subject: Body text in preview not triggering filters when it should
Reply with quote

In an effort to create filters that will auto-delete spam meeting very specific criteria based on specific text strings that I see repeatedly occurring in spams that I seem to repeatedly get in any given period, I have tried to select key text from the Preview (trying Normal, Raw, and Full display) of a given spam message and insert a certain portion of that text into a Filter that will auto-delete any matching messages. I've even set the filter to respond whether the text string appears in the Body or the Header.

But for some reason, only sometimes does this seem to work. All too often, spam that most definitely appears to match the filter is still getting through without being auto-deleted. I'm quite sure I'm doing everything right, and am wondering if the nature of spam (and their techniques) and the nature of MWP are such that, in fact, what "I think" I'm seeing in the Preview body isn't what the filter is actually seeing.

Even when they try to mix text with code as they sometimes do, and the Preview display shows that they did so, and I select and specify in the filter, say, "VIA>jdaig<GRA" as a full text string, the filter doesn't get triggered even when the same message with the identical string shows up. But I also have the problem when there's no apparent mixing with code. In some cases, they link to a web site, and I put that in the filter, but subsequent spam that also has a link to the same URL gets past the filter.

I really think it's safe to assume I'm doing things correctly.

Is this just how it is or is there something I'm missing?

Thanks.

Back to top
View users profile Send private message Visit posters website
Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16509

Forums Admin Moderators MVP Premium SRT Team CC Committee Team F@H

PostPosted: Fri Dec 12, 2003 2:56 pm    Post subject: Re: Body text in preview not triggering filters when it shou
Reply with quote

Whisperer wrote:
I really think it's safe to assume I'm doing things correctly.

Is this just how it is or is there something I'm missing?

The latter statement suggests you're not sure about the former.

Dunno. It seems you know all about 'hidden' HTML codes. But you seem to suggest there might be something else hidden in the character sequence besides the text you can see when viewing raw source. That's just not possible. You're either missing something or auto-delete isn't performing properly.

Why not post a few of your filters? Have you run some of the same messages that bypass your filter, through TestRExp?

Back to top
View users profile Send private message
denn988

Guest
IP: 66.44.*.*






PostPosted: Fri Dec 12, 2003 3:00 pm    Post subject:
Reply with quote

Post the filter and the string of text (RAW SOURCE VIEW) that you beleive should be trapped by the filter.

When you post...be sure to send it within <blockquote>[code] 'Raw Text' or 'filter code' [/code] </blockquote>format so that it will not be translated on this forum. Also...do NOT disable BBCode when you do that...I had to disable it in this post so the the "[code]...[/code]" prtion did not get translated in this post.

Once we can see what your filter is doing....and the text that you think it should be trapping, we can give you advice. Until then, we are as blind as my wife regarding this.

Back to top
stan_qaz

Premium Member


Joined: Mar 31, 2003
Posts: 10612

Premium

PostPosted: Fri Dec 12, 2003 6:35 pm    Post subject:
Reply with quote

One thing the html trash is usually randomly generated and will not be the same from one spam to the next in most cases.


_________________
Questions? Try the wiki
http://wiki.castlecops.com/MailWasher_Pro
Back to top
View users profile Send private message
Whisperer

Sergeant
Sergeant


Joined: Mar 29, 2003
Posts: 134
Location: USA

PostPosted: Mon Dec 15, 2003 5:00 am    Post subject:
Reply with quote

Thanks so much for the help.

I haven't tested this with TestRExp and have no idea where to even find it.

Here's the Raw Source text from a new spam that isn't being filtered (I removed my email address and host name. etc, from the header):

Code:


Status:  U
Return-Path: <j.kyleed@dragon.acadiau.ca>
Received: from mx07.MYHOST.net ([MYHOST'S IP])
   by cave.mail.atl.earthlink.net (EarthLink SMTP Server) with SMTP id ##############
   for <MYADDRESS@mindspring.com>; Sun, 14 Dec 2003 17:48:33 -0500 (EST)
Received: (qmail #### invoked from network); 14 Dec 2003 22:48:33 -0000
Received: from ####.MYHOST.net ([MYHOST'S IP])
  by ####.MYHOST.net ([MYHOST'S IP])
  with #### via TCP; 14 Dec 2003 22:48:33 -0000
Delivered-To: MYADDRESS@MYDOMAIN.com
Received: (MYHOSTmail ##### invoked from network); 14 Dec 2003 22:48:33 -0000
Received: from MYHOST.net (MYHOST'S IP)
  by MYHOST.net (MYHOST'S IP) with MYHOSTDP; 14 Dec 2003 22:48:33 -0000
Received: (qmail ##### invoked from network); 14 Dec 2003 22:48:29 -0000
Received: from ctaz.com (unknown [206.116.0.36])
  by MYHOST.net ([MYHOST'S IP])
  with ##### via TCP; 14 Dec 2003 22:48:29 -0000
To: MYADDRESS@MYDOMAIN.COM
Subject: =?ISO-8859-1?b?aGk=?=
From: "Jana I. Kyle" <j.kyleed@dragon.acadiau.ca>
MIME-Version: 1.0
Date: Sun, 14 Dec 2003 22:44:35 +0000
Content-Type: text/html
Content-Transfer-Encoding: 8bit
Message-Id: <#####################@cave.mail.atl.earthlink.net>

<p>Dr. Albert O’Connor M.D. is a certified Chief of Urology who studied the effects of the Wapiti Elk Extract and found that most men who ingested Alpha Male Plus were transformed into exploding human volcanoes, filled to the brim with intense, over-flowing sexual energy that continued gushing hour after hour and all night long! </p>
<p>This is the basis for the <strong>ALPHAMALE+</strong> Pill System!</p>
<p> * Increase testosterone levels up to 500% <br>
* Prevent premature ejaculation <br>
* Enhance penis size up to 3 inches <br>
* Maintain harder, stronger erections for hours <br>
* Have amazing sex up to 20 times per day </p>
<p>For more information go to this address:

 
<a href="http://www.ghkp.us/alpha/?jacob">http://www.rollie.biz/alpha/?jacob</a></p>
<p>If you're sick of this - go to the page to be taken off!</p>



I created a filter with two strings of text from the first sentence of the message. Here's the actual filter text from the filters.txt file:

Code:


[enabled],"Alpha Male Plus",Filtered,16711680,AND,Hidden,Blacklist,Delete,Body,contains,"Dr. Albert O'Connor M.D. is a ",Body,contains,"Alpha Male Plus"




The filter as it shows on screen can be seen here.

The message is still on the server. If I close and reopen MWP and Check Mail, it still comes up in the list, even though the filter is set to keep it Hidden. The message isn't even being marked for Delete or Blacklist by the filter. I also created another filter with everything the same except that it's set to look in the Entire Header instead of the Body, but still no effect.

Thanks!

Back to top
View users profile Send private message Visit posters website
Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16509

Forums Admin Moderators MVP Premium SRT Team CC Committee Team F@H

PostPosted: Mon Dec 15, 2003 7:09 am    Post subject:
Reply with quote

Whisperer wrote:
I haven't tested this with TestRExp and have no idea where to even find it.

You don't really need it unless you decide to design your own Regular Expressions (i.e. where you place a 'contains RegExp' or 'does not contain RegExp' rule in a filter.

Whisperer wrote:
Here's the Raw Source text from a new spam that isn't being filtered (I removed my email address and host name. etc, from the header):

Code:


Status:  U
Return-Path: <j.kyleed@dragon.acadiau.ca>
Received: from mx07.MYHOST.net ([MYHOST'S IP])
   by cave.mail.atl.earthlink.net (EarthLink SMTP Server) with SMTP id ##############
   for <MYADDRESS@mindspring.com>; Sun, 14 Dec 2003 17:48:33 -0500 (EST)
Received: (qmail #### invoked from network); 14 Dec 2003 22:48:33 -0000
Received: from ####.MYHOST.net ([MYHOST'S IP])
  by ####.MYHOST.net ([MYHOST'S IP])
  with #### via TCP; 14 Dec 2003 22:48:33 -0000
Delivered-To: MYADDRESS@MYDOMAIN.com
Received: (MYHOSTmail ##### invoked from network); 14 Dec 2003 22:48:33 -0000
Received: from MYHOST.net (MYHOST'S IP)
  by MYHOST.net (MYHOST'S IP) with MYHOSTDP; 14 Dec 2003 22:48:33 -0000
Received: (qmail ##### invoked from network); 14 Dec 2003 22:48:29 -0000
Received: from ctaz.com (unknown [206.116.0.36])
  by MYHOST.net ([MYHOST'S IP])
  with ##### via TCP; 14 Dec 2003 22:48:29 -0000
To: MYADDRESS@MYDOMAIN.COM
Subject: =?ISO-8859-1?b?aGk=?=
From: "Jana I. Kyle" <j.kyleed@dragon.acadiau.ca>
MIME-Version: 1.0
Date: Sun, 14 Dec 2003 22:44:35 +0000
Content-Type: text/html
Content-Transfer-Encoding: 8bit
Message-Id: <#####################@cave.mail.atl.earthlink.net>


OK that was the header (demarcated by the blank line). Now here's where the body starts:

Whisperer wrote:
Code:
<p>Dr. Albert O’Connor M.D. is a certified Chief of Urology who studied the effects of the Wapiti Elk Extract and found that most men who ingested Alpha Male Plus were transformed into exploding human volcanoes, filled to the brim with intense, over-flowing sexual energy that continued gushing hour after hour and all night long!


I created a filter with two strings of text from the first sentence of the message. Here's the actual filter text from the filters.txt file:
Code:
[enabled],"Alpha Male Plus", Filtered, 16711680, AND, Hidden, Blacklist, Delete, Body, contains, "Dr. Albert O'Connor M.D. is a ", Body, contains, "Alpha Male Plus"


Ahah! The text in your filter is different than what is in the message. Specifically you placed "Dr. Albert O'Connor M.D. is a " in your filter but "Dr. Albert OConnor M.D. is a " is the actual text in the message. It's pretty easy missing a subtle difference like this. To input phrases like this, it might be best to copy and paste the phrase into the filter rule.

Back to top
View users profile Send private message
Whisperer

Sergeant
Sergeant


Joined: Mar 29, 2003
Posts: 134
Location: USA

PostPosted: Tue Dec 16, 2003 3:56 am    Post subject:
Reply with quote

[quote="IkebAhah! The text in your filter is different than what is in the message. [/quote]
Okay, that one slipped past me. I'll have to dig through to show the data on others but I'm sure the same thing has happened where there were no special characters at all. At least now I know to watch out for such things.

If I understand this correctly, the text I should work from to create a trigger phrase in a filter is the text I see in the "Raw source" preview, is that correct? (Sometimes the body text it shows is very differentthan what I see in the "Normal" preview.)

And whether I specify "Header" or "Body" depends on whether it's above or below the line of demarcation pointed out by Ikeb, correct?

As to the html gibberish being randomly generated and changing often, any idea of it's in random places or if the little pieces of text between > and < symbols tend to stay the same?

Thanks again!

Back to top
View users profile Send private message Visit posters website
denn988

Guest
IP: 66.44.*.*






PostPosted: Tue Dec 16, 2003 4:56 am    Post subject:
Reply with quote

Whisperer,

I noticed something in the header that may be of interest to you.

Code:
Subject: =?ISO-8859-1?b?aGk=?=


The Subject line in the message that you posted is using a Spammer trick that you can trap with a simple filter:

Code:
The entire header....contains
Subject: =?ISO-8859-


When the first charactors in the Subject are   =?ISO-8859-     that tells the e-mail client that the line is encoded. The client (including MWP) will decode the line. Because MWP decodes the line prior to applying the filter to the 'Subject' field, you cannot try to trap this kind of thing by looking into the 'Subject Field'. You have to look at the raw header.

The only reason for someone using the encoding of the Subject line is to obfuscate the Subject line so that word filters will not find anything. If this technique is used it is a major flag, and an excellent SPAM indicator.

Back to top
Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16509

Forums Admin Moderators MVP Premium SRT Team CC Committee Team F@H

PostPosted: Tue Dec 16, 2003 7:02 am    Post subject:
Reply with quote

Whisperer wrote:
If I understand this correctly, the text I should work from to create a trigger phrase in a filter is the text I see in the "Raw source" preview, is that correct? (Sometimes the body text it shows is very differentthan what I see in the "Normal" preview.)

Thqt's right. The idea of filtering on the displayed text has been often suggested but for now we live without it. Besides it's so much fun sussing out the little tricks SPAMers use the hide their dirty deeds.

Whisperer wrote:
And whether I specify "Header" or "Body" depends on whether it's above or below the line of demarcation pointed out by Ikeb, correct?

Yup! Just had that illustrated to me the other day!

Whisperer wrote:
As to the html gibberish being randomly generated and changing often, any idea of it's in random places or if the little pieces of text between > and < symbols tend to stay the same?

Interesting question! HTML tags (the test between < > tags) are intended to contain formatting instructions to an HTML viewer. There are specific codes that can be entered but SPAMers have used the fact that browsers don't display ANYTHING between tag markers, even complete gibberish. So they do just that, knowing that the document being displayed to the viewer will not appear any different but the raw text will be radically different. And they even go further, adding "nice" words within the tags to throw off Bayesian filters looking for the balance between "good" and "bad" words in a message.

So there's truly a world beneath the surface just within the HTML tags. For a way to detect one of the more obvious word "obfuscation" techniques making use of HTML tags and employed by SPAMers, check out the Need help with regex filter. topic started by Cowboy. In fact most of the more interesting filters revolve around the HTML tag tricks SPAMers use. A good reference to MWP filters is found at the FAQ? topic.

Back to top
View users profile Send private message
Whisperer

Sergeant
Sergeant


Joined: Mar 29, 2003
Posts: 134
Location: USA

PostPosted: Tue Dec 16, 2003 7:52 am    Post subject:
Reply with quote

Thanks so much, folks.

In giving me advice, please keep in mind some important factors: I'm venturing into new waters by creating filters that will "hide and delete" what they catch. Normally, I visually scan everything before deleting since I'm adamant about never risking false positives as I have a small business with high stakes. Therefore, I need to stay only with what is POSITIVE to ONLY catch actual spam in doing what we're discussing here.

I just found another instance where a newly created filter doesn't seem to be doing the trick.

Body in Raw source view:

Code:

<html>
<body>
Only Alpha Male Plus can grant guys tonz of climaxes.
At last, any guy can achieve tons of orgasms without downtime and give
his partner the climax they need


<a href="http://zumasxdcaw.unone.us/alpha/?hpsales">get some info</a>


<br><br><a href=http://shlxzvbvck.<br><br><a href=http://beghy.biz/alpha/optout.shtml?hpsales>No more please</a>
</body>
</html>



My filter which isn't catching it:

Code:

[enabled],"Only Alpha Male Plus",Filtered,16711680,AND,Hidden,Blacklist,Delete,Body,contains,"Only Alpha Male Plus can grant guys tonz of climaxes"



I created the filter by copying and pasting the trigger text, then Cleared all email messages from the viewer, closed MWP and reopened it, did Check Mail again... and the message is still there.

Back to top
View users profile Send private message Visit posters website
Whisperer

Sergeant
Sergeant


Joined: Mar 29, 2003
Posts: 134
Location: USA

PostPosted: Wed Dec 17, 2003 8:09 am    Post subject:
Reply with quote

As a follow-up to my previous message, for those who would like to see a graphic of the filter that's described that isn't catching the message, here it is (except the full text string is longer than what it shows, as illustrated in the previous post).

Back to top
View users profile Send private message Visit posters website
Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16509

Forums Admin Moderators MVP Premium SRT Team CC Committee Team F@H

PostPosted: Wed Dec 17, 2003 4:15 pm    Post subject:
Reply with quote

Whisperer wrote:
I created the filter by copying and pasting the trigger text, then Cleared all email messages from the viewer, closed MWP and reopened it, did Check Mail again... and the message is still there.

That's because you didn't process the message. Did you not see the message in MWP when displaying hidden messages?

Back to top
View users profile Send private message
IP: 65.37.*.*

Guest






PostPosted: Wed Dec 17, 2003 4:36 pm    Post subject:
Reply with quote

Ikeb, I don't understand what you mean, but I'm sure I must be missing something which explains why my filters don't seem to always be working, especially newly created ones.

I thought that when I close MWP it starts a fresh check and download of the server when I reopen it and do a Check, and since I have View set to not show hidden emails, as with other types of messages that I have filters set to leave hidden on the server (but not to delete or blacklist), I assume what I'm after is having it stay on the server... I assumed the filter wasn't working AT ALL since it didn't even treat the message as hidden.

But what am I missing here? What do you mean when you say I didn't "process" it? Oh, wait... Process... as in hit the huge Process icon. Okay... but isn't that just to actually delete things that are checks for deletion, and to put things in the approprate lists (blacklist, friends) for the future? If it's got a filter set to keep it hidden, to mark it for delete and for blacklist, and after closing and reopening MWP and checking mail it turns up in the list (it wasn't hidden as it should have been) and it isn't checked for delete or for blacklist, which it wasn't, that tells me the filter didn't get implemented at all.

Back to top
Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16509

Forums Admin Moderators MVP Premium SRT Team CC Committee Team F@H

PostPosted: Wed Dec 17, 2003 6:48 pm    Post subject:
Reply with quote

Anonymous wrote:
I thought that when I close MWP it starts a fresh check and download of the server when I reopen it and do a Check, and since I have View set to not show hidden emails, as with other types of messages that I have filters set to leave hidden on the server (but not to delete or blacklist), I assume what I'm after is having it stay on the server... I assumed the filter wasn't working AT ALL since it didn't even treat the message as hidden.

But what am I missing here? What do you mean when you say I didn't "process" it? Oh, wait... Process... as in hit the huge Process icon. Okay... but isn't that just to actually delete things that are checks for deletion, and to put things in the approprate lists (blacklist, friends) for the future? If it's got a filter set to keep it hidden, to mark it for delete and for blacklist, and after closing and reopening MWP and checking mail it turns up in the list (it wasn't hidden as it should have been) and it isn't checked for delete or for blacklist, which it wasn't, that tells me the filter didn't get implemented at all.

Yes that is what I meant by Process. OK so the message you expected to get caught is showing up as not hidden, not to be deleted, and whatever else you asked the filter to checkoff if such a message is detected.

Dunno. Your filter looks OK to me. (You might want to change the Status named though, so as to distinguish which filter led to the checkboxes as they appear in the message window but that won't help to resolve the basic detection issue you seem to have.)

Back to top
View users profile Send private message
stan_qaz

Premium Member


Joined: Mar 31, 2003
Posts: 10612

Premium

PostPosted: Wed Dec 17, 2003 8:43 pm    Post subject:
Reply with quote

Have you tried deleting the seenemail.txt file to get mailwashder to process your new filters on old e-mail?

I thought they would be run when you saved the filter but this is a sure way to get mailwasher to treat the mail as new.


_________________
Questions? Try the wiki
http://wiki.castlecops.com/MailWasher_Pro
Back to top
View users profile Send private message
Display posts from previous:   
Post new topic   Reply to topic       All -> FavForums -> Mailwasher - Troubleshooting / General All times are GMT
Goto page 1, 2, 3, 4  Next
Page 1 of 4

 
Quick Reply:
Username: 

Quote the last message
Attach signature (signatures can be changed in profile)
 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001 phpBB Group
spacer spacer