CastleCops :: View topic - Status of filtered messages

Status of filtered messages
Goto page 1, 2 Next
CastleCops -> Mailwasher - Troubleshooting / General

Author: TimK, Location: UK

Posted: Tue Nov 18, 2003 9:05 am Post subject: Status of filtered messages

Hello

I have recently imported what I now understand to be Gary's filters from the MW site (thanks Gary). I'm jut beginning to learn about them, and am scouring this forum for more information.

One question has arisen, though, which I haven't yet found addressed: I'm finding many messages filtered with the status [2]Adult(S), even though they are clearly not porn messages and appear to meet none of the criteria for that filter (they are credit card or debt messages, for instance). I've even had some legit. messages filtered with that status. Can anyone explain why that might be?

Another odd thing is that since importing the filters I've had one or two spam messages slipping past MWP altogether -- remaining hidden -- and appearing in my inbox. I do have some other filters which legitimate my mailing list subscriptions etc., and I have my friends list of course, but these have been running for a year or more and have never allowed spam to be hidden from the display. So it must be to do with the new filters, though none of them legitimate mail. It's a puzzle

I'd appreciate any help on these questions. And I'll try not to bombard the forum with more questions until I've scoured the archives

Thanks

Tim

Author: Ikeb,

Posted: Tue Nov 18, 2003 4:27 pm Post subject: Re: Status of filtered messages

TimK wrote:

One question has arisen, though, which I haven't yet found addressed: I'm finding many messages filtered with the status [2]Adult(S), even though they are clearly not porn messages and appear to meet none of the criteria for that filter (they are credit card or debt messages, for instance). I've even had some legit. messages filtered with that status. Can anyone explain why that might be?

This comes down to "one man's SPAM is another's feast." In other words, if you find that a filter is trapping an inordinate amount of what you'd consider legit email, either modify the filter or disable it. BTW, some people would say that even one false positive is an inordinate amount but that to is a personal thing.

TimK wrote:

Another odd thing is that since importing the filters I've had one or two spam messages slipping past MWP altogether -- remaining hidden -- and appearing in my inbox. I do have some other filters which legitimate my mailing list subscriptions etc., and I have my friends list of course, but these have been running for a year or more and have never allowed spam to be hidden from the display. So it must be to do with the new filters, though none of them legitimate mail. It's a puzzle

That's called having false negatives, a fact of life with any anti-SPAM filtering technology known to man. The challenge is balancing the nuisance of dealing with the few SPAM that make it through against your time. Your time could be the time it take to find the false positives by scanning the hidden messages manually (i.e. not hiding them) and/or time spent to sort through such SPAM after the fact so as to find why they made it through, then adjusting your filters.

Author: TimK, Location: UK

Posted: Tue Nov 18, 2003 5:08 pm Post subject:

Thanks for your reply. However, unless I misunderstand your answers, I fear you may have misunderstood my questions!!

1) The first problem is not that the filter is stopping too many legitimate messages, it is that the messages it does stop appear to have nothing whatsoever to do with the filter! The [2]Adult(S) filter is all about porn words and so on, but the messages that are getting stopped are all about credit cards, debt, and cable tv. I'm happy that they are being filtered, I just don't understand why the main screen gives the Adult filter as the status! Confused

2) I have only been getting spam mails slipping through MWP since importing Gary's filters. They are being treated as hidden messages, but only my friends list is set to be hidden. Surely the filters, which increase the protection, can't be responsible for hiding messages that are not on my friends list.

Hope I've explained myself more clearly now.

Thanks for your time

Tim

Author: stan_qaz,

Posted: Tue Nov 18, 2003 5:27 pm Post subject:

TimK wrote:

The filters aren't looking for "porn" as such they are looking for a combination of different letters in the e-mail that the author found to be common to porn spams on his system. You will need to look at the filter and see what it is looking for and at the falsely tagged e-mails and see what it found. You can then either delete or modify the filter to work for you.

Quote:

If you are reviewing the e-mails before processing, most of us have to do this, and correcting mis-marked messages and marking ones that were not identified before processing mail you should be removing all the spam that was in your box when you checked it.

Mail can arrive during this time so it is a good idea to check your mail one more time just before downloading it into your mail program to insure no new spam arrived while you were processing the first batch.

Author: TimeGhost, Location: USA

Posted: Tue Nov 18, 2003 6:05 pm Post subject: Re: Status of filtered messages

TimK wrote:

Loan and credit applications often contain such language as "You must be 18 years of age." I think this might be triggering the Adult filter.

Author: TimK, Location: UK

Posted: Tue Nov 18, 2003 8:56 pm Post subject:

Thanks for your suggestions, but I don't think they apply to this situation. For example, I've just received a message saying this:

No need to cable bills...the best cable box now

and then an image with 'cable box descramble. no more cable bills. Free pay tv. Click here'

If you have the [2] Adult (S) filter, you will see that not one of these words appears in that filter. So why does the status column show the message as being caught by that filter?

I'm just trying to understand how all this works

Thanks

Tim

Author: stan_qaz,

Posted: Tue Nov 18, 2003 10:27 pm Post subject:

Are you looking at the preview message mode or the view full headers mode? The filters work in the full header view and often there are things hidden by html coding and such that can cause problems.

The other thought is that there is something wrong in the copy of the filter that you got.

Author: TimK, Location: UK

Posted: Tue Nov 18, 2003 10:53 pm Post subject:

I'm looking at the full header mode, and I've searched every word and mark for something recognisable in the Adult filter. I think I'll disable it for a while and see if any other filters catch something - funny thing, now I come to think of it, is that no other filter has appeared in the status column at all! It's as if it is acting as a sort of generic filter marker or something (if you see what I mean)

I suppose there could be something in the filter which is catching something more general than just porn words; I don't understand regular expressions enough to be able to tell. But lots of forum users must be using the same filter. Does anyone else have trouble. Does the correct filter always appear in the status box?

I need to crack this

Tim

Author: Ikeb,

Posted: Wed Nov 19, 2003 5:05 am Post subject:

TimK wrote:

I suppose there could be something in the filter which is catching something more general than just porn words; I don't understand regular expressions enough to be able to tell. But lots of forum users must be using the same filter. Does anyone else have trouble. Does the correct filter always appear in the status box?

Does the filter that caught the SPAM show up in the Status field you mean? I've never noticed any such problem with any of the more straightforward filters that I have set up to look for POPFile header mods. But I can't say I've ever crosschecked that with any of the regex filters. That would be very difficult to do I expect.....

But truth be told, I don't really care which filter catches the SPAM, so long as it's caught! Now if I could only get my filters to catch every SPAM I receive, I'd be laughing!

Author: TimK, Location: UK

Posted: Wed Nov 19, 2003 8:58 am Post subject:

Thanks Ikeb for that. Of course, I'm glad too that the filters are catching spam - but at the moment they're catching too much and too indiscriminately.

However, I think I'm getting nearer solving the mystery. If a message meets the criteria for any filter higher in the list than [2] Adult (S) then it gets caught by the appropriate filter, and the filter name appears in the status column. If the appropriate filter is below the Adult one in the list, the message seems to get caught by the Adult filter. So somehow the Adult filter is catching almost any message, spam or not, that is not either in my friends list or caught by filters higher in the list.

Furthermore I decided to import the filters again and remembered that when I first imported them and opened them in Notepad, the [2] Adult (S) filter seemed too long to fill the page, and wrapped the last few words - despite the fact that wordwrap was turned off in Notepad. This applied to another filter to do with debt. I was worried that this would result in the filters failing to work, so I simply deleted the last few words. When importing them this time I didn't and just let them wrap.

I'm sure this is too much of a coincidence: somehow the fact that the filter wraps itself must introduce some sort of break character which becomes a de facto criterion for the filter. Does this make sense? Has anyone had a similar experience?

The simplest solution might be to somehow import the filters without using Notepad, or by persuading it not to wrap these last few words. Or I could just move the Adult filter to the bottom of the list, but I want to know what's making it do what it does.

Any suggestions. I really appreciate all the help you are offering.

Tim

Author: TimeGhost, Location: USA

Posted: Wed Nov 19, 2003 4:54 pm Post subject:

TimK wrote:

Furthermore I decided to import the filters again and remembered that when I first imported them and opened them in Notepad, the [2] Adult (S) filter seemed too long to fill the page, and wrapped the last few words - despite the fact that wordwrap was turned off in Notepad.

Yes is a very helpful clue. Indeed, if the filter wrapped or was truncated such that a wildcard were left at the end, I can imagine that it would catch all sorts of unintended things.

I've just opened it in NotePad. I see that the word Twat and the closing double-quote were put on a new line. That's definitely a problem. If you can't find a better text editor, you might have to remove that last rule, and enter it by hand using the GUI.

Author: Ikeb,

Posted: Wed Nov 19, 2003 4:54 pm Post subject:

Off the top of my head, I'd say 'use another text editor'. Seriously. I wouldn't trust notepad with my grocery list.

Actually I was going to suggest looking at your filters to see if two of them perhaps got combined. But then you posted again saying you checked each of the filters ( I assumed this was via the MWP edit facility) and could find no problem. But now I'm not sure. Have you done that? If the filters look OK there, I'm stumped.

Author: TimeGhost, Location: USA

Posted: Wed Nov 19, 2003 5:15 pm Post subject:

Ikeb wrote:

I wouldn't trust notepad with my grocery list.

Author: TimK, Location: UK

Posted: Wed Nov 19, 2003 5:37 pm Post subject:

Can you recommend a good text editor (preferably a free one!)?

The plot has thickened a little. I cut out the whole of the last rule from [2] Adult (S) and added it to [2] Adult (B), which is a much shorter string. The result was that the latter filter started trapping everything! When I removed the rule altogether, everything started working like it should. I now surmise that there is something in that rule which catches almost everything, but I've no idea what it is.

I won't print the rule here for obvious reasons, but if anyone wants to check it, it starts: "Tight(est)...and ends: ***hole". Though I guess if a lot of people are using it there can't be anything much wrong with it. I'll wait and see if another text editor helps.

Watch this space

Tim

Author: UserFriendly,

Posted: Wed Nov 19, 2003 6:24 pm Post subject:

TimK wrote:

when I first imported them and opened them in Notepad, the [2] Adult (S) filter seemed too long to fill the page, and wrapped the last few words - despite the fact that wordwrap was turned off in Notepad.

It seems that notepad cannot handle lines longer than 1024(?) characters.

TimK wrote:

This applied to another filter to do with debt. I was worried that this would result in the filters failing to work, so I simply deleted the last few words.

I would suggest that you don't edit a RegEx filter unless you know how it works - one character in the wrong place can cause problems such as you're experiencing.

TimK wrote:

When importing them this time I didn't and just let them wrap.

You still had the same problem after this?

TimK wrote:

I'm sure this is too much of a coincidence: somehow the fact that the filter wraps itself must introduce some sort of break character which becomes a de facto criterion for the filter. Does this make sense? Has anyone had a similar experience?

Yes, that does make sense. However, I tried saving the filters with notepad and didn't have any problems. Although you may be using a different version of notepad - which version of Windows do you have? I'm using Windows 2000.

It's possible that there was an error in the filter which has recently been fixed by Gary. Try downloading the filters again (Ctrl-Refresh the page so you're sure you're getting the latest version of the page).

NoteTab Light from www.notetab.com is a good free text editor although it may be a bit bulky for some. It ought to eliminate the text wrapping problem though.

-UserFriendly

CastleCops -> Mailwasher - Troubleshooting / General

All times are GMT

Goto page 1, 2 Next Page 1 of 2