CastleCops® Syntax of "Received:" line in header

Need help? Click here to register for free! Absolutely zero advertisements on this site!

	Donation/Premium

	Donations. Become Premium Today!

	Security Central

	· Home · PIRT/Fried Phish · MIRT · SIRT · Deutsch · Wiki · Newsletter · O16/ActiveX · CLSID List · Contest2007 · Downloads · Feedback (send) · Forums · HijackThis · Hijacktrend · LSPs · My Downloads · O18 · O20 · O21 · O22 · O23 · O9 · Premium · Private Messages · Proxomitron · Reviews · Search · StartupList · Stories Archive · Submit News · WsIRT · Your Account · Acceptable Use Policy

Forum FAQ

Usergroups

Profile

Select FavForums

Syntax of "Received:" line in header
Goto page 1, 2, 3 Next

All -> FavForums -> Mailwasher - Troubleshooting / General

[del.icio.us!] [digg it!] [reddit!]

View previous topic :: View next topic

Author

Message

Walt

Sergeant

Joined: Mar 15, 2003
Posts: 91
Location: USA

Posted: Sat Dec 27, 2003 3:59 pm Post subject: Syntax of "Received:" line in header

I have been using the following RE filter for a few weeks now, and find it works 90% of the time detecting when an email has come from a foreign IP address. However, the remaining 10% of the time, it seems it fails to detect an IP which should match, because of the difference in syntax I am finding in the "Received:" line in a header.

The RE filter I am using is:

Code:

[enabled],FOREIGN,FOREIGN,16711808,AND,Delete,EntireHeader,containsRE,"(?# list of IP blocks assigned to RIPE - must be updated as required )^Received: from [^[]*?\[(62|8[0-4]|19[345]|21[237])(\.[1-2]?\d?\d?){3}\]",EntireHeader,containsRE,"(?# list of IP blocks assigned to APNIC - must be updated as required )^Received: from [^[]*?\[(6[01]|20[23]|21[01]|21[89]|22[0-2])(\.[1-2]?\d?\d?){3}\]",EntireHeader,containsRE,"(?# list of IP blocks assigned to LACNIC - must be updated as required )^Received: from [^[]*?\[20[01](\.[1-2]?\d?\d?){3}\]"

First, my apologies to the author. I thought I got this RE filter from a thread here, and intended to make this posting to the same thread, but I can't seem to find it even after several searches here.

Anyway, it seems like this RE filter expects the IP address to always be enclosed in square brackets ("[" and "]").

However, I am finding cases where the "Received:" line can look like:

Are all these valid?

denn988

Guest
IP: 66.44.*.*

Posted: Sat Dec 27, 2003 6:32 pm Post subject:

Walt,

You should leave the []s in place in that filter.

The reason the []s are in there is to ensure that what the filter is picking out is an actual IP address in the header. I can't quote you the RFC for this, but as I remember, IP addresses in the Routing fields of the header are required to be enclosed by []'s.

The other thing that I seem to remember is that NO succeeding mail server is allowed to try to fix any malformed 'Received:' fields that were included in the routing fields of any message it receives.

Every message that I have ever seen that had a valid ( read that as NOT forged ) 'Recieved:' field in it has had the IP address of the sending server enclosed in []s.

If You remove the []s from the RegExp on this one you could easily find yourself looking into forged 'Received:' lines for the IP addresses specified...and that would be a waste of time. You could write a filter that looks for malformed headers...and that would be a better thing to do.

There is one thing that you do need to do to modify that filter though......

When I posted it, I had not noticed a goof that I had made when I combined the three separate filters ( APNIC, RIPE and LACNIC ) into the one 'FOREIGN' flter, till sometime after I posted it.

The filter is set to AND, meaning all rules need to be satisfied in order for the filter to fire...

That need to be changed to OR, meaning that any rule must be satisfied for the filter to fire.

Or you can combine the separate rules into one rule:

Code:

(?# FOREIGN IP)\[(6[0-2]|8[0-4]|19[345]|20[0-3]|21[0123789]|22[0-2])(\.[1-2]?\d?\d?){3}\]

Which is what I have done. The nice thing about this is that you can combine it into 'multi-stage' rules.
Example:

I have a couple of filters that looks for any sender that connects directly to my ISPs MX (incoming) mail server...and whose IP address fails on the RDNS (Reverse Domain Name Service) lookup that my ISP does with any incoming mail connection.

The fiter with the lower priority of these two (the last one to look at the message) simply looks for an RDNS failure that was returned by my ISP. This filter just flags the message with a warning "RDNS FAIL"

The filter with a higher priority will look into the IP address associated with that RDNS failure...and if it is a 'FOREIGN' IP...it will Auto-Delete the message.

Someone else who is in a different situation than I am (say they live in Europe) might not want to use such a filter...but for me (in the US) it works fine.

By the way...you may want to keep this link handy as it contains the IANA address blocks that the filter is looking for (APNIC, RIPE and LACNIC). You will need to check the above link periodically to keep the blocks updated.

Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16543

Posted: Sat Dec 27, 2003 7:06 pm Post subject:

denn988 wrote:

When I posted it, I had not noticed a goof that I had made when I combined the three separate filters ( APNIC, RIPE and LACNIC ) into the one 'FOREIGN' flter, till sometime after I posted it.

Ahem!! "Sometime after" being like ... oh I don't know .... as soon as your error was pointed out to you perhaps? Razz

denn988 wrote:

The fiter with the lower priority of these two (the last one to look at the message) simply looks for an RDNS failure that was returned by my ISP. This filter just flags the message with a warning "RDNS FAIL"

The filter with a higher priority will look into the IP address associated with that RDNS failure...and if it is a 'FOREIGN' IP...it will Auto-Delete the message.

I'm also beginning to use a number of filter pairings. It sure would be handy if FireTrust would allow mixed ANDs and ORs in each filter so as to facilitate better filter management and faster filtering.

denn988

Guest
IP: 66.44.*.*

Posted: Sat Dec 27, 2003 8:31 pm Post subject:

Ikeb wrote:

denn988 wrote:

When I posted it, I had not noticed a goof that I had made when I combined the three separate filters ( APNIC, RIPE and LACNIC ) into the one 'FOREIGN' flter, till sometime after I posted it.

Ahem!! "Sometime after" being like ... oh I don't know .... as soon as your error was pointed out to you perhaps? Razz

denn988 wrote:

It would be very nice if MWP would incorporate an automatic RDNS lookup on the Recipient' ISP's 'Received:' field in each message header. The Recipient ISP's MX server could be pretty easily determined by doing an MX lookup of the user's domain (for each account entered) and the telnetting to that server on port 25 in order to see the actual server name structure used by the ISP.

As an example....when an MX lookup is done on my domain the MX lookup would return 'mx.mail.mydomain.net'. That is the actual server name that an outside sender would connect to to send mail to me. If MWP then connected to that server name via telnet port 25, they would see the actual naming structure that the ISP uses for their MX servers.

In my case the return might be 'mx09.mrf.mail.mydomain.net' which translates to 'mx\d+\.mrf\.mail\.mydomain\.net'. A test message could be sent back to my account and when it is received back by MWP it would verify the naming structure by looking for it in the 'Received:' field. From then on, MWP could identify the actual 'Received:' field that was placed in the header by my ISP's MX server.

That could be demonstrated by the use of 'Comcast.net'.

If you have nslookup available enter the following:

nslookup -q=mx comcast.net

The return will be:
<blockquote>
<tt>Non-authoritative answer:
comcast.net MX preference = 5, mail exchanger = gateway-r.comcast.net
comcast.net MX preference = 5, mail exchanger = gateway-s.comcast.net

comcast.net nameserver = dns01.jdc01.pa.comcast.net
comcast.net nameserver = dns02.jdc01.pa.comcast.net
gateway-r.comcast.net internet address = 216.148.227.126
gateway-r.comcast.net internet address = 204.127.198.26
gateway-s.comcast.net internet address = 63.240.76.26
gateway-s.comcast.net internet address = 204.127.202.26
dns01.jdc01.pa.comcast.net internet address = 68.87.96.3
dns02.jdc01.pa.comcast.net internet address = 68.87.96.4</tt></blockquote>

Then do a telnet as follows:

telnet gateway-r.comcast.net 25

You will open up a telnet connection to that server and its first response will be something like:

<blockquote><tt>220 rwcrmxc15.comcast.net - Maillennium ESMTP/MULTIBOX rwcrmxc15 #78</tt></blockquote>

Type QUIT then [ENTER] to close the connection.

This could be done for each MX server listed for that domain.

It will give you the structure that the ISP uses for the naming of their MX servers, and that structure could be verified by the test message.

I have already done this for your mailserver Ikeb, and though I will not post the results here, I can tell you that in your case, they use no special naming structure...the MX servers listed in response to the 'nslookup' should be the names used in your header fields.

With that in hand, you could have filters set up for that returned RDNS NAME. You would not have to worry about how your ISP formats the field...or if they do an RDNS.

You could then take this a step further and have MWP create a lookup file for the 'dial-up' type domains that you wish to block....or the individual IP addresses that connect directly to your ISP's MX server.

That would be a most useful tool, and it would probably use much less bandwith than going to the RBL sites. It would also give each user control over the domains and individual IP addresses that you might wish to block.

As far as pointing out my mistakes....I have so many people pointing out my mistakes and shortcomings that it is difficult to remember who pointed out what... Shocked

But I do appreciate them all nonetheless. Razz

IP: 67.74.*.*

Guest

Posted: Sun Dec 28, 2003 12:41 am Post subject:

denn988 wrote:

You should leave the []s in place in that filter.

The reason the []s are in there is to ensure that what the filter is picking out is an actual IP address in the header. I can't quote you the RFC for this, but as I remember, IP addresses in the Routing fields of the header are required to be enclosed by []'s.

The other thing that I seem to remember is that NO succeeding mail server is allowed to try to fix any malformed 'Received:' fields that were included in the routing fields of any message it receives.

Every message that I have ever seen that had a valid ( read that as NOT forged ) 'Recieved:' field in it has had the IP address of the sending server enclosed in []s.

If You remove the []s from the RegExp on this one you could easily find yourself looking into forged 'Received:' lines for the IP addresses specified...and that would be a waste of time. You could write a filter that looks for malformed headers...and that would be a better thing to do.

I was more thinking that the SPAMers are simply being sloppy. Their "quick and dirty" mail server isn't generating syntactically correct "Received" lines. I think that might be more likely than they are intentionally being baddly forged like that.

I am also counting on the SPAMers being sloppy enough to use a true foreign IP; possibly even their own.

denn988 wrote:

When I posted it, I had not noticed a goof that I had made when I combined the three separate filters ( APNIC, RIPE and LACNIC ) into the one 'FOREIGN' flter, till sometime after I posted it.

The filter is set to AND, meaning all rules need to be satisfied in order for the filter to fire...

That need to be changed to OR, meaning that any rule must be satisfied for the filter to fire.

Thanks for the reminder. Actually, I split your RE into three separate ones in order to actually tag them as being from either APNIC, RIPE or LACNIC. It was just for my question, I wanted to go back and quote your exact original as the reference. I am sorry since that seems to have caused more confusion.

Walt

Sergeant

Joined: Mar 15, 2003
Posts: 91
Location: USA

Posted: Sun Dec 28, 2003 1:00 am Post subject:

BTW, here is an actual SPAM header...

Code:

Status: U
Return-Path: <jsinghrv@terastore.de>
Received: from ceylos.de ([68.226.245.35])
by meadowlark (Earthlink/Onemain SMTP Server) with ESMTP id 1aAoor54S3NZFl60
for <Walt@early.com>; Sat, 27 Dec 2003 16:02:50 -0800 (PST)
Received: from 83.82.205.50 by smtp.terastore.de;
Sun, 28 Dec 2003 16:04:01 +0000
From: "Jake Singh" <jsinghrv@terastore.de>

The first "Received:" is an IP which is a valid ARIN address for a company in Atlanta GA. Even if that is false, there is really nothing one can do with it reasonably.

The second "Received:" is an IP from RIPE. I don't know if this "Received:" header line is forged, but it would be better if the same RE filter caught it, and flag it as being a RIPE address.

Anyway, that is what I was hoping to do, and why I asked about variations in "Received:" header line syntax.

denn988

Guest
IP: 66.44.*.*

Posted: Sun Dec 28, 2003 1:02 am Post subject:

Walt, Feel free to modify any filter posted here in any way that you wish. The more that you play with these filters...the more that you will learn from them. As far as the Spammers being sloppy goes.....yes, many of them are so challenged that they would not have a 'snake's chance at the South Pole' of doing anything right...but there are some others who are rather good at what they do. By the way....if you look at it this way....any 'Received:' line that is malformed that badly must be an indication of a forgery. You can take that in the same way that the 'IANA-Reserved' filter views the inclusion of IP addresses that are unnassigned (IANA-Reserved) in a message header. They are forged.... If the header is forged...trash it.

Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16543

Posted: Sun Dec 28, 2003 3:19 am Post subject:

Walt wrote:

Code:

The first "Received:" is an IP which is a valid ARIN address for a company in Atlanta GA. Even if that is false, there is really nothing one can do with it reasonably.

Note that the first Received field doesn't have a RDNS. When I do an nslookup for that IP address, I get cdm-68-226-245-35.laft.cox-internet.com. Looks like one of those dialup servers eh? Wink

Actually I have a regex that picks off the Received field homing to my ISP's server if the sending server doesn't have a domain name:

Code:

If the entire header contains the RegExpr "^Received: from [^/(]*?\(\[[^\n]*?\n[^\n]*?by smtp\d*\.domain\.tld" then mark the message as mail to be deleted. This filter takes priority over the friends list.

It looks like you'd have to modify the regex somewhat since your ISP appears to structure the Received field a bit differently.

Ikeb

Special Response Team
Forums Admin

Joined: Apr 20, 2003
Posts: 16543

Posted: Sun Dec 28, 2003 3:42 am Post subject:

denn988 wrote:

It would be very nice if MWP would incorporate an automatic RDNS lookup on the Recipient' ISP's 'Received:' field in each message header.

Wouldn't that be time consuming? Besides shouldn't the recipient's mail server be doing so already?

denn988 wrote:

The Recipient ISP's MX server could be pretty easily determined by doing an MX lookup of the user's domain (for each account entered) and the telnetting to that server on port 25 in order to see the actual server name structure used by the ISP.

So this would 'train' MWP for each email account that the MWP user has? That would be useful, especially since this holds the promise of allowing non-techies to have MWP help them filter the Received fields. Put that one in the suggestion box for sure!

Walt

Sergeant

Joined: Mar 15, 2003
Posts: 91
Location: USA

Posted: Sun Dec 28, 2003 4:17 pm Post subject:

Just to beat a dead horse a bit more...

Here are some of the "Received:" header lines from the "Topic Reply Notification" from right here at computercops. I don't believe this is suppose to contain any forged lines.

Note the second "Received:" line, the IP is surrounded by parentheses and not brackets. Rolling Eyes

Code:

Received: (qmail 20290 invoked by alias); Sun, 28 Dec 2003 01:04:44 -0000
Received: from unknown (HELO bugsbunny.laudanski.com) (66.227.19.111)
by mx02.nic.name with SMTP; Sun, 28 Dec 2003 01:04:44 -0000
Received: (from nobody@localhost)
by bugsbunny.laudanski.com (8.11.6/8.11.6) id hBS12NU04992;
Sat, 27 Dec 2003 20:02:23 -0500
Message-Id: <200312280102.hBS12NU04992@bugsbunny.laudanski.com>

marke

Corporal

Joined: Jul 14, 2003
Posts: 50
Location: New_Zealand

Posted: Sun Dec 28, 2003 6:25 pm Post subject:

Hi

Am I correct in assuming that the following is an example of an incorrect addressing, i.e. a spoofed address?

Code:

Received: from 203.96.16.164 (unknown [218.18.130.106])
by desire.actrix.co.nz (Postfix) with SMTP id 48A6A1597B

Should I filter for the "unknown " and auto delete? or is there a chance that this could be a legit message.
Best regards,

IP: 66.44.*.*

Guest

Posted: Mon Dec 29, 2003 12:32 am Post subject:

marke wrote:

Hi

Am I correct in assuming that the following is an example of an incorrect addressing, i.e. a spoofed address?

Code:

Received: from 203.96.16.164 (unknown [218.18.130.106])
by desire.actrix.co.nz (Postfix) with SMTP id 48A6A1597B

Should I filter for the "unknown " and auto delete? or is there a chance that this could be a legit message.
Best regards,

Marke,

Your example looks like SPAM, but for a different reason than you might think.

The MX server at 'desire.actrix.co.nz' uses a format that looks something like this:<blockquote>Received from HELO (RDNS Resolved Name [sender IP Address])
by desire.actrix.co.nz ...</blockquote>

In order to explain the above 'Received:' field format, I am going to have to spend a minute to explain what each of those values represent.

I will start with the sender IP address. It is a pretty simple thing, but I want to make sure there is no misunderstanding here.

The sender IP address is the IP address that made the connection to the MX server 'desire.actrix.co.nz'. The MX server upun receiving a connection request from the sender has performed an RDNS (Reverse Domain Name Service) lookup on the sender's IP Address. This is usually done (from what I have seen) prior to responding to the sender and opening an SMTP session.

The result of that RDNS lookup was that the RDNS name could NOT be resolved. If you have a later version of Windows (I have Xp) you may want to do a search of your '\WINDOWS\SYSTEM32\' folder for a file named 'nslookup.exe'. If you have that file, you can do an RDNS lookup yourself.

Open a COMMAND WINDOW (DOS PROMPT) and enter the following:

nslookup 218.18.130.106 [ENTER]

The return of this should include a statement:

<blockquote>can't find 218.18.130.106: Non-existent domain</blockquote>This is an RDNS failure...it did not resolve into a name. Your ISP's (assuming 'desire.actrix.co.nz' is your server here) MX server placed 'unknown' in the field for the RDNS Resolved Name.

AN RDNS failure, in and of itself is not totally unusual, so you would need to have more to go on than that before you dispose of the message as SPAM. An RDNS failure may increase the 'smell' factor...but it does not spell SPAM by itself.

Let's look at another value in that field. The HELO value.

The HELO is the identifier that a sender will provide to the MX server after a connection is accepted by the server. Some MX servers will require this, and with some others it will be optional.

The sender of the above message sent the following to your MX sever once connected:

HELO 203.96.16.164 ...which identified itself to your server as '203.96.16.164' . This is what your server used in the HELO value of the 'Received:' field.

By the way...this can be sent as just about anything. It is meaningless as an identifier of who the actual sender is, because there is no way to verify a HELO.

You can try something though...something that may surprise you.

Try doing an RDNS on that 'IP Address' that the sender provided in their HELO. Call up 'nslookup' again...and this time use that HELO value:

nslookup 203.96.16.164 [ENTER]

The Resolved RDNS Name for that IP Address is:<blockquote>'desire.actrix.co.nz'</blockquote>

Which is the Resolved name of the your ISP's MX server. It is without a doubt...SPAM. I say without a doubt because the sender has gone out of his way to identify himself as the IP address of the MX server he is connecting to. There is no reason for a legitimate sender to do this.

I get a fair amount of SPAM that uses the same tactic...and I have written a filter that looks for...and Auto-deletes...based on that alone. In your case the Filter would look like this:

The entire header.....contains RegExpr....

<blockquote>^Received: from[^(]*?(203\.96\.16\.164) \([^;:]*?by desire\.actrix\.co\.nz </blockquote>

You can also place any other HELO that you want by using the following format:

(203\.96\.16\.164|SPAM incorporated|aya\.office\.ne\.jp)

You can place pretty much as many HELO values as you want...just be sure to include the | between the values entered with no other spaces.

I hope I did not overdo the explaination....or confuse anyone....

denn988

Guest
IP: 66.44.*.*

Posted: Mon Dec 29, 2003 1:27 am Post subject:

Walt wrote:

Code:

Walt,

I figured I would answer Marke's question first....

I don't know why your server would return an RDNS failure on '66.227.19.111'. I get a valid return of 'laudanski.com' when I do an RDNS of that IP Address.

I am assuming that the server 'mx02.nic.name' is your server because ''laudanski.com' is a valid MX server...and it is quite possible that they have a small enough operation that their MX server serves 'double-duty' as their SMTP server. Their SMTP server would most likely connect directly to your MX server...so that is why I assume the MX server is yours.

If that is your server...they are the ones that are failing to enclose the IP address in []s....and you may wish to speak to them about it. Perhaps you could convince them to upgrade their software on that server...or at least conform to this standard. If that is not your server...you may want to speak to whoever it belongs to about it.

If you want to see a little more about reading e-mail headers...here is a good place to start.

You may also wish to search through the RFCs for more information on this. I am not going to take the time myself to do the search....but you might want to, if for no other reason that to have 'ammunition' to present to your provider when you ask them to fix their 'Received:' fields.

denn988

Guest
IP: 66.44.*.*

Posted: Mon Dec 29, 2003 3:50 am Post subject:

Walt wrote:

BTW, here is an actual SPAM header...

Code:

Walt,

I went back a few posts to this one and I wanted to point out something about this one also....

If the MX server that received that one had performed an RDNS on the sender IP address, it would have found that it resolved to:<blockquote>cdm-68-226-245-35.laft.cox-internet.com</blockquote>

A 'Whois' lookup is a valuable tool and it will tell you who the IP address is registered to (in this case - Cox Communications, Inc), but an RDNS lookup provides a little more specific information about the actual IP address in question.

That sure looks like a Cox high-speed dialup connection to me. My 'BANNED DIAL-UP' filter would have Auto-deleted that message before any filter even looked at the Subject: field...or the body of the message.

You need to talk to your ISP about doing RDNS lookups on the IP addresses of any incoming mail. You are missing out on the ability to use some highly effective filtering techniques because your ISP does not perform RDNS lookups.

As to the second line...its a forgery...I say that because there is no continuity whatsoever between it and the next 'Received:' field.

denn988

Guest
IP: 66.44.*.*

Posted: Mon Dec 29, 2003 3:58 am Post subject:

Ikeb.... I just saw your post in reply to the post of Walt's that I just replied to.... Walt certainly can't invoke 'Phil Collins' ('No reply at all') with respect to that post, now can he...

	All -> FavForums -> Mailwasher - Troubleshooting / General	All times are GMT Goto page 1, 2, 3 Next
Page 1 of 3

You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


	2002-2008 � Computer Cops LLC dba CastleCops®. All rights reserved. Acceptable Use Policy. Use signifies your agreement. 174ppm 0.325s (0.049s) Making cybercriminals unhappy since 2002*