A Spam Doctor writes...
Twice this week I've been asked about what the industry is doing to fight spam. Since this is something that plagues all of us I thought I'd write a short article to answer that question. Among other things, if I get asked a third time I can just say, "read my blog". I've tried to keep this in simple terms so some of the descriptions are simplified.
How email works
Email is moved around the Internet by special servers called SMTP (Simple Mail Transfer Protocol) servers. They are also called MTAs (for Mail Transfer Agents), MDAs (Mail Delivery Agents) or simply Mail Relays. As far as anyone but an email system administrator is concerned, all these names are equivalent (and even for an email admin, the distinctions are subtle).
Let's say you write a mail to your friend Albert, whose email address is albert@example.com. When you click the "send" button, your email client program (Outlook Express, Thunderbird or whichever you happen to use) makes a connection to an MTA belonging to your ISP, and sends the mail to it. The information it sends contains your address (called the "From" address), the recipient address (Albert's email address, in this case), the body of the mail, any attachment files you added, and some other control information.
Now the ISP's MTA makes a connection to an MTA belonging to example.com, and sends the exact same information on. That MTA looks at the recipient address and sees that the mail is for the correct domain, and at that point it locates Albert's mailbox file and dumps the mail into it. When Albert reads his mail he sees that the message came from you because it has your email address as the From address.
How do the spammers exploit the email infrastructure?
The problem is that the MTAs take the From addresses that they get in mail as gospel truth. That means that I could make a direct connection to my ISP's MTA and create a mail using your email address as the From address. The person who receives the mail would think it came from you.
Spammers use this trick to label mail with bogus From addresses. Because the ISPs have no way to check that the mail really did originate where it says it did, they have to rely on filtering and other techniques to try to figure out which mails are spam. Every time the anti-spam propeller-heads come up with a better filter, it's not long before the spammers figure out a way to fool it, so it's an arms race with no end in sight.
What measures are being taken?
The industry is trying to fix the problem by coming up with ways for the MTAs to verify that each mail they receive came from the domain it claims in the From address. There are basically two approaches:
The simplest way is for every domain on the Internet to provide a public list of the IP addresses of all of its MTAs. When an MTA receives a mail it can identify the IP address of the system that sent it, then look up the list of valid IPs for the domain in the From address. If the connection came from an IP address that's in the list for the claimed domain, it passes the test. This method is called SPF (Sender Policy Framework) and is being used by AOL and others. There is a similar mechanism called SenderID, but for various reasons that I won't go into here this doesn't seem to be gaining much support at the moment and it's likely to be abandoned.
Another way is more complex but more reliable and secure. In this system each mail has a special label called a Digital Signature attached to it by the sending ISP's MTA. At the receiving end the signature can be checked, and if it checks out it proves that the mail came from the domain claimed in the From address. Without going into too much detail, the signature uses encryption to make sure that it could only have been created by the domain that claims to have sent it. This system is called DomainKeys and is being used by Yahoo!, Google, Earthlink and others.
How does this help?
If you can check a mail and verify the sender then two things are possible. If the checks fail, you know that the mail didn't come from where it claims to have originated. That mail can be put into a "suspect" mail folder, for example.
If the mail does check out, that still doesn't prove that it's not spam - there's nothing to stop a spammer setting up an IP list and/or signature software like the "legitimate" ISPs. But it does make it possible for your and my ISPs to build and share "reputation databases" containing lists of domains that are known to be regular spammers (and also domains that are known to be "good"). When your ISP gets a verified mail from a known spammer domain, they can mark the mail as probable spam so you can decide whether to even open it before you delete it.
As time goes by more ISPs and large-volume mail producers will implement these measures. The spammers that follow suit by providing verification data with their mail will have that mail marked as spam by the time it gets to your in-box. And the ones that don't will have their mail marked as suspect, since the sender can't be verified.
When will this happen?
It's already happening. AOL's servers are checking for SPF information today, and it won't be long before they start insisting on verification data. MSN and Hotmail aren't far behind. Yahoo! checks incoming mail for DomainKeys signatures right now and will be adding features to their web mail client to allow users to decide what to do with unverified mail sometime during the next three months. Google and Earthlink will be doing the same. It's very likely that all the major ISPs and most, if not all, of the smaller ones will have deployed the necessary software by the end of this year.
Is there anything you need to do?
For the time being, no - your ISP will be taking care of the details by making the necessary changes and upgrades to their servers. You won't need to change your email client and for the most part you won't even be aware of the shift - until you realize one day that the volume of spam you get is nothing like what it used to be.
Later it may become possible for your client software to provide a digital signature using a key that is uniquely yours. If and when that happens it will be possible for a receiving ISP to verify that the mail came, not just from your domain, but specifically from you. At that time you may need to think about upgrading your email software, but it's not at all likely that it will make writing and sending mail any different from the way you do it today.
© Pete Ford, 2005
How email works
Email is moved around the Internet by special servers called SMTP (Simple Mail Transfer Protocol) servers. They are also called MTAs (for Mail Transfer Agents), MDAs (Mail Delivery Agents) or simply Mail Relays. As far as anyone but an email system administrator is concerned, all these names are equivalent (and even for an email admin, the distinctions are subtle).
Let's say you write a mail to your friend Albert, whose email address is albert@example.com. When you click the "send" button, your email client program (Outlook Express, Thunderbird or whichever you happen to use) makes a connection to an MTA belonging to your ISP, and sends the mail to it. The information it sends contains your address (called the "From" address), the recipient address (Albert's email address, in this case), the body of the mail, any attachment files you added, and some other control information.
Now the ISP's MTA makes a connection to an MTA belonging to example.com, and sends the exact same information on. That MTA looks at the recipient address and sees that the mail is for the correct domain, and at that point it locates Albert's mailbox file and dumps the mail into it. When Albert reads his mail he sees that the message came from you because it has your email address as the From address.
How do the spammers exploit the email infrastructure?
The problem is that the MTAs take the From addresses that they get in mail as gospel truth. That means that I could make a direct connection to my ISP's MTA and create a mail using your email address as the From address. The person who receives the mail would think it came from you.
Spammers use this trick to label mail with bogus From addresses. Because the ISPs have no way to check that the mail really did originate where it says it did, they have to rely on filtering and other techniques to try to figure out which mails are spam. Every time the anti-spam propeller-heads come up with a better filter, it's not long before the spammers figure out a way to fool it, so it's an arms race with no end in sight.
What measures are being taken?
The industry is trying to fix the problem by coming up with ways for the MTAs to verify that each mail they receive came from the domain it claims in the From address. There are basically two approaches:
The simplest way is for every domain on the Internet to provide a public list of the IP addresses of all of its MTAs. When an MTA receives a mail it can identify the IP address of the system that sent it, then look up the list of valid IPs for the domain in the From address. If the connection came from an IP address that's in the list for the claimed domain, it passes the test. This method is called SPF (Sender Policy Framework) and is being used by AOL and others. There is a similar mechanism called SenderID, but for various reasons that I won't go into here this doesn't seem to be gaining much support at the moment and it's likely to be abandoned.
Another way is more complex but more reliable and secure. In this system each mail has a special label called a Digital Signature attached to it by the sending ISP's MTA. At the receiving end the signature can be checked, and if it checks out it proves that the mail came from the domain claimed in the From address. Without going into too much detail, the signature uses encryption to make sure that it could only have been created by the domain that claims to have sent it. This system is called DomainKeys and is being used by Yahoo!, Google, Earthlink and others.
How does this help?
If you can check a mail and verify the sender then two things are possible. If the checks fail, you know that the mail didn't come from where it claims to have originated. That mail can be put into a "suspect" mail folder, for example.
If the mail does check out, that still doesn't prove that it's not spam - there's nothing to stop a spammer setting up an IP list and/or signature software like the "legitimate" ISPs. But it does make it possible for your and my ISPs to build and share "reputation databases" containing lists of domains that are known to be regular spammers (and also domains that are known to be "good"). When your ISP gets a verified mail from a known spammer domain, they can mark the mail as probable spam so you can decide whether to even open it before you delete it.
As time goes by more ISPs and large-volume mail producers will implement these measures. The spammers that follow suit by providing verification data with their mail will have that mail marked as spam by the time it gets to your in-box. And the ones that don't will have their mail marked as suspect, since the sender can't be verified.
When will this happen?
It's already happening. AOL's servers are checking for SPF information today, and it won't be long before they start insisting on verification data. MSN and Hotmail aren't far behind. Yahoo! checks incoming mail for DomainKeys signatures right now and will be adding features to their web mail client to allow users to decide what to do with unverified mail sometime during the next three months. Google and Earthlink will be doing the same. It's very likely that all the major ISPs and most, if not all, of the smaller ones will have deployed the necessary software by the end of this year.
Is there anything you need to do?
For the time being, no - your ISP will be taking care of the details by making the necessary changes and upgrades to their servers. You won't need to change your email client and for the most part you won't even be aware of the shift - until you realize one day that the volume of spam you get is nothing like what it used to be.
Later it may become possible for your client software to provide a digital signature using a key that is uniquely yours. If and when that happens it will be possible for a receiving ISP to verify that the mail came, not just from your domain, but specifically from you. At that time you may need to think about upgrading your email software, but it's not at all likely that it will make writing and sending mail any different from the way you do it today.
© Pete Ford, 2005
Labels: Computers
posted by Pete Ford at
9:38 PM
0 Comments:
Post a Comment
<< Home