Perl Programmer/Consultant
Remote System Administrator
Free Software
... contact me
  while ($making_other_plans) { life(); }
  location('ipsstb', 'e-commerce', 'bulkmail', 'blacklisted_part_two');

 For Web Designers 2020-04-25 02:37:36 UTC Mail Delivery Problems? 

Monday, August 06 2012

Blacklisted! Part Two

In our previous installment we established that being SMTP blacklisted sucks and that if you've been blacklisted you probably deserved it. I promised to explain to you how those SMTP blacklists work, and being a man of my word here is that explanation.

SMTP blacklists serve two very important purposes. The first is to keep spam out of the inboxes of human users who don't wish to be bothered by it. The second purpose is to conserve the resources of the email service provider so the spam doesn't consume any more network, processing, and storage capacities than are absolutely necessary. Those resources are, after all, costly, finite, and better used for purposes other than the delivery of mail no one wants to see anyway.

Modern SMTP blacklists collect and collate data from many sources, some of which you may never have considered. When you send email of any kind, whether bulk or individual, you are providing information to them. Always. Always, always, always. You decide what that information will be but not how they will use it, so it behooves you to take care about what that information says about you/your mail server/your domain name.

For the rest of this discussion I'm going to assume that we're talking about a modern, fully featured SMTP blacklist implementation. I'm also going to assume a minimal technical background on your part so please excuse me if I overstate the obvious, or if you find yourself having to search the web to find definitions.

Every time your mail server makes a connection the remote mail server and so the blacklist gets your server's IP address and from that its domain name. Then it gets the name by which the mail server calls itself, the email address from which the mail is coming, and the email address of the recipient. Then comes the data, the "full email headers" and the body of the message.

All of that is fair game. You gave it to them willingly, and to some degree it's like negotiating with pirates. Again, you decide what information they will get, but they get to decide what they will do with it. You can't bitch -- it's the same way you treated those whose email addresses you acquired who never gave you permission to send them bulk email. Live by the sword, die by the unmanned aerial drone. You don't have to like it. Embrace The Suck, brother.

Now the parts you might not know about: Spam traps.

There are many different kinds of spam traps. Perhaps the very worst kind to inadvertently include in your mailing lists are those that are used solely to detect address harvesting web spiders. They're sprinkled around the web in ways that either prevent regular site visitors from seeing them at all, or are clearly marked as being what they are. Both kinds appear at the bottom of my contact page if you care to see a typical implementation -- don't send email to any of those addresses you might see there. Oh, and if you find my dynamically generated spider trap, it's designed to be slow and there are reasons for it. Among other things, thousands of web sites around the world have kindly volunteered to direct harvester bots to it and keeping the thing very slow prevents them from overloading my server.

Another form of spamtrap is the long-dead address that's resurrected as a spamtrap. A good candidate for this use is an account that's been closed for more than a year that's still bouncing lots of mail. A diligent ESP will audit the account for a while to ensure that what it's getting really is all spam before using it as a spamtrap. If you're running an inexcusably poorly designed bulk mailer that doesn't automatically unsubscribe undeliverable addresses, then you either already are or eventually will be sending to this sort of spamtrap.

Abandoned domain names make excellent spamtraps. The way they're implemented is as follows: Some domain that handled a lot of mail is abandoned by its owner, and picked up by the blacklist operator when it becomes available. The first, optional step is to set the domain up with DNS records establishing name servers for the domain but offering no mail exchanger or other service records. Properly designed bulk mailers will soon enough unsubscribe addresses in that domain for being undeliverable for too long. After some time, a mail exchanger is set up to reject all email delivery attempts with a 500 series permanent failure code ("hard bounce"). After several months (usually a year or more) of this during which all properly designed bulk mailers will have unsubscribed all email addresses in the domain, the mail server is reconfigured to accept all email regardless of whether or not the address once existed.

Temporary email addresses often become spamtraps. Some temporary addresses, such as those from, are easy to spot and prune from your mailing list, but others are completely undetectable. Similar to temporary addresses are site-specific addresses, which I employ myself and have for many years. A site-specific address is usually an alias to a primary address and is given to only one entity. If that entity misbehaves the address can easily be switched off, or, worse, made into a spamtrap. When I catch spam on a site-specific address it will always become a spamtrap. I'm a nice guy about it and notify the entity that the address has become a spamtrap so they can do whatever is necessary to avoid problems. If the entity is a vendor of some kind, I ask them nicely to remove all credit card and personally identifying information I've provided, and if they balk I demand it. I doubt that most comply even after assuring me that they will, but the evidence trail I create could come in handy if fraud occurs. And of course I take my business elsewhere even though it usually costs a little more.

Some blacklist operators perform content analysis of the messages that they process, and modern analysis tools are remarkably accurate. Their operation is well beyond the scope of our discussion, but it is wise to keep in mind that those systems learn as they go so if you tarnish your reputation today they'll remember you tomorrow. All of those spamtraps we just discussed? In addition to all of the IP address and domain name reputation mechanisms they feed, they also feed the mail they receive to the content filters which learn from it what spam looks like. Spam evolves over time and the content filters do, too. The spammier your mail looks to those filters, the more likely it is that they'll be sorted directly into users' Spam folders rather than into their regular inboxes.

You may have heard by now of DKIM (DomainKeys Identified Mail) and SPF (Sender Policy Framework), two technologies that the major ESP's ask you to use if you're sending bulk mail their way. SPF is one of my least favorite anti-spam measures because its suck cannot be engineered out. Use SPF if you're certain that it fits the ways your organization uses email, but recognize the limitations it will impose upon you. DKIM doesn't have the major suck factor from an engineering standpoint, and I heartily recommend that you use it BUT! DKIM is a knife that cuts both ways. It proves that the mail you send originated within your organization, which is very very good when you have a stellar reputation but very very bad if you have a negative reputation. If you develop a negative reputation when using DKIM you can't just change your IP address and start over -- your domain name is tarnished, too.

So: There are monstrous databases and sophisticated software tracking your every move, email-wise. Not only that, your email is being watched across domain names -- AOL, Comcast, Cox, Excite, Juno, Lycos, Mediacom (MCHSI), MSN/Hotmail, NetZero, Orange (in France), RackSpace, RoadRunner,, Yahoo, and others are all protected by and feeding data into the same systems. If your list is dirty you're going to be introduced to Cloudmark. If you set up complaint feedback loops you're going to be introduced to ReturnPath. If you don't set up complaint feedback loops you're going to get even more familiar with Cloudmark. You want complaint feedback loops.

Shameless plug: If you want to automate your complaint feedback loops so you might never have to see the complaint email, I can provide custom software for that. Feel free to contact me.

You might have noticed that Gmail isn't in that long list of major ESP's who are all hiding behind that common system. They don't have to be because they've got mountains of data, lots of really bright folks, and processing horsepower enough to do their own thing.

No major ESP expects that your mail will never provoke a single complaint -- they know as well as we do that some people love to complain. To get blacklisted, you have to cross some threshold. Good luck figuring out what that threshold might be. There are outfits like ReturnPath who'll give you a little bit of reputation information for free, and a lot more for a fee, and you really should know what your Sender Score or other similar metrics are because those numbers are going to determine how successfully your mail is delivered.

When users click those "Report Spam" buttons at those major ESP's, the data goes to the ESP and to the common systems they all use. Not only will your IP address and domain reputations be affected, that mail will make another pass through the content filters so they can refine their ideas of what spam looks like. This will increase the likelihood that your messages will be automatically sorted into users' Spam folders. Most users never check their spam folders unless they're looking for a message they suspect might have gone missing, and they're going to be focused on finding that message rather than reading yours and poking the "This Is Not Spam" button to salvage some small piece of your reputation.

It's very reasonable to assume that long before your mail server is blacklisted your mail is being automatically sorted into spam folders where most users never look. It's reasonable because the folks who actually know tell us that it's so.

So, what's the take away here? The big ESP's know what you're doing and if you're using a dirty mailing list it's not doing for you what you believe it's doing. All it's doing is getting you into trouble and losing more value every time you use it. The days of spewing bulk email successfully have drawn to a close. Clean up those lists and dedicate some resources to keeping them clean or they're just going to continue causing you pain and the pain will only increase over time.

In our next installment: Some common sense ways to get your lists clean and keep them that way.

→ committed: 8/6/2012 00:17:33

[ / e-commerce / bulkmail] permanent link

Comments: 0    Trackbacks: 0


Comments are closed for this story.


Trackbacks are closed for this story.

Save the Net

Creative Commons License

Project Honeypot Member

August 2012
Mon Tue Wed Thu Fri Sat Sun

By Month:

By category:


Served to at 02:37:36 GMT on Monday, May 25, 2020.