Perl Programmer/Consultant
Remote System Administrator
Free Software
... contact me
  while ($making_other_plans) { life(); }
  location('ipsstb', '2012', '08', 'index');

 For Web Designers 2021-07-04 05:32:50 UTC Mail Delivery Problems? 

Thursday, August 16 2012

Blacklisted! Part Three

Wherein we think about ways to get your mailing lists clean and keep them clean.

First let us define the characteristics of a clean mailing list:

  • All recipients are confirmed opt-ins
  • All addresses are deliverable (valid domains, no bounces)
  • No addresses are spamtraps
  • No addresses are temporary, time-expiring addresses
  • All opt-ins have been reconfirmed by user action within the past year

If you've read Part Two then most of those are pretty obviously vitally necessary.

Those of my clients who've had the greatest problems with blacklisting are those who've employed downright lousy bulk mailing programs, and most of them were of the mindset that throwing mail at as many people as possible was the key to increased sales. It's not, but it seems plausible enough.

There's more ...

→ committed: 8/16/2012 15:17:00

[ / e-commerce / bulkmail] permanent link

Comments: 0    Trackbacks: 0


Monday, August 06 2012

Bad Bots Must Be Punished.

I periodically look through my web server logs to pick out things that are not as they should be. You might recall from previous blog entries that I operate spam traps and so on -- last night I picked out of my server logs that some critter calling itself MJ12bot was going where no legitimate bots belong. But it's apparently trying to be a good bot because it leaves its calling card: - - [06/Aug/2012:01:28:42 -0600] "GET /robots.txt HTTP/1.0" 200 1247 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3;")"

So off I go to that URI, and find that the folks who run the thing have said "If you have reason to believe that MJ12bot did NOT obey your robots.txt commands, then please let us know via email..." And so I did.

We discussed the matter via email a bit, and it seems probable that their bot encountered some kind of network error when it tried to grab my robots.txt file. Not an error response from my server, but a failure to even contact my server. To my way of thinking, in a case like that a properly designed bot will try again to get that file, and will not crawl the site until it gets either the file or a verifiable 404 Not Found. Not MJ12bot, though. The network failure is treated as if it were a 404, and is taken to mean that the whole darn site is wide open to them. Here's what their guy Alex said:

Sadly it's very difficult for us to diagnose this case - as you can see from your logs our bot grabs robots.txt, so we are not intentionally breaking your directives, it's just if bot could not get robots.txt then it could not obey it :(

Huh? Your bot encounters a network error and that gives you license to crawl my site in violation of my terms of service? It seems to me that if you know your crawler is broken in that way, which you do now, and you continue to run it knowing it's broken in that way, then what you're doing is willful negligence and that makes it intentional.

No worries here. I've informed the folks behind the thing that their bot is no longer welcome here and any connections it makes will be considered trespass. The fun part? When their bot comes around it will not see my web site. It will instead see a very, very long joke that will be delivered very, very slowly. How slowly? From start to finish will take from an hour and a half to more than six hours.

If you've seen a bad bot in your logs and want to punish it in this way, feel free to hit my contact form to inquire about it. It's a freebie if all you need is the application itself and very minimal installation/configuration instructions. After all: Bad Bots Must Be Punished!

→ committed: 8/6/2012 17:36:54

[ / internet / web_weirdness] permanent link

Comments: 6    Trackbacks: 0


Blacklisted! Part Two

In our previous installment we established that being SMTP blacklisted sucks and that if you've been blacklisted you probably deserved it. I promised to explain to you how those SMTP blacklists work, and being a man of my word here is that explanation.

SMTP blacklists serve two very important purposes. The first is to keep spam out of the inboxes of human users who don't wish to be bothered by it. The second purpose is to conserve the resources of the email service provider so the spam doesn't consume any more network, processing, and storage capacities than are absolutely necessary. Those resources are, after all, costly, finite, and better used for purposes other than the delivery of mail no one wants to see anyway.

Modern SMTP blacklists collect and collate data from many sources, some of which you may never have considered. When you send email of any kind, whether bulk or individual, you are providing information to them. Always. Always, always, always. You decide what that information will be but not how they will use it, so it behooves you to take care about what that information says about you/your mail server/your domain name.

There's more ...

→ committed: 8/6/2012 00:17:33

[ / e-commerce / bulkmail] permanent link

Comments: 0    Trackbacks: 0


Saturday, August 04 2012


So, you ran a bulk email campaign that got your mail server blacklisted, eh? I don't envy you, fellow space traveler, don't envy you at all. SMTP blacklisting is unpleasant, and it can be costly.

I'd like to share some hard won wisdom with you. It's wisdom that you're going to come to eventually one way or another, at least in part, so you might just as well take the easy way and take heed of my words. You've already started on the course of learning the hard way -- you're smarter than to pursue that course all the way to the bitter end, aren't you?

Before we get into the meat of it, a brief aside: In this article I may make you uncomfortable by challenging your self perception or your concept of fairness. Know that I intend no insult or offense, and that I speak plainly only because I believe I would be doing you a disservice if I were to do otherwise. It's time to confront this SMTP blacklisting problem head on, solve it once and for all, and never have to deal with it again. That's what we both want, and the only way we're going to get it is to observe reality as it is rather than as we wish for it to be. No matter how you choose to come to the wisdom I'm sharing with you, you are going to be uncomfortable at times. You might as well get it out of the way right now and then get on with solving the problem.

We can, between us, solve your problem.

The top four things that I always hear from those whose servers are or have recently been blacklisted:

  1. I only send email to those with whom I have a provable and legitimate business relationship, customers and those who have contacted me in the past requesting product information.
  2. People can opt out of my mailing at any time. There are always opt-out links in the messages, and there's an opt-out form on my web site, too.
  3. All of my promotional mail is CAN-SPAM compliant.
  4. I am a legitimate businessman. I am not a spammer!

First things first: Let us review the definition of spam as it is accepted by most technically savvy individuals. Spam is unsolicited bulk email, typically though not strictly commercial in nature. This is the definition used by those who make the internet work. Please make note of the fact that this definition does not address how the sender came into possession of the email addresses in his list.

The second thing: The only thing being CAN-SPAM compliant gets you is freedom from prosecution in the United States for the bulk email you send. It does not obligate anyone anywhere to conduct or receive your internet communications.

Let us now apply the previous two paragraphs to the Top Four things that I hear from those whose servers have been blacklisted.

"I only send email to those with whom I have... a business relationship...", every time I've heard it, has meant that the speaker has sent bulk email to everyone whose email address he's acquired in the conduct of his business and most of them did not give their express consent to receive that bulk mail. Hmmm. Lack of express consent. Didn't ask or agree. Synonym: Unsolicited. Unsolicited bulk email, the very definition of spam.

"People can opt out..." That's what blacklisting is. It's those recipients of your unsolicited bulk email acting en masse to opt out in a way that does not leave you the option to disregard their will. In my experience, for every spam complaint sent there will be two to four opt-out requests, so overall most people do have a sense of fair play. But those who fire the spam complaint instead? They have evidence that you have no regard for the rules so they have no reason to believe that you will keep your word that opting out will stop your bulk mail. They didn't break the rules. You did.

Rules? But "all of my promotional mail is CAN-SPAM compliant". Again, all that gets you is freedom from legal prosecution and does not obligate anyone to conduct or receive your communications. The rules for gaining access to another's network are whatever they say they are, and most publish their rules for the world to see. The one rule that's most prevalent is Do Not Spam Our Users. Unsolicited bulk mail is spam, and you've sent it to their users. Of course they've blacklisted you.

"I am not a spammer!" Unless all of those people who receive your bulk email have asked you to send it to them, yes, you are. And yes, that blacklisting was an appropriate response whether you like it or not. People don't like receiving unsolicited bulk email. They react badly to it, and almost everyone I've ever discussed the matter with has told me the same thing: If some online vendor spams me once, I will never go back again. It's a simple enough concept, one that most of us employ without even thinking about it: Do not reward bad behavior.

Tangentially: Some years ago I was in a position that required me to attend lots and lots of seminars, several of them being on the topic of how customers behave. Somewhere in the neighborhood of 95% of the customers who have a problem with your company will never give you the opportunity to make it right. They just go away. Then they tell nine or ten of their friends about their bad experience -- and these days, some number of those friends are likely to be social media sites where tens, hundreds, or thousands will view the complaint. When sending unsolicited bulk email, that's the fire you're playing with.

In Part Two, we'll go into how blacklists decide what is and is not spam and who is and is not a spammer. If you've got a dirty mailing list, you're a spammer even if unwittingly so. We can fix this problem. Stay tuned.

→ committed: 8/4/2012 15:19:00

[ / e-commerce / bulkmail] permanent link

Comments: 0    Trackbacks: 0


Thursday, August 02 2012

The Blog Is Back!

Suddenly I've an Elton John song echoing between my ears. :-)

After a long hiatus I've brought my blog back to life. The world may never be the same again. It's changed in the past few minutes, even. All of the working clocks now show a newer time so clearly we have got a little further into the future.

I will be updating fairly regularly with my inconsequential thoughts on life, conducting business on the internet, doing the things I do, and whatever else might hold my interest long enough to be written about.

Be well, fellow space travelers!

→ committed: 8/2/2012 10:25:06

[ / administrivia] permanent link

Comments: 1    Trackbacks: 0

Save the Net

Creative Commons License

Project Honeypot Member

August 2012
Mon Tue Wed Thu Fri Sat Sun

By Month:

By category:


Served to at 05:32:50 GMT on Wednesday, August 4, 2021.