01 April 2008

The "S" in SMTP is a horrible, horrible lie.

3 comments:

あじ said...

You have to take in the whole statement: Simple Mail Transport Protocol. The transport protocol is simple; sadly, that's the only thing about e-mail systems that is simple. Which is like saying barking or grunting is simpler than a formal language such as English. Less complexity, but also less capability.

Percussivity said...

Did you check to see if port 25 was open??

...(kidding)

SQLFunkateer said...

Hahaha, yes, to be sure.

What actually happened was the following. Single server, running (heaven help us) Small Business Server 2003.

Monday, 7PM: From the hospital room I remote in and decide to dig into a problem I was having with Sharepoint services not emailing alerts out. After much research on the web, I found a MS kb article that spoke of a SMTP service entry in the IIS Manager. I noted it was not listed, and the article said it was not installed by default. At this point I make my bloomer...I follow the instructions to install it (Add Remove Windows Components, Application Server, IIS, SMTP Service). It already appears to be installed. Here I do some very inadvisable improvising, and assume that it is somehow corrupted and so I uninstall it, and then reinstall it. Halfway through reinstalling it I get, quite predictably, a message stating that I need to insert CD 1 of the SBS CDs. The sound of my head striking my keyboard resonated throughout the halls of Centerpoint Medical Center, as you can imagine. Exchange was officially broken at this point...none of the Exchange services could be successfully started. I call my boss, let him know the situation and I let him know I'll be in first thing.

Tuesday, 5AM: I get up at the crack of dawn (funny thing about sleeping in a foldout chair bed, one's ability to rise early is quite enhanced compared to more comfortable accomodations) and I get in to work at about 5:45. I finish the install with the correct disc, Exchange fires up beautifully, I bash out a quick email to my boss, and I'm out of there by 6AM, two hours before we open for business. I get home and have a bite of breakfast, and decide to doublecheck some things remotely. At this point I realize that NO email is going out or coming in. It is officially hosed...incoming and outgoing email appears to go through but vanishes into a dark hole of despair. I try without success to fix it, and consider the nuclear option of using a LiveState Recovery backup (losing yesterday's emails), and I then realize that I did not stop the last night's LiveState job which copied over that of the previous night. So, no good system state backup to roll back to anyway. I am outclassed. I call up an IT vendor of ours and get them on the case, and I quickly shower and get ready for work.

At about 8:30 I hear from the engineer...I lay out the specifics of the issue, and he starts working (remotely). I call hourly for updates but he gets nowhere. At noon I hear from the company again; they are sending out their main service manager to work onsite. After 2 hours, he gives up and calls Microsoft. After another 2 hours, they make an initial breakthrough...they fix the system so that sending emails both externally and internally works. I can't describe the elation of that moment, seeing emails flying about, working again. But still a problem...emails coming in from external sources (we still, for the moment, are using the MS POP3 Connector to a webhost server) are getting bounced. The Microsoft guy on the phone continues poking around for about an hour, no success. I get impatient and start Googling, and eventually find the problem...the email NDRs reference that the email address is mspop3connector.*username*@*domain*. The prefix mspop3connector is getting tagged on and is somehow breaking the delivery to Exchange mailboxes. The solution is running imbreg.exe, and then resetting IIS with IISRESET. I bring this to them, and I think the pride of the MS guy is such that he wants to tinker around a bit more and ignore it...the consultant sees the logic, and goes ahead and does it. Bingo, it works. It is 6:15. Time to get back to the hospital and NEVER EVER AGAIN try to uninstall the SMTP service!

As for the cause of the problem, here is how I understand it. Exchange was service-packed, but when I reinstalled the SMTP service, it was an older version, pre service pack. That was the gist of the issue, although it took quite a while to sort out.

Time to take my spare x64 box, and put Exchange 2007 on it. SBS is nice, but all those different technologies on one box inevitably can lead to such horrific collisions. And again, let me just say, no more experimental reinstallations of key services on a production server for ME!