When this Domino admin installed new helpd esk software on her system she didn't read the documentation. As things started to go wrong, she found herself writing the book on the merits of following the vendor's instructions.
We had just gotten in some new help desk and inventory software, and in my experimenting with it (not reading the documentation, of course), I started running it on some of my servers. After carefully looking and seeing no ill effects, I went down the line and ran it on all our servers.
Unfortunately, the audit software came configured out of the box with a Y2K test intact. The audit did nothing toward to our Notes server, but our time server was an older machine about to be decommissioned. After the audit completed, it was left with the date set two months in the future, a fact not noticed by me at the time. Of course, all the other servers came along, picked up the new time and reset their clocks.
It took us about an hour to notice some domain account problems that led us to see that most of the server clocks had been reset. We needed to correct the problem. About that time, I read the audit documentation, which said quite explicitly, "Do not run on production servers."
Now Lotus will tell you that you should never set your Domino server to a date in advance for any kind of testing, because this will definitely screw up the server. And it did. We saw two symptoms: Internet mail exchange became erratic, and we were also experiencing the "non-existent or invalid document" message on sending mail that we'd see when one of our many NABs was corrupted (usually the index). Turning off the "exhaustively search all address books" in favor of "stop after first match" worked to quit triggering the problem, and was our saving grace.
Lotus support told us that to fix things, we had to replace every mail template in the system, and we did. I knew it wasn't going to help, because I moved one of the affected mail files to another server and it functioned fine, but I was desperate. We replaced every NAB on the server, and that didn't help.
Lotus support was stuck, but I had an inspiration and a backup plan. I ended up taking all the mail files off the server. I then rebuilt the Domino server from scratch. I retrieved new copies of the NABs. I put all the mail files back and ended up with a perfectly healthy Domino server. But it is a week that I'll never forget or live down. I still don't know what in the Domino environment had been hosed to this day, but will always feel relief that I got the server back up and running.
Do you have your own blooper? Send it in and claim your fame.
Every story in our bloopers series comes to us directly from a SearchDomino.com administrator, developer or consultant. For obvious reasons, some contributors -- including this tale's author -- choose to remain anonymous.
MORE ON THIS TOPIC:
Read all SearchDomino.com's true bloopers.
View our Best Web Links on enterprise integration.
