When this Lotus administrator built an administration utility he knew better than to test it in a live environment. Unfortunately, the company that tasked him with the job didn't want to bother with a testing environment and ended up turning a worm loose that upset more than a few applecarts.
I worked at a company with over 200 servers, and each server STILL had a full replica of the Admin Requests DB on it. I was contracted to build an automated administration utility to register users, delete users, rename/recertify users and handle group modifications. During the testing phase of the application, I was having some problems getting AdminP to properly process requests. AdminP is very picky about how a request is formed; certain fields have to be signed, some fields have to be protected and all fields need the summary flag set for starters.
I didn't realize this, but the customer was testing the app in the live production environment. I was testing the app in a development environment with only one server. It turns out that if you send a malformed request to AdminP (I won't say what type of request), it causes every other server to generate the same request.
Then every server acts on every additional document created by every other server. And so on, and so on, and so on, and so on, and so on. I got an extremely panicked call from the customer telling me that AdminP was "over 400,000 documents and growing" and several sites had their servers crash due to the drives running out of space. Not to mention the fact that the link from the U.S. to Europe was totally saturated and several sites were down while their servers stayed busy replicating AdminP requests.
I actually thought it was kind of funny at the time because the customer was running it in production. Then I tried to help resolve the problem. Basically we had to shut down AdminP in the whole environment until all the documents could be deleted. We created a program document to kill AdminP and then they spent DAYS deleting requests and compacting Admin Request DBs and restarting servers.
So basically I created an AdminP worm without knowing it. The moral of the story is never, never, ever run something untested or unfamiliar in production!
Do you have your own blooper? Send it in and claim your fame.
Every story in our bloopers series comes to us directly from a SearchDomino.com administrator, developer or consultant. For obvious reasons, some contributors -- including this tale's author -- choose to remain anonymous.
MORE ON THIS TOPIC:
Read all SearchDomino.com's true bloopers.
View our Best Web Links on Domino/Notes administration.
