True Domino Blooper #7: AdminP worm unleashed

When this Lotus administrator built an administration utility, he knew better than to test it in a live environment, but...

When this Lotus administrator built an administration utility he knew better than to test it in a live environment. Unfortunately, the company that tasked him with the job didn't want to bother with a testing environment and ended up turning a worm loose that upset more than a few applecarts.


I worked at a company with over 200 servers, and each server STILL had a full replica of the Admin Requests DB on it. I was contracted to build an automated administration utility to register users, delete users, rename/recertify users and handle group modifications. During the testing phase of the application, I was having some problems getting AdminP to properly process requests. AdminP is very picky about how a request is formed; certain fields have to be signed, some fields have to be protected and all fields need the summary flag set for starters.

I didn't realize this, but the customer was testing the app in the live production environment. I was testing the app in a development environment with only one server. It turns out that if you send a malformed request to AdminP (I won't say what type of request), it causes every other server to generate the same request.

Then every server acts on every additional document created by every other server. And so on, and so on, and so on, and so on, and so on. I got an extremely panicked call from the customer telling me that AdminP was "over 400,000 documents and growing" and several sites had their servers crash due to the drives running out of space. Not to mention the fact that the link from the U.S. to Europe was totally saturated and several sites were down while their servers stayed busy replicating AdminP requests.

I actually thought it was kind of funny at the time because the customer was running it in production. Then I tried to help resolve the problem. Basically we had to shut down AdminP in the whole environment until all the documents could be deleted. We created a program document to kill AdminP and then they spent DAYS deleting requests and compacting Admin Request DBs and restarting servers.

So basically I created an AdminP worm without knowing it. The moral of the story is never, never, ever run something untested or unfamiliar in production!

 


Do you have your own blooper? Send it in and claim your fame.

 Every story in our bloopers series comes to us directly from a SearchDomino.com administrator, developer or consultant. For obvious reasons, some contributors -- including this tale's author -- choose to remain anonymous.

MORE ON THIS TOPIC:

Read all SearchDomino.com's true bloopers.
View our Best Web Links on Domino/Notes administration.

This was first published in July 2003

Dig deeper on Domino Resources - Part 2

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchWindowsServer

Search400

  • iSeries tutorials

    Search400.com's tutorials provide in-depth information on the iSeries. Our iSeries tutorials address areas you need to know about...

  • V6R1 upgrade planning checklist

    When upgrading to V6R1, make sure your software will be supported, your programs will function and the correct PTFs have been ...

  • Connecting multiple iSeries systems through DDM

    Working with databases over multiple iSeries systems can be simple when remotely connecting logical partitions with distributed ...

SearchEnterpriseLinux

SearchDataCenter

SearchExchange

SearchContentManagement

Close