Tip

True Domino Blooper #7: AdminP worm unleashed

When this Lotus administrator built an administration utility he knew better than to test it in a live environment. Unfortunately, the company that tasked him with the job didn't want to bother with a testing environment and ended up turning a worm loose that upset more than a few applecarts.


I worked at a company with over 200 servers, and each server STILL had a full replica of the Admin Requests DB on it. I was contracted to build an automated administration utility to register users, delete users, rename/recertify users and handle group modifications. During the testing phase of the application, I was having some problems getting AdminP to properly process requests. AdminP is very picky about how a request is formed; certain fields have to be signed, some fields have to be protected and all fields need the summary flag set for starters.

I didn't realize this, but the customer was testing the app in the live production environment. I was testing the app in a development environment with only one server. It turns out that if you send a malformed request to AdminP (I won't say what type of request), it causes every other server to generate the same request.

Then every server acts on every additional document created by every other server. And so on, and so on, and so on, and so on, and so on. I got an extremely panicked call from the customer telling me that AdminP was "over 400,000 documents and growing" and several sites had their servers crash due to the drives running out of space. Not to mention the fact that the link from the U.S. to Europe was totally saturated and several sites were down while their servers stayed busy replicating AdminP requests.

I actually thought it was kind of funny at the time because the customer was running it in production. Then I tried to help resolve the problem. Basically we had to shut down AdminP in the whole environment until all the documents could be deleted. We created a program document to kill AdminP and then they spent DAYS deleting requests and compacting Admin Request DBs and restarting servers.

So basically I created an AdminP worm without knowing it. The moral of the story is never, never, ever run something untested or unfamiliar in production!

 


Do you have your own blooper? Send it in and claim your fame.

 Every story in our bloopers series comes to us directly from a SearchDomino.com administrator, developer or consultant. For obvious reasons, some contributors -- including this tale's author -- choose to remain anonymous.

MORE ON THIS TOPIC:

Read all SearchDomino.com's true bloopers.
View our Best Web Links on Domino/Notes administration.

This was first published in July 2003

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.