On the Basics Tab of the Server Document, Domino 6.x offers the Domino Administrators some very useful fault recovery capabilities. You can run NSD to collect diagnostic information, you can have the server automatically restart itself after a fault/crash and finally you have the ability to e-mail a fault notification message to specific users. It is this notification that I would like to address.
As a Domino Admin, I find it highly important that I'm aware of a Domino fault/crash. In specific situations, it is very possible that your Domino server has faulted, automatically restarted itself and went totally undetected.
For example, it's Saturday morning 3am, you have the following field enabled in the Server Document "Automatically Restart Server after Fault/Crash." You also have the "Mail Fault Notification To" field populated with pager numbers, e-mail addresses, etc. You may even use third party monitoring software such as "What's Up Gold" or "Tivoli."
A fault occurs. Unfortunately, it is highly unlikely that your Domino Server will be able to route any type of e-mail notification during a fault/crash. Your e-mail fault notification option is now useless. Many third party monitoring tools are set to do what I call a Check One, Check Two, Check Three. If Check One returns a problem, the monitoring application will wait five minutes and then Check Two. If Check Two fails another five minutes goes by before Check Three. Only when Check Three fails is a notification generated (you get the picture).
Within this time period, your Domino Server may have faulted, restarted itself and gone totally undetected. My solution is to use a very powerful field that I would suspect is left empty on almost everyone's server documents. The Basics Tab, Fault Recovery Section, "Run This Script After Server Fault/Crash:" If used correctly, this field can be used totally independent of Domino to e-mail or page you a notification with regards to a server fault.
The key is to populate this field with non-Domino Script. Domino Script relies on the Domino Server -- if the Domino Server has just faulted, how can this script kick off? Instead, with the help of my trusty Delphi Developer, we built a self-executable file, when launched sends an e-mail to my pager and my mailbox via an outbound SMTP server that is independent of Domino.
I wrote a batch file that calls the self-contained exe file. Both files are placed on my Domino Server in the Data directory and my field value reads as follows: "D:\Lotus\Domino\LotusCrash.bat (This script must not run NSD)". When a fault occurs, LotusCrash.bat kicks off, activates the exe file and I'm paged immediately. This is for a Windows environment, obviously. However, the concept can be used on all operating systems.
Do you have comments on this tip? Let us know.
This tip was submitted to the SearchDomino.com tip exchange by member Scott Noebel. Please let others know how useful it is via the rating scale at the end of the tip. Do you have a useful Notes/Domino tip or code to share? Submit it to our monthly tip contest and you could win a prize and a spot in our Hall of Fame.