Monitoring scheduled agents by email

VIEW MEMBER FEEDACK TO THIS TIP

In one of our larger applications, there are a number of critical scheduled agents that update data in the application from a back-end relational database. If the connection is lost, or the Lotus Notes database is down, the agents don't run to completion. Agent logs report all activity, but you have to open the logs to check for problems.

Recently the data server failed for an extended period and the logs weren't checked. This led to no updates for 48 hours, until the users noticed and reported the problem to our help desk.

It wasn't a problem to fix the connection and get everything running. But the situation prompted me to write some basic code to e-mail me if the Notes/Domino agents I was most interested in had not run for more than 12 hours.

Monitoring scheduled agents by e-mail involves a procedure that you insert at the end of each agent you want to monitor. This updates a keyword document in the application. You then add another scheduled agent, which runs every 12 hours/every day ,and checks when the agents last ran. If the agents haven't run, then an e-mail is sent to the administrator advising them to check the application and activity logs to see what is going on.

I have included the code for the subroutine which used the existing application keyword form, and a simple agent to check that at least one of the agents have run in the last 12 hours.

 
 Sub logAgentCompletion(agentName As
 String)
 Dim s As New notesSession
 Dim db As notesDatabase
 Dim view As notesView
 Dim doc As notesDocument
 Dim item As NotesItem
 
 Set db = s.currentdatabase
 Set view = db.getView("vAllKeywords")
 Set doc = view.getDocumentByKey("Last
 Run of " & agentName, True)
 If Not doc Is Nothing Then
    ' Update the last agent run date and the 
comments field with the agent name 
  doc.values = Now
  doc.save True, False
  
 Else ' If the document can't be found, create another
  Set doc = db.createDocument
  doc.form = "Keyword"
    ' Have to use ReplaceItemValue because
 doc.Key is dissallowed (reserved word) 
  Set item = doc.ReplaceItemValue( "Key",
 "Last Run of " & agentName)
  doc.values = Now
  doc.save True, False
 End If
 
End Sub

The Agent that scans the agent keyword records:

Sub Initialize
 Dim s As New notesSession
 Dim db As notesDatabase
 Dim view As notesView
 Dim collection As notesDocumentCollection
 Dim doc As notesDocument, mailDoc As
 notesDocument
 Dim checkDate As New notesDateTime(Now)
 Dim lastRunDate As notesDateTime, docDate 
As notesDateTime
 Dim msgStr As String
 
 Call checkDate.AdjustHour(-12)
 Set db = s.currentDatabase
 Set view = db.getView("vAllKeywords")
 Set collection = view.getAllDocumentsByKey(
"Last Run of")
 If collection.count > 0 Then
  Set doc = collection.getFirstDocument
  While Not doc Is Nothing
   If Isdate(doc.values(0)) Then
    Set docDate = New notesDateTime
(doc.values(0))
    
      ' Find the most recent date that the
 tracked agents ran 
    If Not lastRunDate Is Nothing Then
     If docDate.lsLocalTime > 
lastRunDate.lsLocalTime Then
      Set lastRunDate = New notesDateTime
(doc.values(0))
     End If
    Else
     Set lastRunDate = New notesDateTime(
doc.values(0))
    End If
   End If
   
   Set doc = collection.getNextDocument(doc)
  Wend
  
    ' Send message to system administrator if n
o activity in last 12 hours 
  If lastRunDate.lsLocalTime < checkDate.
lsLocalTime Then
   Set mailDoc = New NotesDocument(db)
   mailDoc.Form = "Memo"
   mailDoc.SendTo = "Andrew Broxholme/Sutton
/Security/UK/Corporate"
   mailDoc.Subject = "Alert - OMS Agent's are not 
running"
   mailDoc.body = "The last time that one of the
 tracked OMS agents ran was " & 
lastRunDate.lsLocalTime
   mailDoc.SaveMessageOnSend = False
   Call mailDoc.Send(False)
   
  Else ' Temporary code, can be commented 
out once we know tracking is working OK
   Set mailDoc = New NotesDocument(db)
   mailDoc.Form = "Memo"
   mailDoc.SendTo = "Andrew Broxholme/Sutton/
Security/UK/Corporate"
   mailDoc.Subject = "OMS Agent's running are OK"
   mailDoc.body = "The last run time for the tracked 
OMS agents ran was " & lastRunDate.lsLocalTime
   mailDoc.SaveMessageOnSend = False
   Call mailDoc.Send(False)
  End If
  
 Else ' If no keyword tracking documents then 
send alert
  Set mailDoc = New NotesDocument(db)
  mailDoc.Form = "Memo"
  mailDoc.SendTo = "Andrew Broxholme/
Sutton/Security/UK/Corporate"
  mailDoc.Subject = "Alert - OMS Agent's are not running"
  mailDoc.body = "No agent activity keywords 
detected in " & db.title
  mailDoc.SaveMessageOnSend = False
  Call mailDoc.Send(False)
 End If
End Sub
  

MEMBER FEEDBACK TO THIS TIP

If the agent is in the database, what would monitor whether this agent is running or not? I "tweaked" it a little to remove the keywords, run against all enabled scheduled agents, and e-mail a list of the agents that were not running. Now I can use the same agent in my databases. Here is my new code:

 'Send Notices of agents not running: Option Public 
Option Declare Dim db As notesDatabase Dim mailDoc As 
notesDocument Dim lastRunDate As notesDateTime, docDate 
As notesDateTime Dim agentList As String, agentName As 
String, okAgentList As String Dim aTrigger As Integer 
Dim aLastRun As Variant Dim compareDate As 
Variant Sub Initialize Dim s 
As New notesSession Set db = s.currentDatabase 
Forall a In db.Agents 'only process for Scheduled 
enabled agents aTrigger = 
a.Trigger If aTrigger = 1 And a.isenabled Then '1 
is Scheduled agents 
'now test when it was last run aLastRun = 
a.LastRun 'variant 'if it hasn't run for more than 24 hours, 
include in email compareDate = Today If 
compareDate - aLastRun > 1 
Then agentList = agentList + Chr(13) + 
Cstr(a.Name) + " " + Cstr(a.LastRun) 
Else 'otherwise, add to list of agents that are running 
ok okAgentList = okAgentList + Chr(13) + 
Cstr(a.Name) + " " + Cstr(a.LastRun) 
End If End If End Forall If agentList <> "" 
Then 'send email that agents aren't running 
Set mailDoc = New NotesDocument(db) 
mailDoc.Form = "Memo" mailDoc.SendTo 
= "Your name here" mailDoc.Subject = "Alert - " & db.
Title & " agents are not running" mailDoc.body = 
"The last time dates of the 
tracked agents are: " & Chr(13) & agentList mailDoc.
SaveMessageOnSend = False Call 
mailDoc.Send(False) End If If 
okAgentList <> "" Then 'send email of list of 
scheduled agents that are running ok 
Set mailDoc = New NotesDocument(db) 
mailDoc.Form = "Memo" mailDoc.SendTo = 
"Your name here" mailDoc.
Subject = db.title & " agents running are OK" 
mailDoc.body = "The scheduled agents have
 all run in the last 24 hours. " 
& Chr(13) & okAgentList 
mailDoc.SaveMessageOnSend = False Call mailDoc.
Send(False) End If ' Messagebox( agentList ) End Sub

—Pat C.

******************************************

It's an interesting idea to have an agent check if they are enabled or have been turned off using agent properties, which might occur if the application design was updated from a template with agents turned off. However, it doesn't solve the problem that I designed the process for, because it won't tell you when an agent failed to run to completion because of a run-time processing error. This is why the code that updated the keyword is being placed in the very last lines of the script.
—Andrew Broxholme, tip author

******************************************

Just a thought about this tip: What if there is a problem running scheduled agents on the server? It could be a problem on the server, changes in access rights for the agent signer, or any number of things. If the agent doesn't run then there is no mail.
—Erik R.

******************************************

It could be a problem where agents are not running at all, but system administrators should detect this. This process is designed to monitor a single application, which has core processes that need to run to maintain the application's integrity. If you need to monitor an entire server, you could have servers monitor one another.
— Andrew Broxholme, tip author

******************************************

We have designed a fairly extensive system of error reporting that is fed by all Lotus Notes Domino agents -- whether they are scheduled or triggered. The agents create a custom "log" document and start with a "failed" state. If they run successfully, they change the status to "Success," and the Lotus Notes document gets deleted. But if there is any error with the execution (system error, data error, rights problem, etc.), then it is reported on this log document, along with detailed information (error #, line #, user, server stats, etc.). This method has worked quite well.

— Nadeem M.

Do you have comments on this tip? Let us know.

This tip was submitted to the SearchDomino.com tip library by member Andrew Broxholme. Please let others know how useful it is via the rating scale below. Do you have a useful Lotus Notes, Domino, Workplace or WebSphere tip or code snippet to share? Submit it to our monthly tip contest and you could win a prize.