Domino Web traffic analyzers

Chuck Connell offers his product comparisons and recommendations on Web log traffic analyzers for Domino.

View member feedback to this tip.

Chuck Connell
Lotus Domino can act as an HTTP Web server. As with all Web servers, Domino produces log files of user traffic. And like all Web log files, the results contained in the logs are nearly indecipherable to the human eye. This problem is even greater for Domino servers than for other Web servers, because many Domino URLs contain long hexadecimal strings rather than normal directory and file names. Therefore, it is almost a requirement to use an analysis tool to understand and interpret Domino Web logs.

At the request of one of my customers, I researched the availability of Web log analysis for Domino and found five primary options, although there are certainly other choices as well.

  • WebTrends Log Analyzer -- This product, from NetIQ, is the dominant one among Web traffic analyzers. Participants of the Notes.net discussion groups report good results and satisfaction with its reports. Version 8.1 is the most recent release, and it is available for download (purchase or trial) from WebTrends.com. The purchase price is $495 for the base product on a single server, plus $89 per year for software maintenance. Other, more advanced, log analyzers from WebTrends provide additional customization and features, for higher prices.

    WebTrends is certainly a high-quality product; however, it has its limits when used with Domino log files; it does not translate Domino hexadecimal URLs into their meaningful equivalent. Consider the following two Domino URLs, which are equivalent:

    Products.nsf/f67f31f8faa31a2285256dd6007055dd!OpenView

    Products.nsf/ProductName!OpenView

    When reading a report about Domino Web traffic, you really want to see the second url listed above. But this requires the analysis tool to translate each raw Domino URL into its readable alternative. WedTrends (and many other log analyzers) does not do so for Domino log files.

  • Workflo!Log Analyzer -- This product from Workflo Systems provides many of the standard aggregation and statistical reports as WebTrends, with the key difference that it translates Domino URLs to their readable form. The traffic reports will show that the view named ProductName was accessed, rather than the view with the ID f67f31f8faa31a2285256dd6007055dd.

    The price of this product is $500, plus $250 per year for maintenance and upgrades beyond the first year. Workflo!Log Analyzer can also create various kinds of customized reports, which are only available with more expensive "enterprise" versions of WebTrends. The software and a demonstration movie are available at WFS.com.

  • Webalizer -- Webalizer is a free open-source log analyzer that runs on the Linux operating system. It is not geared specifically to Domino Web traffic, but will work with Domino since Domino produces raw log files in an industry-standard format.

  • Sandbox -- There is a Sandbox area of Notes.net, which contains free software submitted by participants of the Notes.net discussions boards. One of the Sandbox downloads is a Domino Web log analyzer named DomLog1.ntf. Several users of Notes.net reported they did not find this software to be satisfactory, however, and posted comments such as "Needs a lot of work to be useful" and "You can use the views as a place to start, but still requires more customization." So I do not recommend this solution, except as a point of departure if you want to create your own log views.

  • Custom-developed analyzers -- As with any software project, it is always possible to create a new application from scratch to solve the problem. Even though I make part of my living doing this for Domino-based organizations, I do not recommend doing so in this case. There are at least three mature products, cited above, that provide good-quality analysis of Domino Web log files. All three of these products are available for reasonable prices and cost less than a custom solution would cost.
My recommendations:
  • If price is important, I recommend Webalizer. It is virtually free to create a separate Linux computer to run this software. You can host it on an out-of-date PC, with the SUSE 9.0 Personal version of Linux ($40). The Webalizer application itself is free.

  • If you don't mind spending $500 for this project, I suggest Workflo!Log Analyzer. This is the only product I could find that is specifically designed to analyze log files from a Domino Web server, and it appears to do so in an impressive manner. As with any product comparison, I expect there are some features where WebTrends wins against this product, but Workflo!Log Analyzer is worth a serious look if you are in the market for a Domino Web log viewer.
If anyone is aware of other Domino Web log analyzers that also perform automatic translation of Domino hex URLs (to their readable equivalent) please let me know.


Chuck Connell is president of CHC-3 Consulting, which helps organizations with all aspects of Domino and Notes.


MEMBER FEEDBACK TO THIS TIP

You asked about products that translated view IDs into names -- sorry, I don't know. To be honest, I'm more interested in databases than individual views/pages etc. But a standard product that I think you might include in your list is available from Analog.cx -- it works well with Domino text Weblogs and apparently works in 32 languages on lots of OSes.

—Mark M.

******************************************

Thanks for writing. You are right. That is actually the Web reporter that my own ISP uses. I should not have overlooked it!

—Chuck Connell, tip author

******************************************

We use WebTrends Reporting Centre...a big leap from the basic log analyzer tool (in cost as well).

It has a feature called "URL Search and Replace" (which we actually haven't used yet), but it looks like you could configure it to recognize a pattern via a regular expression back to something simpler. Sounds like it could work for this issue.

From the help file...

URL Search And Replace*

Related Topics:
About Advanced Feature Definitions
Using URL Search And Replace To Group Pages



Each entry in the web server log contains a field that references the URL of the page or file that was accessed. Reporting Center uses this field to track activity on the web site, and counts each unique URL as a separate page or file. However, some web server plug-ins or shopping cart software add a unique string to the page name in order to track visitors and their shopping carts. In these situations, every hit to such a page will appear in the log as a unique URL (containing the ID of the visitor). This prevents accurate counting of hits to these pages.

URL Search and Replace allows you to delete specific identifiers from URLs or replace all unique identifiers with a common string. Doing so restores accuracy to page counts, makes your reports more informative, and avoids some out of memory conditions when the number of unique IDs is extremely high.

To create a new URL search and replace definition:

  1. From the links at the top of the WRC AdminConsole, click Advanced Features.

  2. Select the URL Search And Replace tab and click Add.

  3. In the Name text box, type the name of this URL Search and Replace definition. This is used to identify it in the definitions list and in reports.

  4. From the Replace From drop-down list, select the starting point for the replacement. Enter the string to match in the next text box. You can specify multiple strings by separating them with a pipe character (|). The pipe is interpreted as an "OR"--so if it finds any of the strings listed, it will replace it with your text. The search is not case sensitive.

  5. From the Up To drop-down list, select the ending point for the replacement and enter the text that indicates the end of the replacement string.

  6. In the With text box, type the text to replace the text you specified in Replace From.

  7. If desired, select the Perform Replacement Only If The URL Contains check box, and specify a value to match. The text is replaced only if the URL contains this value.

  8. Use the Test Area to see how the replacement definition works before running an analysis. Enter a URL to test. The Resulting URL text box displays the URL after the replacement is performed. Text That Was Replaced displays the text that was removed from the URL.

  9. Click OK to close this dialog box.
*Note: If you have not been granted full administrative rights in WebTrends Reporting Center, or if you are a WebTrends Reporting Service or Log Analyzer user, you may not be able to configure or use this feature. Please see your administrator if you have any questions.

—Michael C.

******************************************

Thanks for writing. Yes, the more expensive products from WebTrends do include this "URL alias" feature. The WFS product does, too, in its base version. The problem with this as a Domino solution, though, is that you have to manually set up the mapping between the complex hex URLs and their meaningful name. Every time you add a new document to a Domino database, you have to add its URL/name mapping to the alias list. That's why you want a Domino Web log analyzer to do this automatically.

—Chuck Connell, tip author

******************************************

Here is another product that we use to analyze the Domino Weblogs. I'm not to sure about translating hex-URLs to readable URLs, because we use an older version of the product. Here's a quote from their Web site:

Domino URL Analysis WebAbacus Professional Edition will turn Domino URLs back into meaningful data which matches the structure of the documents in your Domino-based content management system.

—Richard C.

******************************************

I read your article on the above with interest. I did develop a free utility, Beaver, that converts Domlog documents into W3C-compatible data. I prefer to use Domlog, as it's easier to manage the data from within Domino than it is to manage all those text files on servers. For more information click here.

Anyway, inspired by your article, I'm now working on getting Beaver to translate those horrible URLs to something more meaningful. It's not always straightforward as the title of a document may be in a field called "subject" or some even less obvious field, but I'll see how far I can get.

There is also something related on OpenNTF.org, but last time I looked it wasn't up to much.

—David C.

******************************************

I use AWStats for analyzing my Domino Web traffic. It is an open-source project written in Perl that runs as a CGI executable. Since it is Perl, it will run on any platform that supports the Perl runtime, and it has configuration examples for the various Domino server versions.

The only thing that I haven't been able to get to work correctly is a plug-in that assigns geo-locations to IP addresses. Everything else works like a champ. You can learn more about AWStats at AWStats.org.

—Andy B.

******************************************

I enjoy reading your tips and newsletters and expert advice. I just had one quick note on Webalizer. There is a Windows port of the product available so you can run it without Linux, which may be prohibitive for some shops. It's on the downloads page as "Win32 - Intel" pointing here.

—Adrian C.

******************************************

Chuck, some time ago we tested WebTrends and as I recall there was a feature where the tool would open each link to lookup the <title> tag to put some meaningful information in the Web statistics. I find it the only practical solution to "rename" the Domino unique IDs, because you would never know in which field in the Notes document the page title is. But in the HTML output, you should expect the title tag to be something meaningful.

—Markus K

******************************************

I have used WebTrends for a while and I have been annoyed by the same thing -- the unreadable URLs in the reports. You can use the Title tag but it doesn't give you much flexibility. In WebTrends you can make scripts that are supposed to change the actual log-file data before analyzing it. If anyone has examples of scripts like this, I would be interested.

My idea is to export a comma-separated textfile with the [universalID],[Subject?], each night and the have the script do the conversion in the actual log file. This could work.

In WebTrends there is already the function to use an external file for translation, but this only works on parameters so you can only do translations if the URL has a "?" and a "= " after the parameter. It is a nice function but I can't understand why you can't choose an external file in the URL replacement section.

—Fredric B.

Do you have comments of your own? Let us know.


This was first published in March 2004

Dig deeper on Lotus Notes Domino Administration Tools

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchWindowsServer

Search400

  • iSeries tutorials

    Search400.com's tutorials provide in-depth information on the iSeries. Our iSeries tutorials address areas you need to know about...

  • V6R1 upgrade planning checklist

    When upgrading to V6R1, make sure your software will be supported, your programs will function and the correct PTFs have been ...

  • Connecting multiple iSeries systems through DDM

    Working with databases over multiple iSeries systems can be simple when remotely connecting logical partitions with distributed ...

SearchEnterpriseLinux

SearchDataCenter

SearchExchange

SearchContentManagement

Close