This topic has come up a lot. Despite quite a bit of searching to research your question, I was unable to find a firm answer in an official Lotus publication. The best answer I could find, based on discussion at www.notes.net, is 64-bit RC4. Many people seemed to agree that this is the algorithm used. But I cannot promise that it is correct. Note that port encryption for Notes clients is different from securebrowser sessions with a Domino web server. The latter uses SSL.
Dig Deeper on Lotus Notes Domino Administration Tools
Related Q&A from Chuck Connell
Is it possible to encrypt a user's name before sending an email? SearchDomino.com expert Chuck Connell weighs in. Continue Reading
Learn how to change authentication timeout interval for Domino Web Access logins. Continue Reading
SearchDomino.com expert Chuck Connell provides a resource for a Lotus Notes administrator who wants to filter out email containing the word "spam," ... Continue Reading