This topic has come up a lot. Despite quite a bit of searching to research your question, I was unable to find a firm answer in an official Lotus publication. The best answer I could find, based on discussion at www.notes.net, is 64-bit RC4. Many people seemed to agree that this is the algorithm used. But I cannot promise that it is correct. Note that port encryption for Notes clients is different from securebrowser sessions with a Domino web server. The latter uses SSL.
This was first published in January 2002