Writing more than 64k data to a single document

Domino only allows for writing 64k of data to standard fields.

While it is possible to write unlimited amounts of data to RichText fields, it is not possible to retrieve more than 64k from a RichText field programmatically. The reading procedure simply will stop at 64k. Writing to multiple standard fields probably will stop you with a "32k limit" error message when you are trying to save the target document. The standard solution as it is used in log.nsf is writing to multiple documents instead...

of just a single doc.

But assume you need to write big amounts of data to a single document programmatically, with the option to retrieve that data later on (again programmatically). The solution is to write arrays of data to multiple standard fields. These fields are created on the fly through your script code with the ISSUMMARY flag set to FALSE; if a field's size passes a given threshold value, the next field is created (for an example see 1. example code below).

You also need a second field recording the amount of dynamically created fields. The value stored here is used when retrieving the data at a later time (for an example see 2. example code below). When I tested this solution, I also tried various threshold sizes for the target fields. I observed big differences in the time my script needed to write the data depending on the threshold size. The result was that it takes less time to write many small fields than writing few big fields. In fact, the time needed seems to rise exponentially, while the total size of the resulting document is hardly growing at all. See example measurements block below.

The goal for this in my case is a data analysis performed on a very "lively" application on a regular basis. Of course it take minutes just to open the resulting analysis document because currently there are some 10,000 documents analyzed and logged to a single log document. Since data are structured the way they are, I can compare the log documents created at various times (such as for finding out document movement within a workflow application, tracking document failures etc.).


1. example code: writing

Set logdoc = targetdb.CreateDocument
logdoc.Form = "log"
Set logitm = New NotesItem (logdoc, "LogField" & Cstr (j), "")  '--> j is initialized as 0
     For i = 0 To 3000
          If logitm.ValueLength >= 5000 Then
               j = j + 1
               Set logitm = New NotesItem (logdoc, "LogField" & Cstr (j), "")  '--> IsSummary is set to FALSE by default
          End If
          Call logitm.AppendToTextList (strText)
     Next
     logdoc.NumLogFields = j  ' --> record number of fields created

2. example code: retrieving

for i = 0 to logdoc.NumLogFields (0)
  set logitm = logdoc.GetFirstItem ("LogField" & Cstr (i))
    Forall entry in logitm
      'entry represents a line that previously was written to the item; do whatever you need to do with it
    End Forall
  Next

3. Here are some example measurements for 3000 lines of text (150 characters each) written to text fields:

threshold size 5000 bytes:
number of fields = 94
time needed / write = 2 s
time needed / retrieve < 1 s
doc size = 476.298 bytes

threshold size 15000 bytes:
number of fields = 32
time needed / write = 8 s
time needed / retrieve < 1 s
doc size = 476.174 bytes

threshold size 35000 bytes:
number of fields = 14
time needed / write = 36 s
time needed / retrieve < 1 s
doc size = 476.138 bytes

threshold size 65000 bytes:
number of fields = 8
time needed / write = 113 s
time needed / retrieve < 1 s
doc size = 476.126 bytes
This was first published in July 2001

Dig deeper on Domino Resources - Part 6

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchWinIT

Search400

  • iSeries tutorials

    Search400.com's tutorials provide in-depth information on the iSeries. Our iSeries tutorials address areas you need to know about...

  • V6R1 upgrade planning checklist

    When upgrading to V6R1, make sure your software will be supported, your programs will function and the correct PTFs have been ...

  • Connecting multiple iSeries systems through DDM

    Working with databases over multiple iSeries systems can be simple when remotely connecting logical partitions with distributed ...

SearchEnterpriseLinux

SearchVirtualDataCentre.co.UK

Close