Tip

Writing more than 64k data to a single document

While it is possible to write unlimited amounts of data to RichText fields, it is not possible to retrieve more than 64k from a RichText field programmatically. The reading procedure simply will stop at 64k. Writing to multiple standard fields probably will stop you with a "32k limit" error message when you are trying to save the target document. The standard solution as it is used in log.nsf is writing to multiple documents instead of just a single doc.

But assume you need to write big amounts of data to a single document programmatically, with the option to retrieve that data later on (again programmatically). The solution is to write arrays of data to multiple standard fields. These fields are created on the fly through your script code with the ISSUMMARY flag set to FALSE; if a field's size passes a given threshold value, the next field is created (for an example see 1. example code below).

You also need a second field recording the amount of dynamically created fields. The value stored here is used when retrieving the data at a later time (for an example see 2. example code below). When I tested this solution, I also tried various threshold sizes for the target fields. I observed big differences in the time my script needed to write the data depending on the threshold size. The result was that it takes less time to write many small fields than writing few big fields. In fact, the time needed seems to rise exponentially, while the total size of the resulting document is hardly growing at all. See example measurements block below.

The goal for this in my case is a data analysis performed on a very "lively" application on a regular basis. Of course it take minutes just to open the resulting analysis document because currently there are some 10,000 documents analyzed and logged to a single log document. Since data are structured the way they are, I can compare the log documents created at various times (such as for finding out document movement within a workflow application, tracking document failures etc.).


1. example code: writing

Set logdoc = targetdb.CreateDocument
logdoc.Form = "log"
Set logitm = New NotesItem (logdoc, "LogField" & Cstr (j), "")  '--> j is initialized as 0
     For i = 0 To 3000
          If logitm.ValueLength >= 5000 Then
               j = j + 1
               Set logitm = New NotesItem (logdoc, "LogField" & Cstr (j), "")  '--> IsSummary is set to FALSE by default
          End If
          Call logitm.AppendToTextList (strText)
     Next
     logdoc.NumLogFields = j  ' --> record number of fields created

2. example code: retrieving

for i = 0 to logdoc.NumLogFields (0)
  set logitm = logdoc.GetFirstItem ("LogField" & Cstr (i))
    Forall entry in logitm
      'entry represents a line that previously was written to the item; do whatever you need to do with it
    End Forall
  Next

3. Here are some example measurements for 3000 lines of text (150 characters each) written to text fields:

threshold size 5000 bytes:
number of fields = 94
time needed / write = 2 s
time needed / retrieve < 1 s
doc size = 476.298 bytes

threshold size 15000 bytes:
number of fields = 32
time needed / write = 8 s
time needed / retrieve < 1 s
doc size = 476.174 bytes

threshold size 35000 bytes:
number of fields = 14
time needed / write = 36 s
time needed / retrieve < 1 s
doc size = 476.138 bytes

threshold size 65000 bytes:
number of fields = 8
time needed / write = 113 s
time needed / retrieve < 1 s
doc size = 476.126 bytes

This was first published in July 2001

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.