Talk:How-To Improve Large String Performance Using Buffers

If the final destination for the data (BUFFER in this example) is a file, I tend to use WRITESEQ APPEND to directly send the data there. In my application it would be unusual to store a 12MB record in a hashed file..

The opposite problem is having a large string that you need to process. I convert any delimiters to @AM or @VM, then use the REMOVE trick to make stepping through the string orders of magnitude faster. E.g.

READ BUFFER FROM .. LOOP

 REMOVE NEXT.VAL FROM BUFFER SETTING MORE

WHILE MORE DO

 GOSUB PROCESS.NEXT.VAL

REPEAT

instead of

READ BUFFER FROM .. MAX=DCOUNT(BUFFER,@AM) FOR I=1 TO MAX

 NEXT.VAL=BUFFER
 GOSUB PROCESS.NEXT.VAL

NEXT I