Dynamic array memory management

[Snorri B.]

Hi y'all and a happy new year!

I've been investigating memory usage in some of our modules.

Apparently, if you allocate memory with a dynamic array, there is no way to reclaim it. I have tried
let a = null
call a.clear()
loop and call a.deleteElement

But it seems that once memory is allocated, it never gets released, only reused if possible. Even if the array is a part of an object type, nulling the object will not release the memory. I've tried this with modal and local arrays, same result.

I find it particularly strange that destroying an object does not release all of it's resources.

What do fellow BDL-ers think?

Best regards,
-Snorri

[Rene S.]

Hello Snorri,
your observation is correct: "once memory is allocated, it never gets released, only reused if possible"

Array elements can not be released while the array is alive. This is an (incomplete) workaround to avoid crashes when dereferencing array elements used in INPUTs or used as host-variables in CURSORs.

Informix-4GL can create at runtime references to array-elements. Examples: INPUT array[index].*, DECLARE cursor-name CURSOR FOR SELECT ... INTO array[index].*. The runtime assumes: the referenced array element is alive. A better saying for "assumes" is "blindly trusts". For exactly this reason each array element once created can not be released.

Can not be released? There is an (unsafe) trick:

Code (genero) Select Expand


DEFINE a DYNAMIC ARRAY OF INT
...
CALL a.clear() -- does not release unused array elements
LET a = getEmptyArray() -- releases the old array (and all of its elements)
...
FUNCTION getEmptyArray()
    DEFINE r dynamic array of int
    RETURN r
END FUNCTION


DYNAMIC ARRAY is a reference type, for that reason the "old" array will be released when assigning a "new" one. Be careful. If elements of the "old" array are still referenced, then the program crashes when dereferencing the elements.

BTW: the semantics of DICTIONARY elements when removing them from a DICTIONARY is much better. DICTIONARY elements will be released when removing *and* not being referenced.

[Snorri B.]

Thanks Rene.

Our situation is a little bit more complex. We have an array with dimension 2 and we need to release memory for the last element.

Code (genero) Select Expand


DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF INT
...
FOR i = 1 TO 2
  FOR j = 1 TO 100
    LET a[i,j] = j
  END FOR
END FOR
CALL a[2].clear()
-- I Want to release this memory!


But I guess there is no way, is it?

Regards,
-Snorri

[Rene S.]

Code (genero) Select Expand


IMPORT util

MAIN
    DEFINE i, j INT
    DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF INT

    FOR i = 1 TO 3
        FOR j = 1 TO 10
            LET a[i, j] = j
        END FOR
    END FOR
    DISPLAY util.JSON.stringify(a)
    LET a[2] = getEmptyArrayOfInt() -- replaces the 2nd sub-array by an empty array
    DISPLAY util.JSON.stringify(a)
END MAIN

FUNCTION getEmptyArrayOfInt() RETURNS DYNAMIC ARRAY OF INT
    RETURN NULL
END FUNCTION



[Snorri B.]

Thanks again. It seems though that the memory the program uses is not released.

Code (genero) Select Expand



DEFINE pid INT

MAIN
   DEFINE i, j INT
   DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF CHAR(1000)

   LET pid = fgl_getPid()
   FOR i = 1 TO 5
       FOR j = 1 TO 10000
           LET a[i, j] = sfmt("Element %1",j)
       END FOR
   END FOR
   CALL showmem("After loop")
   FOR i = 5 TO 1 STEP -1
     LET a[i] = getEmptyArrayOfChar()
     CALL showmem(sfmt("After clear %1", i))
   END FOR
END MAIN

FUNCTION getEmptyArrayOfChar() RETURNS DYNAMIC ARRAY OF CHAR(1000)
   RETURN NULL
END FUNCTION

FUNCTION showmem(s STRING)
DISPLAY s
RUN SFMT("pmap %1 | tail -n 1 ", pid)
DISPLAY ("----------------------------------\n")
END FUNCTION



[Rene S.]

Hello Snorri,

It seems though that the memory the program uses is not released.

The memory has been allocated with malloc(). Memory allocated by malloc is owned by the process forever.

Malloc and free cause a kind of "heap fragmentation". A call to free() could "theoretically" release the memory by calling sbrk() if and only if no memory has been allocated with an address greater then the address of the memory to be freed. This is not case.

Java can release memory (theoretically?). The Java garbage collector moves any reachable memory to a new position (kind of memory defragmentation). This implies, memory can be released.

Rene

[Reuben B.]

I could probably do an Ask Reuben article on this topic.  From the days of my first Genero migration many years ago now, something along the lines of what Rene said  "The memory has been allocated with malloc(). Memory allocated by malloc is owned by the process forever." is what I recall.

Having done an aggressive transformation of static arrays to dynamic arrays, and chars to strings, we felt we were ahead.  However it was still prudent to put a cap on QBEs for report/enquiry programs, and similarly a limit on records in a zoom window etc so that a user did not select all records (1000+) from a table and eat up some memory that would be retained for the life of a program.  Also taking into account the expected lifetime of a program, most programs had lifetime of less than 5 minutes so not too big a problem if it has excessive memory use due to user input, but for the programs that ran all day e.g. menu, pos programs, a bit more important that an unchecked QBE does not end up holding memory unnecessarily all day.

One thing I am curious that I have never got round to looking at is whether in the case of a sparse array, are dictionaries better than dynamic array e.g.

Code (genero) Select Expand

MAIN
TYPE y_type DICTIONARY OF INTEGER
TYPE xy_type DICTIONARY OF y_type

DEFINE xy_value xy_type
     
    LET xy_value[100000][50000] = 100
    DISPLAY xy_value[100000][50000]
END MAIN

... so if you were worried about memory consumption of a dynamic array with dimension 2, is something like that better? at the cost of losing some array functionality.

[Snorri B.]

Thank you Rene and Reuben.

I think the only option I have right now is try to limit the number of rows QBE returns.
Usually this is not a problem, but some users really take things to the limit although they are being warned many times.

Best regards,
-Snorri

[Sebastien F.]

Hi Snorri,

I did not realize that your fetch rows from a database from a QBE.
I thought you need to load a lot of data into an array for some computation.

For sure with DB queries / QBE, you better limit the total number of rows that can be fetched.

To me it does not make sense to load thousands of rows, and let the user scroll in the list of in a TABLE for example!

The users should add better filter the query.

You may want to propose some automatic filter (when the QBE is empty), for large tables in your application.

Seb