Print Page - Dynamic array memory management

Title: Dynamic array memory management
Post by: Snorri B. on January 18, 2021, 05:37:44 pm

Hi y'all and a happy new year!

I've been investigating memory usage in some of our modules.

Apparently, if you allocate memory with a dynamic array, there is no way to reclaim it. I have tried
let a = null
call a.clear()
loop and call a.deleteElement

But it seems that once memory is allocated, it never gets released, only reused if possible. Even if the array is a part of an object type, nulling the object will not release the memory. I've tried this with modal and local arrays, same result.

I find it particularly strange that destroying an object does not release all of it's resources.

What do fellow BDL-ers think?

Best regards,
-Snorri

Title: Re: Dynamic array memory management
Post by: Rene S. on January 19, 2021, 09:15:49 am

Hello Snorri,
your observation is correct: "once memory is allocated, it never gets released, only reused if possible"

Array elements can not be released while the array is alive. This is an (incomplete) workaround to avoid crashes when dereferencing array elements used in INPUTs or used as host-variables in CURSORs.

Informix-4GL can create at runtime references to array-elements. Examples: INPUT array[index].*, DECLARE cursor-name CURSOR FOR SELECT ... INTO array[index].*. The runtime assumes: the referenced array element is alive. A better saying for "assumes" is "blindly trusts". For exactly this reason each array element once created can not be released.

Can not be released? There is an (unsafe) trick:

Code

DEFINE a DYNAMIC ARRAY OF INT
...
CALL a.clear() -- does not release unused array elements
LET a = getEmptyArray() -- releases the old array (and all of its elements)
...
FUNCTION getEmptyArray()
    DEFINE r dynamic array of int
    RETURN r
END FUNCTION

DYNAMIC ARRAY is a reference type, for that reason the "old" array will be released when assigning a "new" one. Be careful. If elements of the "old" array are still referenced, then the program crashes when dereferencing the elements.

BTW: the semantics of DICTIONARY elements when removing them from a DICTIONARY is much better. DICTIONARY elements will be released when removing *and* not being referenced.

Title: Re: Dynamic array memory management
Post by: Snorri B. on January 19, 2021, 11:23:47 am

Thanks Rene.

Our situation is a little bit more complex. We have an array with dimension 2 and we need to release memory for the last element.

Code

DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF INT
...
FOR i = 1 TO 2
  FOR j = 1 TO 100
    LET a[i,j] = j
  END FOR
END FOR 
CALL a[2].clear()
-- I Want to release this memory!

But I guess there is no way, is it?

Regards,
-Snorri

Title: Re: Dynamic array memory management
Post by: Rene S. on January 19, 2021, 11:38:56 am

Code

IMPORT util
 
MAIN
    DEFINE i, j INT
    DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF INT
 
    FOR i = 1 TO 3
        FOR j = 1 TO 10
            LET a[i, j] = j
        END FOR
    END FOR
    DISPLAY util.JSON.stringify(a)
    LET a[2] = getEmptyArrayOfInt() -- replaces the 2nd sub-array by an empty array
    DISPLAY util.JSON.stringify(a)
END MAIN
 
FUNCTION getEmptyArrayOfInt() RETURNS DYNAMIC ARRAY OF INT
    RETURN NULL
END FUNCTION

Title: Re: Dynamic array memory management
Post by: Snorri B. on January 19, 2021, 11:56:11 am

Thanks again. It seems though that the memory the program uses is not released.

Code

 
DEFINE pid INT
 
MAIN
   DEFINE i, j INT
   DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF CHAR(1000)
 
   LET pid = fgl_getPid()
   FOR i = 1 TO 5
       FOR j = 1 TO 10000
           LET a[i, j] = sfmt("Element %1",j)
       END FOR
   END FOR
   CALL showmem("After loop")
   FOR i = 5 TO 1 STEP -1
     LET a[i] = getEmptyArrayOfChar()
     CALL showmem(sfmt("After clear %1", i))
   END FOR
END MAIN
 
FUNCTION getEmptyArrayOfChar() RETURNS DYNAMIC ARRAY OF CHAR(1000)
   RETURN NULL
END FUNCTION
 
FUNCTION showmem(s STRING)
DISPLAY s
RUN SFMT("pmap %1 | tail -n 1 ", pid)
DISPLAY ("----------------------------------\n")
END FUNCTION

Title: Re: Dynamic array memory management
Post by: Rene S. on January 19, 2021, 12:13:37 pm

Hello Snorri,

Quote

It seems though that the memory the program uses is not released.

The memory has been allocated with malloc(). Memory allocated by malloc is owned by the process forever.

Malloc and free cause a kind of "heap fragmentation". A call to free() could "theoretically" release the memory by calling sbrk() if and only if no memory has been allocated with an address greater then the address of the memory to be freed. This is not case.

Java can release memory (theoretically?). The Java garbage collector moves any reachable memory to a new position (kind of memory defragmentation). This implies, memory can be released.

Rene

Title: Re: Dynamic array memory management
Post by: Reuben B. on January 20, 2021, 12:54:37 am

I could probably do an Ask Reuben article on this topic. From the days of my first Genero migration many years ago now, something along the lines of what Rene said "The memory has been allocated with malloc(). Memory allocated by malloc is owned by the process forever." is what I recall.

Having done an aggressive transformation of static arrays to dynamic arrays, and chars to strings, we felt we were ahead. However it was still prudent to put a cap on QBEs for report/enquiry programs, and similarly a limit on records in a zoom window etc so that a user did not select all records (1000+) from a table and eat up some memory that would be retained for the life of a program. Also taking into account the expected lifetime of a program, most programs had lifetime of less than 5 minutes so not too big a problem if it has excessive memory use due to user input, but for the programs that ran all day e.g. menu, pos programs, a bit more important that an unchecked QBE does not end up holding memory unnecessarily all day.

One thing I am curious that I have never got round to looking at is whether in the case of a sparse array, are dictionaries better than dynamic array e.g.

Code

MAIN
TYPE y_type DICTIONARY OF INTEGER
TYPE xy_type DICTIONARY OF y_type
 
DEFINE xy_value xy_type
 
    LET xy_value[100000][50000] = 100
    DISPLAY xy_value[100000][50000]
END MAIN

... so if you were worried about memory consumption of a dynamic array with dimension 2, is something like that better? at the cost of losing some array functionality.

Title: Re: Dynamic array memory management
Post by: Snorri B. on January 20, 2021, 05:02:05 pm

Thank you Rene and Reuben.

I think the only option I have right now is try to limit the number of rows QBE returns.
Usually this is not a problem, but some users really take things to the limit although they are being warned many times.

Best regards,
-Snorri

Title: Re: Dynamic array memory management
Post by: Sebastien F. on January 20, 2021, 06:03:38 pm

Hi Snorri,

I did not realize that your fetch rows from a database from a QBE.
I thought you need to load a lot of data into an array for some computation.

For sure with DB queries / QBE, you better limit the total number of rows that can be fetched.

To me it does not make sense to load thousands of rows, and let the user scroll in the list of in a TABLE for example!

The users should add better filter the query.

You may want to propose some automatic filter (when the QBE is empty), for large tables in your application.

Seb

Four Js Development Tools Forum

Discussions by product => Genero BDL => Topic started by: Snorri B. on January 18, 2021, 05:37:44 pm