Subscribe for automatic updates: RSS icon RSS

Login icon Sign in for full access | Help icon Help
Advanced search

Pages: [1]
  Reply  |  Print  
Author Topic: Dynamic array memory management  (Read 3526 times)
Snorri B.
Posts: 103


« on: January 18, 2021, 05:37:44 pm »

Hi y'all and a happy new year!

I've been investigating memory usage in some of our modules.

Apparently, if you allocate memory with a dynamic array, there is no way to reclaim it. I have tried
let a = null
call a.clear()
loop and call a.deleteElement

But it seems that once memory is allocated, it never gets released, only reused if possible. Even if the array is a part of an object type, nulling the object will not release the memory. I've tried this with modal and local arrays, same result.

I find it particularly strange that destroying an object does not release all of it's resources.

What do fellow BDL-ers think?

Best regards,
-Snorri
Rene S.
Four Js
Posts: 95


« Reply #1 on: January 19, 2021, 09:15:49 am »

Hello Snorri,
your observation is correct: "once memory is allocated, it never gets released, only reused if possible"

Array elements can not be released while the array is alive. This is an (incomplete) workaround to avoid crashes when dereferencing array elements used in INPUTs or used as host-variables in CURSORs.

Informix-4GL can create at runtime references to array-elements. Examples: INPUT array[index].*, DECLARE cursor-name CURSOR FOR SELECT ... INTO array[index].*. The runtime assumes: the referenced array element is alive. A better saying for "assumes" is "blindly trusts". For exactly this reason each array element once created can not be released.

Can not be released? There is an (unsafe) trick:
Code
  1. DEFINE a DYNAMIC ARRAY OF INT
  2. ...
  3. CALL a.clear() -- does not release unused array elements
  4. LET a = getEmptyArray() -- releases the old array (and all of its elements)
  5. ...
  6. FUNCTION getEmptyArray()
  7.    DEFINE r dynamic array of int
  8.    RETURN r
  9. END FUNCTION
  10.  

DYNAMIC ARRAY is a reference type, for that reason the "old" array will be released when assigning a "new" one. Be careful. If elements of the "old" array are still referenced, then the program crashes when dereferencing the elements.

BTW: the semantics of DICTIONARY elements when removing them from a DICTIONARY is much better. DICTIONARY elements will be released when removing *and* not being referenced.

Snorri B.
Posts: 103


« Reply #2 on: January 19, 2021, 11:23:47 am »

Thanks Rene.

Our situation is a little bit more complex. We have an array with dimension 2 and we need to release memory for the last element.
Code
  1. DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF INT
  2. ...
  3. FOR i = 1 TO 2
  4.  FOR j = 1 TO 100
  5.    LET a[i,j] = j
  6.  END FOR
  7. END FOR
  8. CALL a[2].clear()
  9. -- I Want to release this memory!
  10.  

But I guess there is no way, is it?

Regards,
-Snorri
Rene S.
Four Js
Posts: 95


« Reply #3 on: January 19, 2021, 11:38:56 am »

Code
  1. IMPORT util
  2.  
  3. MAIN
  4.    DEFINE i, j INT
  5.    DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF INT
  6.  
  7.    FOR i = 1 TO 3
  8.        FOR j = 1 TO 10
  9.            LET a[i, j] = j
  10.        END FOR
  11.    END FOR
  12.    DISPLAY util.JSON.stringify(a)
  13.    LET a[2] = getEmptyArrayOfInt() -- replaces the 2nd sub-array by an empty array
  14.    DISPLAY util.JSON.stringify(a)
  15. END MAIN
  16.  
  17. FUNCTION getEmptyArrayOfInt() RETURNS DYNAMIC ARRAY OF INT
  18.    RETURN NULL
  19. END FUNCTION
  20.  
  21.  
Snorri B.
Posts: 103


« Reply #4 on: January 19, 2021, 11:56:11 am »

Thanks again. It seems though that the memory the program uses is not released.
Code
  1.  
  2. DEFINE pid INT
  3.  
  4. MAIN
  5.   DEFINE i, j INT
  6.   DEFINE a DYNAMIC ARRAY WITH DIMENSION 2 OF CHAR(1000)
  7.  
  8.   LET pid = fgl_getPid()
  9.   FOR i = 1 TO 5
  10.       FOR j = 1 TO 10000
  11.           LET a[i, j] = sfmt("Element %1",j)
  12.       END FOR
  13.   END FOR
  14.   CALL showmem("After loop")
  15.   FOR i = 5 TO 1 STEP -1
  16.     LET a[i] = getEmptyArrayOfChar()
  17.     CALL showmem(sfmt("After clear %1", i))
  18.   END FOR
  19. END MAIN
  20.  
  21. FUNCTION getEmptyArrayOfChar() RETURNS DYNAMIC ARRAY OF CHAR(1000)
  22.   RETURN NULL
  23. END FUNCTION
  24.  
  25. FUNCTION showmem(s STRING)
  26. DISPLAY s
  27. RUN SFMT("pmap %1 | tail -n 1 ", pid)
  28. DISPLAY ("----------------------------------\n")
  29. END FUNCTION
  30.  
  31.  
Rene S.
Four Js
Posts: 95


« Reply #5 on: January 19, 2021, 12:13:37 pm »

Hello Snorri,
Quote
It seems though that the memory the program uses is not released.

The memory has been allocated with malloc(). Memory allocated by malloc is owned by the process forever.

Malloc and free cause a kind of "heap fragmentation". A call to free() could "theoretically" release the memory by calling sbrk() if and only if no memory has been allocated with an address greater then the address of the memory to be freed. This is not case.

Java can release memory (theoretically?). The Java garbage collector moves any reachable memory to a new position (kind of memory defragmentation). This implies, memory can be released.

Rene
Reuben B.
Four Js
Posts: 834


« Reply #6 on: January 20, 2021, 12:54:37 am »

I could probably do an Ask Reuben article on this topic.  From the days of my first Genero migration many years ago now, something along the lines of what Rene said  "The memory has been allocated with malloc(). Memory allocated by malloc is owned by the process forever." is what I recall.

Having done an aggressive transformation of static arrays to dynamic arrays, and chars to strings, we felt we were ahead.  However it was still prudent to put a cap on QBEs for report/enquiry programs, and similarly a limit on records in a zoom window etc so that a user did not select all records (1000+) from a table and eat up some memory that would be retained for the life of a program.  Also taking into account the expected lifetime of a program, most programs had lifetime of less than 5 minutes so not too big a problem if it has excessive memory use due to user input, but for the programs that ran all day e.g. menu, pos programs, a bit more important that an unchecked QBE does not end up holding memory unnecessarily all day.

One thing I am curious that I have never got round to looking at is whether in the case of a sparse array, are dictionaries better than dynamic array e.g.

Code
  1. MAIN
  2. TYPE y_type DICTIONARY OF INTEGER
  3. TYPE xy_type DICTIONARY OF y_type
  4.  
  5. DEFINE xy_value xy_type
  6.  
  7.    LET xy_value[100000][50000] = 100
  8.    DISPLAY xy_value[100000][50000]
  9. END MAIN

... so if you were worried about memory consumption of a dynamic array with dimension 2, is something like that better? at the cost of losing some array functionality.






Product Consultant (Asia Pacific)
Developer Relations Manager (Worldwide)
Author of https://4js.com/ask-reuben
Contributor to https://github.com/FourjsGenero
Snorri B.
Posts: 103


« Reply #7 on: January 20, 2021, 05:02:05 pm »

Thank you Rene and Reuben.

I think the only option I have right now is try to limit the number of rows QBE returns.
Usually this is not a problem, but some users really take things to the limit although they are being warned many times.

Best regards,
-Snorri
Sebastien F.
Four Js
Posts: 463


« Reply #8 on: January 20, 2021, 06:03:38 pm »

Hi Snorri,

I did not realize that your fetch rows from a database from a QBE.
I thought you need to load a lot of data into an array for some computation.

For sure with DB queries / QBE, you better limit the total number of rows that can be fetched.

To me it does not make sense to load thousands of rows, and let the user scroll in the list of in a TABLE for example!

The users should add better filter the query.

You may want to propose some automatic filter (when the QBE is empty), for large tables in your application.

Seb
Pages: [1]
  Reply  |  Print  
 
Jump to:  

Powered by SMF 1.1.21 | SMF © 2015, Simple Machines