How to implement memory management for Java import

[Pierre F.]

Hi,

We are importing a java class into our 4gl programs and we need to manage memory with heap limit settings like this: "fglrun --java-option -Xms <size> -Xmx <size> myprog.42r. .. ".
We need this because we have encountered excessive use of virtual memory by our programs which import the java class.

Are there any of you who have some experience or advice on this, like how to assess the size requirements for the -Xms and -Xmx parameters?

Regards,
Pierre

[Reuben B.]

Something to consider is rather than having every program in your suite using the same imported Java class is to wrap the Java class up in a Web Service (either 100% Java or using Genero to create and manage the web service).  This is illustrated here http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_gws_java_APIs_001.html which uses Java to create the web service

This approach lessens the memory usage to the number of processes in the web service pool and is similar to the approach you can take with Genero Report Writer to use a distributed mode http://4js.com/online_documentation/fjs-gst-manual-html/#gst-topics/c_grw_distributed_mode.html .  With GRW distributed mode, this not only lessens memory but also improves processing time as initialisation only occurs once.  Using a web service you may also gain similar benefits and get to a position where the defaults are good and not need to worry and calculate Xms, Xmx

[Leo S.]

Hi Pierre , it would be interesting perhaps to figure out what causes the need for memory boundary adjustments.
Did you try to analyze heap dumps ?
Regards, Leo

[Pierre F.]

Hi Ruben, thank you very much for this advice. It is a good one. As we are using Java class to encrypt and decrypt data, we need high performance. In the documentation here: http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_gws_java_APIs_003.html   it says   "However, due to the XML serialization process and transport protocol HTTP in web services, there can be performance issues, so if your primary concern is performance, it is recommended to use the Genero Java Bridge." It's the reason why we want stay with the Genero Java Bridge.

[Pierre F.]

Hi Leo, thanks for the advise. What I see for the moment is that the "Committed virtual memory" climbs to 881M for one of our programs for exemple (Current heap size: 44M, Maximum heap size: 7G) if I don't limit the memory. If I adjust the memory limits with -Xmx4M for this program, I get a better result (Committed virtual memory: 86M, Current heap size: 3M, Maximum heap size: 3.5M). I tried to analyze the dump but I lack the knowledge to fully understand it at this time. I see well by experimentation that I gain there by limiting the memory. But I don't know how to determine the limits to assign to avoid a possible Out Of Memory in production except by experimentation.

Regards,
Pierre

[Reuben B.]

Hi Ruben, thank you very much for this advice. It is a good one. As we are using Java class to encrypt and decrypt data, we need high performance. In the documentation here: http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_gws_java_APIs_003.html   it says   "However, due to the XML serialization process and transport protocol HTTP in web services, there can be performance issues, so if your primary concern is performance, it is recommended to use the Genero Java Bridge." It's the reason why we want stay with the Genero Java Bridge.

That documentation was written a number of years ago when there was more SOAP than RESTful.  With RESTful web services, there is less overhead.  Encryption/Decryption are CPU intensive processes in any case, the additional cost of wrapping of a web service call may be minor in comparison.  Also note that you may get benefits as any initialisation is only performed once, this is why Genero Report Writer has such a performance increases when running in distributed mode.  It would be an interesting exercise using the high-level REST API to  crunch some numbers.

The other thing I'll point seen as you are encrypting and decrypting data, if there is an algorithm that you think should be available in our libraries then make sure you have passed that onto your support contact.  I know there is a proposed enhancement https://4js.com/support/issue/?id=GWS-1094 to make sure that we have what is available in every Java implementation https://docs.oracle.com/javase/7/docs/api/javax/crypto/Cipher.html

Reuben


 

[Leo S.]

Hi Leo, thanks for the advise. What I see for the moment is that the "Committed virtual memory" climbs to 881M for one of our programs for exemple (Current heap size: 44M, Maximum heap size: 7G) if I don't limit the memory. If I adjust the memory limits with -Xmx4M for this program, I get a better result (Committed virtual memory: 86M, Current heap size: 3M, Maximum heap size: 3.5M). I tried to analyze the dump but I lack the knowledge to fully understand it at this time. I see well by experimentation that I gain there by limiting the memory. But I don't know how to determine the limits to assign to avoid a possible Out Of Memory in production except by experimentation.

Regards,
Pierre

Hi Pierre, it would be perhaps good to invest a bit time for a heap dump analyzer such as Eclipse Mat https://www.eclipse.org/mat/ or https://www.ej-technologies.com/products/jprofiler/overview.html or https://heaphero.io ...
I did do for example a

jmap  -dump:live,format=b,file=xx.dmp <fgljp-pid>

for my fgljp tool (see https://github.com/FourjsGenero/tool_fgljp)
and uploaded xx.dmp to heaphero to get some more detailed insights why fglrun needs 40MB for this program...


HTH and Regards, Leo

[Pierre F.]

Hi Ruben, thank you very much for this advice. It is a good one. As we are using Java class to encrypt and decrypt data, we need high performance. In the documentation here: http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_gws_java_APIs_003.html   it says   "However, due to the XML serialization process and transport protocol HTTP in web services, there can be performance issues, so if your primary concern is performance, it is recommended to use the Genero Java Bridge." It's the reason why we want stay with the Genero Java Bridge.

That documentation was written a number of years ago when there was more SOAP than RESTful.  With RESTful web services, there is less overhead.  Encryption/Decryption are CPU intensive processes in any case, the additional cost of wrapping of a web service call may be minor in comparison.  Also note that you may get benefits as any initialisation is only performed once, this is why Genero Report Writer has such a performance increases when running in distributed mode.  It would be an interesting exercise using the high-level REST API to  crunch some numbers.

The other thing I'll point seen as you are encrypting and decrypting data, if there is an algorithm that you think should be available in our libraries then make sure you have passed that onto your support contact.  I know there is a proposed enhancement https://4js.com/support/issue/?id=GWS-1094 to make sure that we have what is available in every Java implementation https://docs.oracle.com/javase/7/docs/api/javax/crypto/Cipher.html

Reuben

We are encrypting and decrypting data with an AES 256/CBC/PKCS5Padding algorithm. I already had a discussion with a support contact about it. But i think i will specifically proposed an enhancement to them. We use Java also because we need to do it on our Java Web platform connected to the same database.

Other thing, i just find out an interresting alternative on FourjsGenero github here:   https://github.com/FourjsGenero/ex_cryptography_AES256. It use the 4gl XML Encryption in the algorithm we need and it offer a Java conterpart. I whish that i had saw it before. Can i have your insight about this alternative?

Pierre

[Reuben B.]

Other thing, i just find out an interresting alternative on FourjsGenero github here:   https://github.com/FourjsGenero/ex_cryptography_AES256. It use the 4gl XML Encryption in the algorithm we need and it offer a Java conterpart. I whish that i had saw it before. Can i have your insight about this alternative?

If you can do what you want using ...

http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_gws_XmlEncryption_EncryptString.html
http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_gws_XmlEncryption_DecryptString.html

... and with an algorithm listed ...

http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/r_gws_XmlCryptoKey_supported_keys.html

... then do that, i.e lines 11-15 of  https://github.com/FourjsGenero/ex_cryptography_AES256/blob/master/src/crypto.4gl

Code (genero) Select Expand

      # Creation and assign of the cryptokey
        let cryptoKey = xml.CryptoKey.Create("http://www.w3.org/2001/04/xmlenc#aes256-cbc")
        call cryptokey.setKey(symkey)

        # Encryption
        let encrypted_str = xml.Encryption.EncryptString(cryptoKey, plaintext)

...  (maybe doing the padding manually),  and then there is no need for Java.  If what you want isn't in the list of  supported keys that is when you need to look at a solution potentially using IMPORT JAVA.

It is a pity that these methods are in packages named xml.  I think it came first as part of the XML and then someone said as well as encrypting/decrypting XML documents why can't it also encrypt/decrypt a string.

Reuben

[Pierre F.]

The other thing I'll point seen as you are encrypting and decrypting data, if there is an algorithm that you think should be available in our libraries then make sure you have passed that onto your support contact.  I know there is a proposed enhancement https://4js.com/support/issue/?id=GWS-1094 to make sure that we have what is available in every Java implementation https://docs.oracle.com/javase/7/docs/api/javax/crypto/Cipher.html

Reuben

Just to let know that the AES/CBC/PKCS5Padding (256) algorithm have been added by the support team to the GWS-1094 issue at our request. Thank you!

Pierre