implementing event-based communication between microservices

[Francois G.]

Hi,

Is there a way for a Genero BDL application to register a callback function to a remote (across TCP/IP network) microservice, so that the remote microservice can call this BDL function when something notable happens?

https://learn.microsoft.com/en-us/dotnet/architecture/microservices/multi-container-microservice-net-applications/integration-event-based-microservice-communications

At the moment I have an infinite loop that polls the remote microservice very often (every 2 seconds), which is OK for a one off demo.
I know that I could write a non-BDL daemon (in Google Go, in Java...) that would then use an event bus (such as RabbitMQ), and then insert / update my Informix database.
Could this be done natively in any version of the BDL, present or future?

Regards,

[Sebastien F.]

Hello,

In such case, I usually recommend to define an SQL database table to store "events" with timestamps, that can be filled by the source.

Genero applications can the poll for new/recent events

The advantage of an SQL events table is that you have full control on the data your want to pass and you can filter with SQL queries.
And events data cannot be lost since it's written in the database (you can have a history of the events)

In the Genero code, either the user is actively using the app and most not be disturbed, or the app is idle and waits for user interaction.
When idle, the app code can execute the SQL query to poll for recent events.

We have ON IDLE trigger in dialogs, to execute (little) code when the user does not interact with the application forms.

We also have ON TIMER, but I would not use this one in such case.

Seb

[Francois G.]

Thanks Seb,

This is what I have already done :-)
I have written a Linux daemon that talks to the remote microservice on the Internet, which populates an Informix event table, which is then polled (ACTION ON IDLE) by the Genero BDL GUI.
In the 4GL, I have used an ON IDLE action with a timeout of 3 seconds, but the Genero BDL manual says that I should not do this (the manual says that ON IDLE is intended for larger values such as 10 seconds).
After reading this, I do not know if my demo code can be made production quality.
I will have to benchmark this solution with and without this ON IDLE action to see what the impact is on the Informix server and on the GUI app.

Are there any plans for future Genero BDL (or Genero web services) to allow network callbacks, where a Genero BDL function is called, similar to the existing WSThrows for REST Web Services in 4GL, or the older WHENEVER ERROR CALL handler?

Regards,

[Rene S.]

Are there any plans for future Genero BDL

yes.
Distinguish interactive and none-interactive applications.
For none-interactive applications: you can already use (in version 4) util.Channels.select(channels DYNAMIC_ARRAY_OF_CHANNEL) RETURNS INT
For interactive programs: Since a very long time the idea of a dialog trigger like ON DATA AVAILABLE(channels DYNAMIC_ARRAY_OF_CHANNEL) exists. The interaction statement (INPUT etc.) receives this trigger if on of the registered channels is ready for reading. It's not in progress yet but frequently discussed.

[Francois G.]

Thanks Rene,

I will look out for this in the future

[Leo S.]

Francoise,
Rene mentions util.Channels.select()...it's not in the docs yet and brand new (version 4.01.04).
If you want to see how it is used in a real world program, check the code in https://github.com/FourjsGenero/tool_fgljp in the wip/no_java branch (fgljp.4gl) which also includes binary operations for channels.
You could however check also the code in the main branch how one can use the java bridge to write a Genero program doing socket select() by using java.nio classes with older Genero versions.

Regards, Leo

[Francois G.]

Thanks Leo,

I did see the previous base.Channel docs, and I assumed that the util.Channels would be still work in progress.

I am glad to see util.Channels.select(_channels), which looks like the traditional blocking C select API call, and this would indeed replace polling with OS level sleep awaken when data is seen on the sockets.

Regards,

[Sebastien F.]

The util.Channels class will appear in the next Genero BDL 4.01 documentation publication.
This doc update should be available very soon.
Seb

[Reuben B.]

I'll add some comments that skirt around the issue but might trigger something ...

1.

In the 4GL, I have used an ON IDLE action with a timeout of 3 seconds, but the Genero BDL manual says that I should not do this (the manual says that ON IDLE is intended for larger values such as 10 seconds).

Don't get too spooked about the warning, understand why the warning is there.  ON IDLE 3 can be thought of as the equivalent of a user pressing TAB or ENTER every 3 seconds.  That is every 3 seconds sending an action from the front-end to back-end, some processing (if any) occurring on the back end, and the update to the AUi Tree (if any) being returned.  If one process has an ON IDLE 3, chances are the impact on the system will barely be noticed, if all processes have an ON IDLE 3, these extra messages and processing will add up.  So that is what you have to watch out for. 

2. I did an Ask-Reuben on Genero being single threaded https://4js.com/ask-reuben/ig-127/ which may have some points relevant to this discussion.  If your program has some user input, how do you expect it to respond when the user is busy doing something.  So if you are looking for a callback function to be triggered, if your user is busy typing something, would you want that callback function triggered then?, hence the path of polling after an ON IDLE.  If your program is non-interactive then you are probably looking for a solution like on the server side with Web Services where your register the function to occur and then a method like com.WebServiceEngine.processServices blocks and then calls the appropriate function when something is received.

3. As I speculate in the article, for mobile we had a predefined action, ON ACTION notificationpushed that the Mobile host triggers when a notification is received for that which you have registered for Push Notifications.   http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_fgl_mobile_push_notifications.html  Conceivably gdc.exe or gbc.js or a web component could take on the role of the mobile O/S and register for and receive notifications and trigger the predefined action.

4. One final point, RabbitMQ (https://www.rabbitmq.com/devtools.html) , the AMPQ protocol (https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol) , etc these terms don't appear a lot in our support portal.  If there are developers out there that want to interact this way, are looking for a client library to use let your support contact know so that we can gauge the correct amount of interest.

Reuben

 

[Francois G.]

Thanks Reuben,

1.
On each (large) server we have about 1k (at least) such processes with a short ON IDLE, and the stats on the server are OK.
Is there anything on GPAAS that I need to monitor to see if anything is of concern?

2.
The use case (I hope) is that a user, for example, uses the GDC to operate our 4GL application (enter initial data), then switches over to a native Windows application (further data processing), not using GDC for a few minutes (maybe they have 2 monitors, and they move their mouse away from the GDC app to the other Windows app).
So, having an ON IDLE in the GDC (usually with no GUI changes) allows the backend to decide if work is required (which it is not, most of the time), and in the rare cases where it is (every few minutes, not every 3 seconds), then frontend calls (such as file transfers) take place between the GDC backend and the user's frontend PC.
If the ON IDLE handler decided that some front-end call had to be made, then some UI changes will be made (update the status bar for example).
In case the ON IDLE decides that a bigger UI change is needed, we have global variables that indicate if the user in the middle of an operation (in the middle of an INPUT, some data not saved...), in which case the planned "big" GUI operation is not performed.

3.
A GDC / GBC notificationpushed event handler for GUI applications sounds interesting: I will read up the GMC docs on this.

4.
I was looking at event busses and their components (at various levels, Azure Service Bus, Google Pub / Sub, Kafka, RabbitMQ, ...), but, for my current project, they are all too heavy for an initial proof of concept.  I might come back to these for a later production quality solution, more investigation will be required.

I am now looking at web sockets, that would maintain a bidirectional link between a remote web API / microservice.
Does Genero have support for the ws:// protocol (web sockets)?

I am investigating if it would be possible for Genero 4.01 headless daemon (not a GUI application) to make a web API call (REST), such as subscribing to a topic of interest, then have the daemon sleep using a blocking select call on the web socket, waiting for the remote microservice to have (very small) messages ready to be read with another REST call.

Is there any well established Genero "event bus" programming pattern or strategy for such a daemon?  Are people usually polling the remote microservice?

As my time with this initial investigation is almost up, I have taken some notes related to any Genero solution, and then moved on to code my ideas in Google Go, but this is intended to be throwaway code.

Thanks again for your detailed and prompt responses.

Regards,

[potyn s.]

I'll add some comments that skirt around the issue but might trigger something ...

1.

In the 4GL, I have used an ON IDLE action with a timeout of 3 seconds, but the Genero BDL manual says that I should not do this (the manual says that ON IDLE is intended for larger values such as 10 seconds).

Don't get too spooked about the warning, understand why the warning is there.  ON IDLE 3 can be thought of as the equivalent of a user pressing TAB or ENTER every 3 seconds.  That is every 3 seconds sending an action from the front-end to back-end, some processing (if any) occurring on the back end, and the update to the AUi Tree (if any) being returned.  If one process has an ON IDLE 3, chances are the impact on the system will barely be noticed, if all processes have an ON IDLE 3, these extra messages and processing will add up.  So that is what you have to watch out for. 

2. I did an Ask-Reuben on Genero being single threaded https://4js.com/ask-reuben/ig-127/ which may have some points relevant to this discussion.  If your program has some user input, how do you expect it to respond when the user is busy doing something.  So if you are looking for a callback function to be triggered, if your user is busy typing something, would you want that callback function triggered then?, hence the path of polling after an ON IDLE.  If your program is non-interactive then you are probably looking for a solution like on the server side with Web Services where your register the function to occur and then a method like com.WebServiceEngine.processServices blocks and then calls the appropriate function when something is received.

3. As I speculate in the article, for mobile we had a predefined action, ON ACTION notificationpushed that the Mobile host triggers when a notification is received for that which you have registered for Push Notifications.   http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_fgl_mobile_push_notifications.html  Conceivably gdc.exe or gbc.js or a web component could take on the role of the mobile O/S and register for and receive notifications and trigger the predefined action. drift boss

4. One final point, RabbitMQ (https://www.rabbitmq.com/devtools.html) , the AMPQ protocol (https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol) , etc these terms don't appear a lot in our support portal.  If there are developers out there that want to interact this way, are looking for a client library to use let your support contact know so that we can gauge the correct amount of interest.

Reuben

 

I got it. Thx

[Barbara H.]

I'll add some comments that skirt around the issue but might trigger something ...

1.

In the 4GL, I have used an ON IDLE action with a timeout of 3 seconds, but the Genero BDL manual says that I should not do this (the manual says that ON IDLE is intended for larger values such as 10 seconds).

Don't get too spooked about the warning, understand why the warning is there.  ON IDLE 3 can be thought of as the equivalent of a user pressing TAB or ENTER every 3 seconds.  That is every 3 seconds sending an action from the front-end to back-end, some processing (if any) occurring on the back end, and the update to the AUi Tree (if any) being returned.  If one process has an ON IDLE 3, chances are the impact on the system will barely be noticed, if all processes have an ON IDLE 3, these extra messages and processing will add up.  So that is what you have to watch out for. 

2. I did an Ask-Reuben on Genero being single threaded https://4js.com/ask-reuben/ig-127/ which may have some points relevant to this discussion.  If your program has some user input, how do you expect it to respond when the user is busy doing something.  So if you are looking for a callback function to be triggered, if your user is busy typing something, would you want that callback function triggered then?, hence the path of polling after an ON IDLE.  If your program is non-interactive then you are probably looking for a solution like on the server side with Web Services where your register the function to occur and then a method like com.WebServiceEngine.processServices blocks and then calls the appropriate function when something is received.

3. As I speculate in the article, for mobile we had a predefined action, ON ACTION notificationpushed that the Mobile host triggers when a notification is received for that which you have registered for Push Notifications.   http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_fgl_mobile_push_notifications.html  Conceivably gdc.exe or gbc.js or a web component could take on the role of the mobile O/S and register for and receive notifications and trigger the predefined action.

4. One final point, RabbitMQ (https://www.rabbitmq.com/devtools.html) , the AMPQ protocol (https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol geometry dash world) , etc these terms don't appear a lot in our support portal.  If there are developers out there that want to interact this way, are looking for a client library to use let your support contact know so that we can gauge the correct amount of interest.

Reuben

Useful!

[Victor G.]

I'll add some comments that skirt around the issue but might trigger something ...

1.

In the 4GL, I have used an ON IDLE action with a timeout of 3 seconds, but the Genero BDL manual says that I should not do this (the manual says that ON IDLE is intended for larger values such as 10 seconds).

Don't get too spooked about the warning, understand why the warning is there.  ON IDLE 3 can be thought of as the equivalent of a user pressing TAB or ENTER every 3 seconds.  That is every 3 seconds sending an action from the front-end to back-end, some processing (if any) occurring on the back end, and the update to the AUi Tree (if any) being returned.  If one process has an ON IDLE 3, chances are the impact on the system will barely be noticed, if all processes have an ON IDLE 3, these extra messages and processing will add up.  So that is what you have to watch out for. 
basketball stars
2. I did an Ask-Reuben on Genero being single threaded https://4js.com/ask-reuben/ig-127/ which may have some points relevant to this discussion.  If your program has some user input, how do you expect it to respond when the user is busy doing something.  So if you are looking for a callback function to be triggered, if your user is busy typing something, would you want that callback function triggered then?, hence the path of polling after an ON IDLE.  If your program is non-interactive then you are probably looking for a solution like on the server side with Web Services where your register the function to occur and then a method like com.WebServiceEngine.processServices blocks and then calls the appropriate function when something is received.

3. As I speculate in the article, for mobile we had a predefined action, ON ACTION notificationpushed that the Mobile host triggers when a notification is received for that which you have registered for Push Notifications.   http://4js.com/online_documentation/fjs-fgl-manual-html/#fgl-topics/c_fgl_mobile_push_notifications.html  Conceivably gdc.exe or gbc.js or a web component could take on the role of the mobile O/S and register for and receive notifications and trigger the predefined action.

4. One final point, RabbitMQ (https://www.rabbitmq.com/devtools.html) , the AMPQ protocol (https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol) , etc these terms don't appear a lot in our support portal.  If there are developers out there that want to interact this way, are looking for a client library to use let your support contact know so that we can gauge the correct amount of interest.

Reuben

 

Thank for your sharing.