Four Js Development Tools Forum

Not sure if this is the best place to post this but here goes anyway. One common requirement of most applications I have deployed over many years is to handle files, both into and out of our applications. We normally achieve this using a concept of inboxes and outboxes. Files can a variety of content (ASCII, XML, Edifact) etc and can be delivered by various means, i.e.

• Dropped into directories on the filesystem.
• Email
• FTP.
• AS2.
• Message queues
• etc

Currently various 3rd products have to be used to develop Genero solutions and we normally end up configuring these to poll every few minutes for activity via a task scheduler. What I'd like to be able to do is to develop server based Genero applications hosted in GAS which could be configured to wake up and process files based upon events, i.e:-

For Email, say when a message say matching a certain criteria appeared in a monitored mailbox.
For a drop directory when a file was created, modified or deleted in a monitored directory.

Gradually over time additional interfaces and event monitors could be added in to support a wide range of delivery types.

Hopefully you get the idea!

David

Hello David
This is a good place indeed to discuss that.
I understand very well what you mean. The idea is very interesting and we will investigate on that. Did you already evaluate other tools to do the job ?
Maybe somebody else there use such tools and can let us know their experience.
Olivier.
--
Olivier IMBERT
Projects manager
Genero Clients and web Services
Four J's Development Tools

I have looked in the past and the tools I found were very expensive to deploy. 3rd party costs (application and database licenses etc) are already an issue for us so the idea never got off the ground. An implementation directly in GAS without AN other product or products would also be a much neater solution I think...

I'm not 100% sure on the purpose of 'dropping files' with todays systems.

Email Out:we can send using a variety of methods either from server or via the client using various APIs in Genero.

Email In: is tricky but is possible using the Java Mail API directly in a 4gl program, I'm using this for our call logging system, I have a Genero program running on the server that checks our Google Mail box every 5 minutes and brings back new mails and moves them to a processed folder on the gmail server, it creates the new calls/events in our support system and the main support program we run is sitting there using ON IDLE to check the tables, or using the 2.30 MessageServer feature.
https://4js.com/techdocs/genero/fgl/devel/DocRoot/User/ClassMessageServer.html

Files to client PC's: done using frontCalls to open a standard windows dialog box for saveFile or open directory, these return a file name or directory name on the server then you use fgl_putFile(source,target) to send the file to client ( no extra network ports required ).

Files from client PC's: same as above but using the openFile dialog and fgl_getFile to retrieve the file to server.

Files to another server: I'd just use the standard 'scp' command on Linux or on Windows os.path.copy(see below to link to manual for os.path)

I guess the problem is legacy 3rd party applications that don't have Web Services yet to transfer data, so your only choice is to dump a file in a folder and then run their program to upload that file.
I tackled this recently at a site that's Windows based, we used the os.path.copy the file produced to the server with the 3rd party application, then used  psexec \\MACHINE -u USER -p PASS <cmd> [args] to run the 3rd party 'loader' application to load the file.
http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx
At the same site they use the Genero Report Writer for all their reports( legacy 4gl ascii reports ) and have them all produced as PDF and sent to the client that requested the report. Background reports are sent directly to a named printer I think or dumped as PDF's in a folder on a server somewhere to be forgotten about. I wonder what percentage of reports ( especially big ones ) are actually ever read by a human being?

The other way around is the legacy 3rd party App just dumps a file somewhere and you have to then do something with that file ?
It's simple enough to write a Genero program to read a directory and do something what it finds then move that file somewhere else, we have all the methods to do that in a platform independent way using:
https://4js.com/techdocs/genero/fgl/devel/DocRoot/User/Ext_os_Path.html
https://4js.com/techdocs/genero/fgl/devel/DocRoot/User/ClassChannel.html

Files are 'dropped' usually to interface with other computer systems. It has been my experience that these interfaces are normally quite simplistic as each environment has restrictions on what it can handle and how it can deliver it.  I guess there where two main parts to my request.

1. To provide a standard interface for files/messages in and out of the application server without requiring additional 3rd party solutions. Different transfer methods could be plugged into that interface where the transfer method was effectively invisible to the Genero developer, they just pick up or generate their files going in and out. Functionality exists currently in Genero to do some of this today, like SMTP and working with files/directories etc, but each option needs a different coding solution and others like FTP, Message Queuing or AS2 etc don't exist yet and therefore require 3rd party software and further custom coding. Also where functionality does exist in Genero today it has limitations, take email for instance, what if my mailserver uses secure authentication or a proxy? How do you receive email using POP3 or IMAP4 etc,etc...

2. To provide a standard method to react to files coming in. Currently I see it as there are two main alternatives:-

       1. Run continuously, monitoring locations and reacting accordingly, idling when nothing to do. . You also need a mechanism to stop/start the monitoring program.

       2. Poll every so often for any new files at the monitored locations using a scheduler or 3rd party solution to run the program at the poll time.

I'd prefer it if I could just write my program to process a file, without having to worry about the delivery mechanism and then have it called automatically whenever there is something for it to do. Currently we use the polling method but find it unreliable in practice. We have experimented with various 3rd party products which can monitor directories and run external programs but these also go wrong from time to time, so not ideal and they typically only handle files dropped into a directory, they can't for instance monitor a mailbox or a directory on an FTP server etc...

On example of a product capable of doing what I've attempted to describe (and a lot more besides!) would be BizTalk server. Have a look here (http://www.microsoft.com/biztalk/en/us/messaging.aspx (http://www.microsoft.com/biztalk/en/us/messaging.aspx)) for an overview on how it handles messaging and here (http://www.microsoft.com/biztalk/en/us/adapters-included.aspx (http://www.microsoft.com/biztalk/en/us/adapters-included.aspx)) for the current list of adapters. Its totally overkill feature wise for our requirements and ruled out price wise too, but it hopefully gives you a better idea of what I was hoping to see one day...

Hi,

Just to let you know that we filed enhancements:
- 18683 -  Provide a dedicated file watcher proxy
- 18684 -  Provide a dedicated email POP3 watcher proxy

Many regards,

Sisa.

Request 1 sounds like mission impossible to me and the reason Web Services were invented!
As someone that has written code to talk to mail servers and such and even using java mail api in 4gl ( which is
how our support system works with Google mail using Imapi api ) I can say to write an API that can handle email
with all it's complexities(SSL & Proxy etc) and FTP and SCP(a far easier way to send and receive files in a secure way)
to just give a simple way that can be configured for X number of legacy APIs sounds like a very serious amount of work.
I'd rather partition the 3rd parties to join the modern age and use Web Services instead or at least wrap their old API's
in a Web Service so that modern languages can use a simple modern way to send / receive their data.

Well obviously not too impossible as several products exist which do this already (and a variety of other communication mechanisms) using plugins/adapters! I've mentioned Biztalk but another example would be Jitterbug and another called Apatar.  If I can free up some time I'm going to have a closer look at Jitterbug but I suspect it'll be ruled out on cost too.

RE: Webservices. I do agree but persuading customers to use them is another matter altogether! I've yet to come across a single customer of ours who we could interface with this way. I am actually in the process of using Webservices to communicate with a 3rd party service (UK Customs), but even this is proving challenging as they require Soap 1.2 and WSDL 2, not currently supported by GWS... :-(

Thanks. Files and Email were just examples, if ever implemented it would be great if the proxy architecture could be extended to other communication mechanisms via plugins.

Oops, re: Jitterbug, I meant to say Jitterbit!