next »

[Jos? V.]

Hi guys,
we are having an issue with httpdispatch crashing everyday at the same time with a "out of memory error".
The process isn't responding to any request at the time it crashes.
This does seem to us as a some external process forcing httpdispatch to crash somehow but we are having some dificulties figuring out what it is.

Did anyone had this issue elsewere? Any clue on what to might be happening?
httpdispatch is version 3.20

16:18:49.319805 "Incoming request" /ws/r/wsServer/LeaseContractInfo
16:18:49.335360 "Response sent" 10.101.128.5 - [Fri, 27 Jan 2023 16:18:49 GMT] "POST /ws/r/wsServer/LeaseContractInfo HTTP/1.1" 200 874 "-" "Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.42000)" query string
:
16:23:04.222685 "SYSTEM ERROR" At (httpd.c:1988) : Error 32 = 'Broken pipe'
16:23:04.222733 "httpd_write" Failed to write buffer
16:23:05.249770 "SYSTEM ERROR" At (httpd.c:1988) : Error 32 = 'Broken pipe'
16:23:05.249837 "httpd_write" Failed to write buffer
16:23:06.253697 "SYSTEM ERROR" At (httpd.c:1988) : Error 32 = 'Broken pipe'
16:23:06.253763 "httpd_write" Failed to write buffer
16:23:07.255852 "SYSTEM ERROR" At (httpd.c:1988) : Error 32 = 'Broken pipe'
16:23:07.255886 "httpd_write" Failed to write buffer
16:23:08.262486 "SYSTEM ERROR" At (httpd.c:1988) : Error 32 = 'Broken pipe'
16:23:08.262529 "httpd_write" Failed to write buffer
MEMORY Out of memory error. Ending program

[Reuben B.]

If and when you also raise a support call, please include the full set of log files and set CATEGORIES_FILTER to ALL to capture as much information as possible.

In this instance, I'd be curious to see the ulimits values that are written at the beginning of the log file.  It maybe that what you perceive as the "some external process forcing httpdispatch to crash somehow ", could be the O/S ulimits protections kicking in.   

The fact you say that it crashes everyday at the same time makes me suspicious that httpdispatch is being used in production.  (See note here http://4js.com/online_documentation/fjs-gas-manual-html/#gas-topics/c_gas_tools_httpdispatch.html)

[Jos? V.]

Thanks for the quick reply Reuben.
We are going to monitor again today, install the latest GAS and set CATEGORIES_FILTER to all.
hopefully it will lead us in the right direction.

This is not a production environment. We are only testing the webservice while waiting for IBM to install apache on the client's machine(takes a lot of time........).

These are the ulimits on the begining of the httpdispatch.log:

#Version: 1.0
#Date: 2023/01/30 08:37:16.511971
#Fields: time event-type event-params
#Ulimits:
#   core file size     : 1073741312 (unlimited)
#   data seg size      : 268435456 (unlimited)
#   file size          : unlimited (unlimited)
#   max memory size    : 33554432 (unlimited)
#   open files         : 2000 (unlimited)
#   stack size         : 33554432 (4294967296)
#   cpu time           : unlimited (unlimited)
#   virtual memory     : unlimited (unlimited)

[Reuben B.]

Hi Jos,

I don't recall seeing a support case come through.  Did the logs lead you to a solution?  We do like to keep a knowledge base of errors that popup and the solution

Reuben

[Jos? V.]

Hi Reuben,
We did follow up on a support case -> SUPIBERICA-189.

We are still trying to find the culprit, we have a process that runs about the same time that seems to be a port scanner and we are trying to find with the client what it is and if it can be the responsible for this crash

[Reuben B.]

Hi Reuben,
We did follow up on a support case -> SUPIBERICA-189.

We are still trying to find the culprit, we have a process that runs about the same time that seems to be a port scanner and we are trying to find with the client what it is and if it can be the responsible for this crash

thanks, found it,   (I was searching on the error message you reported in the original forum post but that had not been explicitly listed in the case!).    The port scanner looks an interesting angle, note that httpdispatch listens on two ports (search on port in the .xcf)

Reuben

PS Just for anyone reading this thread and wondering why we ask for ulimits I wrote on Ask-Reuben article on this https://4js.com/ask-reuben/ig-58/

[Sisavanh S.]

Hi,

Just to keep the forum updated.
We have filed bug:
GAS-3823: Enhance TLV format with hash and prefix to avoid allocating too big buffers and crash dispatcher.
The issue was a big request has been sent to the GAS admin port which was not expected.
The fix is planned for next 3.20 and 4.01 GAS MR.
If possible in the meantime, avoid sending any big or non Genero requests to the GAS admin port.

Regards,
Sisa.