[Home]

Summary:ASTERISK-27258: PJSIP issues and possible memory leaks????
Reporter:Leon Zetekoff (wa4zlw)Labels:
Date Opened:2017-09-06 12:47:07Date Closed:2017-09-19 13:37:29
Priority:MajorRegression?
Status:Closed/CompleteComponents:pjproject/pjsip
Versions:14.6.0 14.6.1 Frequency of
Occurrence
Related
Issues:
Environment:Plus Freepbx 14.0.1.4 running on vultr.com with 2048 ram (had 1024 ram and upped it)Attachments:( 0) debugoutput10.txt
( 1) debugoutput3.txt
( 2) debugoutput4.txt
( 3) debugoutput5.txt
( 4) debugoutput5.txt
( 5) debugoutput6.txt
( 6) debugoutput7.txt
( 7) debugoutput9.txt
( 8) FreePBX.7z
Description:Potentiallly related to closed ASTERISK-25653

While I do have sip clients on cellular right now this is all occuring in-house over wifi or wired.
My Asterisk 14/Freepbx14 is located on vultr and now is 2048GB ram up from 1024 ram last night

Is anyone having weird issues with PJSIP? Since I migrated from ASterisk 13/Freepbx 12 I’ve had nothing but extensions dropping off. I was running on a 1024 RAM on Vultr and upped it to 2048 ram last night and TOP is showing me free memory keeps dropping:
{noformat}
top - 10:21:35 up 12:16, 1 user, load average: 0.02, 0.13, 0.15
Tasks: 107 total, 1 running, 106 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1883708 total, 212440 free, 778244 used, 893024 buff/cache
KiB Swap: 2097148 total, 2094288 free, 2860 used. 790064 avail Mem
{noformat}
This started off as about 650000 free last night. looking at a asterisk debug I see pjsip running out of memory. Before I go over to Asterisk on this has anyone been seeing anything weird?

Thanks leon

some of the asterisk debug
{noformat}
[2017-09-06 09:45:48] ERROR[7284] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[3640] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[3640] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[3640] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[3640] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[7285] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[7285] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[7285] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[7285] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[7285] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 09:45:48] ERROR[2803] pjproject: ssl0x7f0b5c0ccd40 Renegotiation failed: Not enough memory (PJ_ENOMEM)
[2017-09-06 09:45:48] VERBOSE[2809] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:45384;transport=TLS is now Unreachable. RTT: 0.000 msec
[2017-09-06 09:54:06] VERBOSE[8236] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:38964;transport=TLS has been deleted
[2017-09-06 09:56:06] VERBOSE[8236] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:45367;transport=TLS has been deleted
[2017-09-06 09:59:30] VERBOSE[7602] pbx_variables.c: Setting global variable ‘SIPDOMAIN’ to ‘pbx.backwoodswireless.net’
{noformat}
more interesting debugs:
{noformat}
[2017-09-06 06:23:45] ERROR[32309] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 06:23:45] ERROR[32309] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 06:23:45] ERROR[7602] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 06:23:45] ERROR[2803] pjproject: ssl0x7f0b641d3e60 Renegotiation failed: Not enough memory (PJ_ENOMEM)
[2017-09-06 06:23:45] VERBOSE[2809] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:38938;transport=TLS is now Unreachable. RTT: 0.000 msec
[2017-09-06 06:25:44] VERBOSE[7163] res_pjsip_registrar.c: Added contact ‘sips:702@70.44.10.180:45328;transport=TLS’ to AOR ‘702’ with expiration of 3600 seconds
[2017-09-06 06:25:44] VERBOSE[8236] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:45328;transport=TLS has been created
[2017-09-06 06:25:44] ERROR[31134] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 06:25:44] ERROR[31134] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 06:25:44] ERROR[31134] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702



[2017-09-06 06:25:44] ERROR[8841] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 06:25:44] ERROR[7163] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 06:25:44] ERROR[2803] pjproject: ssl0x7f0b4c626b10 Renegotiation failed: Not enough memory (PJ_ENOMEM)
[2017-09-06 06:25:44] VERBOSE[2809] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:45328;transport=TLS is now Unreachable. RTT: 0.000 msec
[2017-09-06 06:33:36] ERROR[2803] pjproject: sip_endpoint.c Error processing packet from 70.44.10.180:5060: Missing required header(s) (PJSIP_EMISSINGHDR) Via [code 171050]:
SIP/2.0 400 Bad Request
From: sip:701@209.222.10.59;tag=bcd2606e-5c15-492c-b6d3-0c934bb00eb4
To: sip:701@70.44.10.180;tag=9ffa1a1b62bb6a06
Call-ID: 62929999-00d7-44ed-bb3f-a43ec24ff7a9
CSeq: 6193 NOTIFY
User-Agent: Grandstream GXP2020 1.2.5.3
Warning: 398 "You are not allowed to CANCEL an established dialog"
Content-Length: 0

– end of packet.
[2017-09-06 06:34:03] VERBOSE[8236] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:38926;transport=TLS has been deleted
[2017-09-06 06:34:04] ERROR[2803] pjproject: sip_endpoint.c Error processing packet from 70.44.10.180:5066: Missing required header(s) (PJSIP_EMISSINGHDR) Via [code 171050]:
SIP/2.0 400 Bad Request
From: sip:702@209.222.10.59;tag=2c13f4fb-d808-43ef-ba65-b6455e6f978e
To: sip:702@70.44.10.180;tag=86962d2d9f6b5e8b
Call-ID: b6f21b3c-85c5-44bc-a7b7-6b8400528244
CSeq: 4265 NOTIFY
User-Agent: Grandstream GXP2020 1.2.5.3
Warning: 398 "You are not allowed to CANCEL an established dialog"
Content-Length: 0

– end of packet.
[2017-09-06 06:35:52] VERBOSE[2809] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:45312;transport=TLS has been deleted
[2017-09-06 06:36:02] VERBOSE[2767] asterisk.c: Remote UNIX connection
[2017-09-06 06:36:02] VERBOSE[2871] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 06:36:02] VERBOSE[2767] asterisk.c: Remote UNIX connection
[2017-09-06 06:36:02] VERBOSE[2873] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 06:36:02] VERBOSE[2767] asterisk.c: Remote UNIX connection
[2017-09-06 06:36:02] VERBOSE[2875] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 06:38:40] VERBOSE[2809] res_pjsip/pjsip_configuration.c: Contact 701/sip:701@70.44.10.180:5060;transport=TCP is now Unreachable. RTT: 0.000 msec
[2017-09-06 06:41:34] ERROR[2803] pjproject: sip_transport.c Error processing 161 bytes packet from TCP 70.44.10.180:5066 : PJSIP syntax error exception when parsing ‘’ header on line 2 col 7:
SIP/2.0 400 Bad Request
CSeq: User-Agent: Grandstream GXP2020 1.2.5.3
Warning: 398 "You are not allowed to CANCEL an established dialog"
Content-Length: 0

– end of packet.
[2017-09-06 06:41:34] ERROR[2803] pjproject: sip_endpoint.c Error processing packet from 70.44.10.180:5066: Missing required header(s) (PJSIP_EMISSINGHDR) Via [code 171050]:
SIP/2.0 400 Bad Request
From: sip:702@209.222.10.59;tag=c69e843a-2268-4b15-8253-752439dec7ec
To: sip:702@70.44.10.180;tag=750369d3-f4fa-406f-a596-7b68de87290e
Call-ID: cce7be3d-4c88-498f-82dd-bb3b58bac473
CSeq: 15435 NOTIFY
User-Agent: Grandstream GXP2020 1.2.5.3
Warning: 398 "You are not allowed to CANCEL an established dialog"
Content-Length: 0

– end of packet.
[2017-09-06 06:41:36] VERBOSE[2809] res_pjsip/pjsip_configuration.c: Contact 702/sip:702@70.44.10.180:5066;transport=TCP has been deleted
[2017-09-06 07:13:45] VERBOSE[8841] res_pjsip_registrar.c: Added contact ‘sips:702@70.44.10.180:38939;transport=TLS’ to AOR ‘702’ with expiration of 3600 seconds
[2017-09-06 07:13:45] VERBOSE[2809] res_pjsip/pjsip_configuration.c: Contact 702/sips:702@70.44.10.180:38939;transport=TLS has been created
[2017-09-06 07:13:45] ERROR[32309] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
[2017-09-06 07:13:45] ERROR[32309] res_pjsip.c: Error 70007 ‘Not enough memory (PJ_ENOMEM)’ sending NOTIFY request to endpoint 702
{noformat}
Thanks leon

jcolpAsterisk Developer
2h

I haven’t seen any issues with such things and I’m aware of a few different major installs. It may be unique to your environment or what you are doing. More information would be useful. How many endpoints, subscriptions, type of transport, etc.
wa4zlw
1h

AM using PJSip for endpoints. THis all worked under Asterisk 13/fpbx 12. very weird
basically I have three physical extensions 701-703. 703 is not connected now so just two 701 and 702. I have multiple devices talking to those extensions around the house (all grandstream and a zoiper android client which I disabled last night to debug this) On my Watchguard firewall I noticed in my traffic management window I was noticed weird sip behavior with lots of sip traffic. see this display which is no where near as bad as it was last night going over 120kbps!

image
image.png738x762 29.5 KB

Usually the SIP traffic is a flat line with very low bandwidth. It was hoping all around last night.
Also PJSIP extensions keep dropping out and I have to wait for the timers to expire. Going the debug log from last night you can see errors from the GXP2020 phone up above near the end of my original post.

On 701 I have PJSIP set for 8 and on 702 I have it set to 6. I usually add an extra 2 count in case things get out of sync

I have MWI subscriptions set on.

WOuld you like to get access to the server? If so, please post me privately wa4zlw@arrl.net

WHat else do you need? I’ve been running top all night and the free memory keeps going down pops up a bit which is what I would expect but something doesnt seem right to me

THanks leon
jcolpAsterisk Developer
1h

I don’t provide one on one help like that. You’d need to look at the log before errors occur to see if anything looks out of the ordinary, for example is the Asterisk server under attack?
wa4zlw
1h

I also have BLF setup as well for the extensions

no there is a firewall that freepbx runs so things are locked down.
the log entries I entered into the ticket is whta I thought unusual and why I added them to the ticket.

I looked at the previous ticket and it said to open a new one if you were not the original person which is what I did.

One thing I did do is cut down from 3600 --> 900 seconds the SIP max time

other than that I’m stumped.

thanks leon
wa4zlw
1h

here’s another unregistration/disconnection:
{noformat}
[2017-09-06 11:53:57] VERBOSE[21467] asterisk.c: Remote UNIX connection
[2017-09-06 11:53:57] VERBOSE[15247] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 11:53:57] VERBOSE[21467] asterisk.c: Remote UNIX connection
[2017-09-06 11:53:57] VERBOSE[15249] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 11:53:57] VERBOSE[21467] asterisk.c: Remote UNIX connection
[2017-09-06 11:53:57] VERBOSE[15251] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 12:02:41] VERBOSE[369] res_pjsip/pjsip_configuration.c: Contact 701/sip:701@70.44.10.180:44141;transport=TLS is now Unreachable. RTT: 0.000 msec
[2017-09-06 12:36:03] VERBOSE[21467] asterisk.c: Remote UNIX connection
[2017-09-06 12:36:03] VERBOSE[22779] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 12:36:03] VERBOSE[21467] asterisk.c: Remote UNIX connection
[2017-09-06 12:36:03] VERBOSE[22781] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 12:36:03] VERBOSE[21467] asterisk.c: Remote UNIX connection
[2017-09-06 12:36:03] VERBOSE[22783] asterisk.c: Remote UNIX connection disconnected
{noformat}
wa4zlw
1h

more keep dropping like flies
{noformat}
[2017-09-06 12:36:03] VERBOSE[21467] asterisk.c: Remote UNIX connection
[2017-09-06 12:36:03] VERBOSE[22783] asterisk.c: Remote UNIX connection disconnected
[2017-09-06 12:49:43] VERBOSE[369] res_pjsip/pjsip_configuration.c: Contact 701/sip:701@70.44.10.180:44142;transport=TLS is now Unreachable. RTT: 0.000 msec
[2017-09-06 12:49:44] VERBOSE[369] res_pjsip/pjsip_configuration.c: Contact 701/sip:701@70.44.10.180:44143;transport=TLS is now Unreachable. RTT: 0.000 msec
{noformat}
jcolpAsterisk Developer
33m

That would mean that the TLS connection has dropped, or the endpoint did not respond to our OPTIONS request.
wa4zlw
24m

my latency to the pbx is like 40ms

and this just happened
{noformat}
[2017-09-06 12:59:13] VERBOSE[369] res_pjsip/pjsip_configuration.c: Contact 701/sip:701@70.44.10.180:44142;transport=TLS has been deleted
[2017-09-06 12:59:13] VERBOSE[369] res_pjsip/pjsip_configuration.c: Contact 701/sip:701@70.44.10.180:44141;transport=TLS has been deleted
[2017-09-06 12:59:13] VERBOSE[369] res_pjsip/pjsip_configuration.c: Contact 701/sip:701@70.44.10.180:44143;transport=TLS has been deleted
{noformat}
this is going on all day up and down…what about the other errors above?

Thanks leon
jcolpAsterisk Developer
22m

I don’t really have anything to add. You can file an issue[1] with all the information you can. There’s no timeline on when it would get looked into, and as it only seems to be impacting you it could prove difficult to figure out.

[1] https://issues.asterisk.org/jira
Comments:By: Asterisk Team (asteriskteam) 2017-09-06 12:47:14.520-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

By: Leon Zetekoff (wa4zlw) 2017-09-06 18:19:08.003-0500

there is definitely a memory leak. I rebooted the pbx and here's the TOP. Before I  rebooted it mem free was about 200000 and I was getting those memory errors again
rebooted 9/6/17 @ 710pm


By: Leon Zetekoff (wa4zlw) 2017-09-06 18:32:51.628-0500

another item is the phones still show registered yet the pbx doesnt :-(

By: Richard Mudgett (rmudgett) 2017-09-06 19:07:30.532-0500

There isn't any information to help identify where in Asterisk memory may be leaking.

* Enabling MALLOC_DEBUG [1] and periodically looking at CLI "memory show summary" output for to see which module file memory is growing.  Since you are running out of memory, it should be fairly obvious which module is consuming the memory.  A CLI "memory show summary <file>" will show which allocations in that module are leaking.  Please attach captured output as a {{.txt}} file [2].
* Are you using bundled pjproject? [3]  When using bundled pjproject we also get a view into the memory consumption of pjproject with MALLOC_DEBUG.

[1] https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag
[2] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information
[3] https://wiki.asterisk.org/wiki/display/AST/PJSIP-pjproject

By: Leon Zetekoff (wa4zlw) 2017-09-06 19:46:45.431-0500

Hi Richard.....I am using precompiled binaries provided by freepbx. Never had this problem with Asterisk 13/Freepbx 12 (Incredible PBX).
Now that I have the debugs running per the steps I have to ASSUME a logfile is being written somewhere and where wouold that be?

Thanks leon



By: Leon Zetekoff (wa4zlw) 2017-09-06 19:48:14.357-0500

9/6/17 @ 845PM asterisk debugs running


By: Leon Zetekoff (wa4zlw) 2017-09-06 20:09:35.952-0500

I'm piping the output to a file from the console


By: Leon Zetekoff (wa4zlw) 2017-09-06 22:13:49.146-0500

9/6 1111PM

free mem 370996 still capturing logs more in the morning
all devices are registered


By: Leon Zetekoff (wa4zlw) 2017-09-07 08:08:50.586-0500

Stopped debug 9/7/17 @ 0850 AM

one endpoint registered


By: Leon Zetekoff (wa4zlw) 2017-09-07 08:13:26.568-0500

two debug files ran over night for pjsip/memory leak endpoint disconnections

used 7zip to compress @ maximum level since .zip was > 10mb

By: Leon Zetekoff (wa4zlw) 2017-09-07 08:47:54.419-0500

and this again in the debug log

{noformat}
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[3089] res_pjsip.c: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 702
[2017-09-07 09:44:17] ERROR[2740] pjproject: ssl0x7fbae40e0890 Renegotiation failed: Not enough memory (PJ_ENOMEM)
[2017-09-07 09:44:17] VERBOSE[16960] res_pjsip/pjsip_configuration.c: Contact 702/sip:702@70.44.10.180:48716;transport=TLS is now Unreachable. RTT: 0.000 msec
[2017-09-07 09:44:17] VERBOSE[16960] res_pjsip/pjsip_configuration.c: Endpoint 702 is now Unreachable
{noformat}

By: Richard Mudgett (rmudgett) 2017-09-07 10:43:06.823-0500

The attached logs might be useful but it was not what I was asking for.  Pasting more output of 'top' or out of memory log messages to the issue does not help narrow down what is leaking.

The "memory show summary" command is available when Asterisk is compiled with MALLOC_DEBUG enabled.  See ASTERISK-27067 for examples of the output of the asterisk Command Line Interface (CLI) output of the "memory show summary" that we need.  That output shows a breakdown of which modules have memory allocated.  One of the modules should have a huge amount of memory allocated.  That module needs to be drilled down further to see which allocations are not being released using the CLI "memory show summary <file>" command.

By: Leon Zetekoff (wa4zlw) 2017-09-07 10:50:37.094-0500

Hi Richard...ugh...I'll run it again i was following the instructions from here https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information

Thanks leon

By: Leon Zetekoff (wa4zlw) 2017-09-07 11:11:56.466-0500

Hi Richard...the comment I previously posted was deleted? the CLI has no memory show memory command as I pasted into the comment.

Suggestions please? THanks ldz

By: Richard Mudgett (rmudgett) 2017-09-07 11:21:27.742-0500

Please do not paste large output to comments [1].  It is not helpful.  Please attach files to the issue and reference things in the attached files in the issue comments instead.

You have to rebuild Asterisk [2] from source and enable the Asterisk compiler flag MALLOC_DEBUG using menuselect to get the CLI "memory show summary" commands.

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines
[2] https://wiki.asterisk.org/wiki/display/AST/Installing+Asterisk+From+Source

By: Leon Zetekoff (wa4zlw) 2017-09-07 11:50:20.671-0500

Hi Richard...ugh ok I need to get with freepbx then since they provided the asterisk binaries.
in the meantime can you look at the debug output I posted this morning?

THanks leon

By: Rusty Newton (rnewton) 2017-09-07 18:57:35.493-0500

Leon, please attach any debug to the issue as a file with the .txt extension when possible. This makes it easier to find all in one place and easy to view in the browser. It helps prevent so much scrolling through comments.

By: Leon Zetekoff (wa4zlw) 2017-09-07 19:19:40.281-0500

Hi Rusty, Richard et al...I couldnt since there is a 10mb limit hence my only option was to compress it to an archive.
FreePBX says they can't generate what you ask for not simple per their words. Also will cause performance issues.
Supposedly they have details how to do it but I couldnt find it anywhere on their wiki.

So barring no new code for me to try can you go through the debugs I sent this morning? I had to reboot the PBX again since it ran out of memory again.

This is getting very frustrating.

Thanks leon

PS: This is the file I uploaded this morning to the ticket:
FreePBX.7z
   Today 9:13 AM
   5.43 MB



By: Leon Zetekoff (wa4zlw) 2017-09-08 13:53:44.715-0500

Hi folks...I'm running out of memory multiple times a day requiring manual reboot. This is not good for a live system. Is there anything you can please do to look at the stuff I already sent and/or see what the differences in from asterisk 13 to asterisk 14 with pjsip?

Thanks in advance leon

By: Joshua C. Colp (jcolp) 2017-09-08 13:56:07.085-0500

[~wa4zlw] We'll look at things as soon as we are able. As this is an open source project there is no guarantee on when things will get looked into or resolved.

By: Leon Zetekoff (wa4zlw) 2017-09-13 14:47:24.449-0500

Here is current taskprocessors since i see others reporting similar issues as well https://community.asterisk.org/t/memory-leak-consumes-16gb-in-less-then-24-hours/71916

here is my taskprocessors:

[Edit by Rusty - Please attach debug to the issue in a .txt file as per the guidelines]

By: Leon Zetekoff (wa4zlw) 2017-09-14 09:54:06.961-0500

more debugs
[Edit by Rusty - Please attach debug to the issue in a .txt file as per the guidelines]

By: Rusty Newton (rnewton) 2017-09-14 17:56:34.215-0500

I've looked a bit into what you have provided and I can't find an obvious issue. I don't think we can do much here without a better defined method of reproduction to produce the leak. That is you would need to provide an exact configuration for us to setup and a serious of steps to reproduce the issue. The other item that is really needed is the MALLOC_DEBUG output.

As we have already asked you to do previously, again, please attach all further debug to the issue, either in compressed archive files or in individual .txt files.

https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines

By: Leon Zetekoff (wa4zlw) 2017-09-14 17:57:20.680-0500

The debugs were >10mb so I had to put them into an archive there was no other way to do it since there is a size restriction

By: Leon Zetekoff (wa4zlw) 2017-09-14 18:03:59.727-0500

Hi rusty....seems I am not the only one with memory issues.
FOr starters take the latest Freepbx distro and install it. That will get you the codebase I am running asterisk 14/freepbx 14.
freepbx doesnt make it easy so I asked them to provide a test with the compile flag on and they declined said I had to do it. SInce I dont have a development environment and not a linux guru I am not going to do this at this time as I dont have the time for it.

Also putting the compile flag on will change the image that is producted and potentially could make it go away or get worse. I know back in my embedded days putting debug code in would usually change where buffers went and things would magically start to work.

If you want I can give you access to the VM guest so you can take a looksee.

thanks leon

By: Leon Zetekoff (wa4zlw) 2017-09-14 18:05:08.675-0500

any new debugs will be attached as txt unless they are over the 10mb limit again

By: Leon Zetekoff (wa4zlw) 2017-09-15 07:53:43.631-0500

woke up this morning and nothing registered. Attached debugoutput3.txt

Please note that I have seen the PJSIP out of memory errors when the TOP free mem number was much higher. Attached includes core show taskprocessors as well as TOP

Leon

By: Leon Zetekoff (wa4zlw) 2017-09-15 07:54:23.189-0500

per previous comment just made

many pjsip out of memory errors

By: Leon Zetekoff (wa4zlw) 2017-09-15 08:07:42.756-0500

When i did an amportal stop free memory jumped up as expected but not totally to where I believe it was when rebooted

top - 08:53:43 up 14:14,  2 users,  load average: 0.90, 0.53, 0.46
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.7 us,  1.0 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1883708 total,   856272 free,   644384 used,   383052 buff/cache
KiB Swap:  2097148 total,  2007600 free,    89548 used.   939504 avail Mem

as I sit here and watch it lookos like linux is releasing memory slowly but is ping ponging up and down slightly. I will let things sit PRIOR to a amportal restart

when PHP goes active free memory drops and then comes back up.

just prior to amportal restart


just after restart




By: Leon Zetekoff (wa4zlw) 2017-09-15 09:18:09.320-0500

well that was fast see attached debugoutput4.txt


By: Leon Zetekoff (wa4zlw) 2017-09-15 09:18:37.858-0500

more pjsip memory errors

By: Leon Zetekoff (wa4zlw) 2017-09-15 09:19:20.289-0500

if you want these task connections in a text file please let me know as well
thanks

By: Leon Zetekoff (wa4zlw) 2017-09-15 09:21:59.475-0500

this time I've completely rebooted the VM instance and will report back later once it dies again :-(

By: Rusty Newton (rnewton) 2017-09-15 09:45:14.137-0500

Leon, I've edited your comments again to remove the debug and command line output that you keep posting in comments. Please stop it, as we have already requested that you follow the guidelines.

In addition, we cannot move forward without the MALLOC_DEBUG output that Richard has requested. If you are unable to provide it, then we will close out the issue as we have nothing further that we can do here. The data you have provided does not contain anything helpful to solving the issue at the current time.

By: Leon Zetekoff (wa4zlw) 2017-09-15 09:49:52.865-0500

the ones you did were older ones richard.
I added two txt files this morning. those errors messages show a specific line where the error is happening.
I have also offered you guys access to my server so you can investigate.

Leon

By: Leon Zetekoff (wa4zlw) 2017-09-15 13:03:45.361-0500

Just got memory error with almost 1gb free ram. But this time we got TLS errors first.

See debug5

I believe this started when I enabled my zoiper fone on my android. i Notice sip traffic rises tremendously when this happens. Turned it back off

By: Leon Zetekoff (wa4zlw) 2017-09-15 13:05:15.286-0500

more SIP memory errors with almost 1gb free ram but first TLS errors then SIP errors

I believe this started when I enabled my zoiper fone on my android. i Notice sip traffic rises tremendously when this happens. Turned it back off

By: Leon Zetekoff (wa4zlw) 2017-09-16 07:16:48.777-0500

updated debug5 start to stop data


By: Leon Zetekoff (wa4zlw) 2017-09-16 07:17:20.640-0500

updated from start to stop of PBX

By: Leon Zetekoff (wa4zlw) 2017-09-18 10:01:14.299-0500

new debug6 start top stop of pbx


By: Leon Zetekoff (wa4zlw) 2017-09-18 10:01:38.248-0500

another debug start to stop

By: Leon Zetekoff (wa4zlw) 2017-09-19 08:38:49.149-0500

another debug start to stop - file debug7

lately two different pjzip messages popping up,:

[2017-09-19 09:29:46] ERROR[4736]: res_pjsip.c:3770 endpt_send_request: Error 70007 'Not enough memory (PJ_ENOMEM)' sending NOTIFY request to endpoint 701
[2017-09-19 09:29:47] ERROR[2795]: pjproject:0 <?>:          ssl0x7f8a7805bcb0 Renegotiation failed: Not enough memory (PJ_ENOMEM)


By: Leon Zetekoff (wa4zlw) 2017-09-19 08:39:14.288-0500

another log

By: Rusty Newton (rnewton) 2017-09-19 13:37:29.976-0500

We cannot move forward without the MALLOC_DEBUG output that Richard has requested. The data you have provided does not contain anything helpful to solving the issue at the current time.

Therefore we are closing out the issue for now.

Feel free to open a new issue when you are able to gather MALLOC_DEBUG output for the issue, along with debug logs from the run.

By: Leon Zetekoff (wa4zlw) 2017-09-19 18:35:14.697-0500

different error messages function can only be used on SIP channelds plus out of memory

By: Asterisk Team (asteriskteam) 2017-09-19 18:35:15.418-0500

This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable.

By: Leon Zetekoff (wa4zlw) 2017-09-19 18:36:00.894-0500

more logs function can only be used on SIP channels
out of memory

By: Leon Zetekoff (wa4zlw) 2017-09-22 16:53:26.459-0500

new error related to tls

By: Asterisk Team (asteriskteam) 2017-09-22 16:53:27.749-0500

This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable.

By: Leon Zetekoff (wa4zlw) 2017-09-22 16:53:45.745-0500

TLS errors

By: Rusty Newton (rnewton) 2017-10-02 15:50:48.086-0500

Leon, we've already informed you that any further debug other than what we have requested (MALLOC_DEBUG) likely won't help.

Do you have anything else you want to attach to the issue before we close it out again?

By: Leon Zetekoff (wa4zlw) 2017-10-03 09:08:15.383-0500

Rusty...I am from the old school and believe to DOCUMENT as much as I can. When I saw more and different errors I posted it to the ticket. I can't believe that the info in the debugs which have the module and line number can't point you in a direction. I am not the only one having these issues.

WHy isn't that compile flag used as a default?

Thanks leon

By: Leon Zetekoff (wa4zlw) 2017-10-04 01:34:47.543-0500

GM ... I've reinstalled twice now. The second time I have not been getting out of memory errors and registrations seemed to be holding on the endpoints. Once I upgraded asterisk to latest asterisk endpoints are dropping again but still no out of memory errors.

this is the ISO I installed from FreePBX

https://downloads.freepbxdistro.org/ISO/SNG7-FPBX-64bit-1707-1.iso
There's definitely something fubar with PJSIP but I can't put my finger on it.

Leon

By: Leon Zetekoff (wa4zlw) 2017-10-07 02:51:16.448-0500

Rusty...here's another comment from PBX developer. The FreePBX folks aren't the only ones that have this same issue:

2- The Flag Mallog Debug was not included due make asterisk slow.

This is from the Ombutel folks.

So it seems to me the ball is in Asterisks courton this?

Thanks leon

By: Malcolm Davenport (mdavenport) 2017-10-09 11:53:14.868-0500

Requested information has not been provided by user.  Issue is being suspended until such time as the requested information has been provided.