[Home]

Summary:ASTERISK-28431: Asterisk memory increases and CentOS OEM killer kills Asterisk process
Reporter:Deepak Lohani (deepaklohani)Labels:pjsip
Date Opened:2019-05-30 07:31:55Date Closed:2019-07-01 12:00:02
Priority:MajorRegression?
Status:Closed/CompleteComponents:General
Versions:13.25.0 Frequency of
Occurrence		Frequent
Related
Issues:
Environment:CentOS 7.6.1810 core kernel 3.10.0-957.el7.x86_64 RAID1 configured for OS and RAID5 configured for dataAttachments:( 0) End_Date_31May.zip
( 1) Hardware_info.log
( 2) memory_23_May_2019.txt
( 3) memory_30_May_2019.txt
( 4) Top_30-05-2019-1700.jpg
Description:We are continuously facing an issue of High RAM consumption over Asterisk 13.25.0 with Centos 7.6. Here Asterisk will consume all available RAM (32GB) of our server over a 15 days period,
and consume whole swap portion too.In such scenario OOM-killer invoked by Centos kills the process which has consumed highest memory i.e Asterisk.
After that server will resume total RAM and runs smoothly.
Comments:By: Asterisk Team (asteriskteam) 2019-05-30 07:31:56.625-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.
By: Deepak Lohani (deepaklohani) 2019-05-30 07:44:25.932-0500

memory show summary  command output of dated 23 May 2019 and 30 May 2019
centos top command output of dated 30 May 2019.
Hardware info log details

By: Benjamin Keith Ford (bford) 2019-05-30 10:32:55.746-0500

I've looked through the logs you've provided, and noticed that {{chan_sip.c}}, {{res_rtp_asterisk.c}}, and {{smoother.c}} seem to be the main sources of memory consumption. Here's some things that can help track the problem down:
# Try PJSIP. If this is a chan_sip issue, PJSIP is the correct solution for it, as chan_sip is now under extended support [1].
# If possible, try using a different system or virtual environment and seeing if the problem persists. We want to eliminate the possibility that it's not Asterisk causing the problem.
# Provide output of {{memory show allocations}} the same way you did with {{memory show summary}}, with a large enough amount of time between them so that we can compare allocations.
# If you're using conferencing, are you using dialplan to set them up dynamically? For example:
{code}
...
same => n,SET(CONFBRIDGE(menu,0)=hello-world)
...
{code}
There's a known memory leak with this [2].
# Can you provide any other information that may be impacting the system? Other programs running in conjunction with Asterisk, any custom patches you might have applied to your build, or anything along those lines that we should be aware of?

[1]: https://wiki.asterisk.org/wiki/display/AST/Asterisk+Module+Support+States
[2]: ASTERISK-28422
By: Deepak Lohani (deepaklohani) 2019-05-31 08:19:34.087-0500

To overcome this issue we have used the Asterisk version 13.17.0 with Centos 7.4 also but still issue persist,even we have changed the servers also .In these servers RAID1 configured for OS and RAID5 configured for data.The leakage in asterisk is approx 40MB only whereas the centos is showing in GB. When we restart asterisk, all system memory becomes available again

'memory show allocation' and top command output , Kindly check attachments
By: Joshua C. Colp (jcolp) 2019-05-31 08:24:15.295-0500

The attachments are "memory show summary" and not "memory show allocations". The "show allocations" actually shows, down to the function and line number, where the allocations came from.
By: Deepak Lohani (deepaklohani) 2019-05-31 08:24:39.553-0500

Attached files is the output of 'memory show allocations' and top command output
By: Joshua C. Colp (jcolp) 2019-06-17 04:56:46.085-0500

Thank you for taking the time to report this bug and helping to make Asterisk better. Unfortunately, we cannot work on this bug because your description did not include enough information. Please read over the Asterisk Issue Guidelines [1] which discusses the information necessary for your issue to be resolved and the format that information needs to be in. We would be grateful if you would then provide a more complete description of the problem. At a minimum, we need:

1. The specific steps or actions you took that caused you to encounter the problem.
2. The behavior you expected and the location of documentation that led you to that expectation.
3. The behavior you actually encountered.

To demonstrate the issue in detail, please include Asterisk log files generated per the instructions on the wiki [2]. If applicable, please ensure that protocol-level trace debugging is enabled, e.g., 'sip set debug on' if the issue involves chan_sip, and configuration information such as dialplan and channel configuration.

Thanks!

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines

[2] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information


By: Joshua C. Colp (jcolp) 2019-06-17 04:57:38.165-0500

In particular we need to also see how you are using Asterisk. Configuration/dialplan that exhibits the problem.
By: Asterisk Team (asteriskteam) 2019-07-01 12:00:02.209-0500

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines