[Home]

Summary:ASTERISK-26712: rtp: Continuously increase in memory of asterisk process and causing crash in the process.
Reporter:Mandeep Singla (MandeepSingla)Labels:
Date Opened:2017-01-11 06:12:55.000-0600Date Closed:2020-01-14 11:13:48.000-0600
Priority:MajorRegression?Yes
Status:Closed/CompleteComponents:Resources/res_rtp_asterisk
Versions:14.2.1 14.3.0 Frequency of
Occurrence		Constant
Related
Issues:
Environment:Linux ip-172-30-0-94.ap-southeast-1.compute.internal 3.10.0-327.36.2.el7.x86_64 #1 SMP Mon Oct 10 23:08:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux CentOS Linux release 7.2.1511 (Core)Attachments:( 0) backtrace.txt
( 1) CLI.txt
( 2) debug_log_123456
( 3) Memory_show_summary.txt
( 4) Top_Command_showing_asterisk_process_Memory.png
( 5) valgrind.txt
Description:I am getting continuously increase in memory of asterisk process and causing crash in the process.

after analyzing the "memory show summary" it is observed that rtp_engine.c can have mamory leak in code .
Comments:By: Asterisk Team (asteriskteam) 2017-01-11 06:12:56.591-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Joshua C. Colp (jcolp) 2017-01-13 09:59:01.920-0600

Thank you for taking the time to report this bug and helping to make Asterisk better. Unfortunately, we cannot work on this bug because your description did not include enough information. Please read over the Asterisk Issue Guidelines [1] which discusses the information necessary for your issue to be resolved and the format that information needs to be in. We would be grateful if you would then provide a more complete description of the problem. At a minimum, we need:

1. The specific steps or actions you took that caused you to encounter the problem.
2. The behavior you expected and the location of documentation that led you to that expectation.
3. The behavior you actually encountered.

To demonstrate the issue in detail, please include Asterisk log files generated per the instructions on the wiki [2]. If applicable, please ensure that protocol-level trace debugging is enabled, e.g., 'sip set debug on' if the issue involves chan_sip, and configuration information such as dialplan and channel configuration.

Thanks!

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines

[2] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information

In this case the output of the memory CLI commands that you've mentioned you've used, console output, and details about what the system is doing.
By: Mandeep Singla (MandeepSingla) 2017-01-15 23:35:37.233-0600

Attach  is Linux Top command output showing asterisk process memory usage of asterisk and this is continuosly increasing
By: Mandeep Singla (MandeepSingla) 2017-01-15 23:37:27.476-0600

I have attached Linux Top command snapshot showing 70% usage of asterisk process and this is continuosly increasing.Have a look on this.
By: Mandeep Singla (MandeepSingla) 2017-01-16 00:22:35.743-0600

Attached is output of "memory show summary" CLI command showing memory allocated by each astertisk source code file.

In this attachment rtp_engine memory is very high and also that is continuously increasing. So it is suspected that memory is not freed in rtp_engine.c file somewhere in source code causing  overall increase in memory of asterisk process.
By: Joshua C. Colp (jcolp) 2017-01-17 09:23:13.401-0600

Please include details about your environment including configuration, console output, and details about what is being done. Without this it would be VERY difficult to reproduce and isolate what is going on.
By: Mandeep Singla (MandeepSingla) 2017-01-18 07:12:24.457-0600

Environment details already present in Environment fileld above.

We are running TLS users (around 1000) on asterisk PJSIP. CLI aoutut is attached.
By: Rusty Newton (rnewton) 2017-01-18 17:26:01.400-0600

Your backtrace appears to contain a memory corruption. We need one or both of the following items to continue investigation of the issue:
1. Valgrind output. See https://wiki.asterisk.org/wiki/display/AST/Valgrind for instructions on how to use Valgrind with Asterisk.
2. MALLOC_DEBUG output. See https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag for instructions on how to use the MALLOC_DEBUG option.

Note that MALLOC_DEBUG and Valgrind are mutually exclusive options. Valgrind output is preferable, but will be more system resource intensive and may be difficult to get on a production system. In such a case, you may have better luck getting the necessary output from MALLOC_DEBUG.


By: Rusty Newton (rnewton) 2017-01-18 17:26:19.744-0600

Thank you for the crash report. However, we need more information to investigate the crash. Please provide:

1. A backtrace generated from a core dump using the instructions provided on the Asterisk wiki [1].
2. Specific steps taken that lead to the crash.
3. All configuration information necesary to reproduce the crash.

Thanks!

[1]: https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace


By: Rusty Newton (rnewton) 2017-01-18 17:32:49.871-0600

Please get the MALLOC_DEBUG and the backtrace mentioned in the two previous comments. In addition the log you attached is hard to read. If you can, just use Asterisk's logging facilities and attach the actual, original log. The guide is here: https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information

Make sure to get VERBOSE and DEBUG log levels, with both turned up to 5 or higher.

If you can get a log that matches your MALLOC_DEBUG and backtrace that would be excellent. In addition actual dialplan, (extensions.conf), endpoint configuration (pjsip.conf) and more detail about the call flow would help. Of course, be sure to sanitize any private information.

I know this is a lot to gather, but we won't likely be able to track it down without all of this.
By: Mandeep Singla (MandeepSingla) 2017-01-18 23:51:58.983-0600

Plz find attached backtrace  and valgrind output but when we executed using valgrind user were not able to register on asterisk may be it cause slow in preocessing.
By: Rusty Newton (rnewton) 2017-01-19 12:45:04.558-0600

I'm not sure I understand in regards to the reproduction under valgrind. Were you unable to reproduce the issue when running with valgrind?

If not, then you should try the MALLOC_DEBUG option as mentioned in the documentation.
By: Asterisk Team (asteriskteam) 2017-02-03 12:00:01.903-0600

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines
By: Mandeep Singla (MandeepSingla) 2017-03-08 02:10:27.464-0600

This is output log having MALLOC_DEBUG flag enabled.
By: Mandeep Singla (MandeepSingla) 2017-03-08 02:11:23.919-0600

File Name is :debug_log_123456
By: Asterisk Team (asteriskteam) 2017-03-08 02:11:24.294-0600

This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable.
By: Joshua C. Colp (jcolp) 2017-03-12 18:16:25.811-0500

What was the memory consumption at with the entries in this debug log? Can you also do "memory show summary" and "memory show allocations"? Were there any calls?

How was PJSIP built? As well - you need to be using the latest version of everything. We've fixed memory leaks.
By: Mandeep Singla (MandeepSingla) 2017-03-15 06:37:59.808-0500

Lemme install asterisk version 14.3.0.

By: Mandeep Singla (MandeepSingla) 2017-03-15 06:42:34.851-0500

Can you share the document for compete instalaltion of pjroject + asterisk 14.3.0
By: Joshua C. Colp (jcolp) 2017-03-15 08:23:47.576-0500

The wiki includes information[1] about building Asterisk with PJSIP using an easy bundled approach. If there are specific things you need help with then I'd suggest looking on the wiki and if you need further help using the community site[2].

[1] https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject#BuildingandInstallingpjproject-bundledUsingtheBundledVersionofpjproject
[2] https://community.asterisk.org/
By: Mandeep Singla (MandeepSingla) 2017-03-16 08:37:28.187-0500

I have installed asterisk 14.3.0 version astrrisk .

Still same issue, memory leak is there.

Plz check. If possible plz check it online on server.I can arrange the same.


Regards,
Mandeep
By: Rusty Newton (rnewton) 2017-03-17 13:08:02.269-0500

Can you update your logs, debug, backtrace along with valgrin/malloc output (if available) since you have installed the latest release?

bq. What was the memory consumption at with the entries in this debug log? Can you also do "memory show summary" and "memory show allocations"?

Please additionally attach extensions.conf if possible.

Attach all files in .txt format for accessibility.
By: Mandeep Singla (MandeepSingla) 2017-03-21 02:30:14.382-0500

Dear Rusty,

Can you check this online if possible.

I will provide all access of the system .

Regards,
Mandeep
By: Rusty Newton (rnewton) 2017-03-21 09:24:45.286-0500

{quote}
Can you check this online if possible.
I will provide all access of the system .
{quote}

No, I can't. The burden is on the issue reporter to provide everything that is needed to investigate or reproduce the issue.

If you need that sort of help, you might consider hiring a consultant. Or there may be other community members who could help you out.
By: Asterisk Team (asteriskteam) 2017-04-04 12:00:01.835-0500

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines