[Home]

Summary:ASTERISK-28180: ami: High memory (increased upto 15 GBs) while pushing 500 calls continuously for 7 hours
Reporter:Mohit Dhiman (mohitdhiman)Labels:Memory asterisk
Date Opened:2018-11-25 06:28:31.000-0600Date Closed:2018-12-06 10:49:55.000-0600
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Core/ManagerInterface
Versions:13.21.0 Frequency of
Occurrence		Constant
Related
Issues:
is related toASTERISK-28197 stasis: ast_endpoint struct holds the channel_ids of channels past destruction in certain cases
Environment:Centos 7.2Attachments:( 0) memory_show_summary_during_call.txt
( 1) memory_show_summary_without_any_call.txt
Description:I am doing load testing by continuously pushing around 500 calls through AMI for around 6 hours, Asterisk is maintaining around 500 calls at any instance and continuously hanging up the call after few seconds.

Scenario:
500 channels are getting originated through AMI
Asterisk then dialled call to another asterisk which answers and randomly hangup the call after few minutes.

In around 4 hours asterisk memory started to increase and reaches upto 15 GBs in 7 hours.
Comments:By: Asterisk Team (asteriskteam) 2018-11-25 06:28:32.617-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Malcolm Davenport (mdavenport) 2018-11-25 13:30:48.567-0600

This is not a critical issue. Do not reclassify it as a critical issue unless you are one of the bug marshals or a member of the development team.
By: Mohit Dhiman (mohitdhiman) 2018-11-25 21:52:29.697-0600

OK, but this is a major issue for me as it affects the deployment process.
By: Joshua C. Colp (jcolp) 2018-11-26 04:43:55.975-0600

While a problem may be major to you, that does not mean it is major to the project itself.
By: Benjamin Keith Ford (bford) 2018-11-26 14:48:05.123-0600

Sounds like this could be a leak somewhere. Can you provide some logs, including debug with Valgrind, and enable reference count debugging [1] in your asterisk.conf?

[1]: https://wiki.asterisk.org/wiki/display/AST/Reference+Count+Debugging
By: Mohit Dhiman (mohitdhiman) 2018-11-27 04:15:44.269-0600

ok i'll collect the necessary logs, meanwhile i am getting this warning " WARNING[21703][C-0002fdaf] taskprocessor.c: The 'subp:SIP/201-0000004d' task processor queue reached 500 scheduled tasks again" frequently

also i forgot to mention that as soon as memory consumption tends to increase AMI connection also keep on disconnecting.
By: Benjamin Keith Ford (bford) 2018-11-27 10:33:55.290-0600

The task processor messages are to be expected when putting that much load on a system, and it will slow other things down, including AMI. What are your system specs? If you are overloading your system, it can make Asterisk seem like the culprit, but this could be due to the system itself not being able to handle the load.
By: Mohit Dhiman (mohitdhiman) 2018-11-27 23:34:39.233-0600

i'm running Asterisk on a 32 core server with 32 GBs of RAM and 1 TB harddisk. also i'm running two asterisk on this system but the one with the AMI connection is leaking memory and the other one is running smooth with under 200 MBs of memory throughout the load.
Important Note: Asterisk with the AMI connection has plenty of customisation in it to support the business need.
By: Joshua C. Colp (jcolp) 2018-11-28 04:13:18.899-0600

Customization in what way? Asterisk modifications?

As well actual console log and configuration+application that can be used to reproduce the problem would also be good.
By: Mohit Dhiman (mohitdhiman) 2018-12-05 05:36:10.860-0600

I did some debugging with MALLOC_DEBUG flag enabled at 250 calls at any instance and goes as high as 2.5 GBs of memory consumption by asterisk and it shows the {{endpoints.c}} is taking 90% of consumed memory in function {{ast_endpoint_snapshot_create}} at line no. 444 that is {{snapshot = ao2_alloc_options(...)}} but after i stopped all the call flow then memory consumption reduced from 2.5 GBs to 1.3 GBs but {{memory show summary}} doesn't indicate any file that might even nearly be consuming this much memory.

also the modification is just of Action-Id attachment to AMI Actions and Events only.

*memory increase with time:* 150 MBs to 500 MBs in 3 hours, 500 MBs to 1 GBs in 15 minutes and 1 GBs to 2.5 GBs in under 3 minutes.
By: Mohit Dhiman (mohitdhiman) 2018-12-05 05:43:50.147-0600

attached the memory show summary output of during call and after call ended
By: Mohit Dhiman (mohitdhiman) 2018-12-05 23:37:41.306-0600

it maybe related to ASTERISK-28197 as i'm using Originate and Bridge to make the call.
By: Mohit Dhiman (mohitdhiman) 2018-12-06 10:48:42.513-0600

this is definitely related to ASTERISK-28197 because when i replaced the {{Originate}} and {{Bridge}} application with the {{Dial}} application, the issue resolves and no more memory leak observed after running the same load for 8 continuous hours.