[Home]

Summary:ASTERISK-27591: Frack errors in stasis.c and memory leakage
Reporter:Siruja Maharjan (sirujam)Labels:pjsip
Date Opened:2018-01-17 02:36:45.000-0600Date Closed:2018-07-27 05:38:58
Priority:MajorRegression?
Status:Closed/CompleteComponents:Core/Stasis
Versions:14.3.0 15.1.2 15.1.4 Frequency of
Occurrence		Occasional
Related
Issues:
Environment:RHEL 7.4, Centos 7.3Attachments:( 0) backtrace_stasis.log
( 1) debug_log
( 2) extensions.conf
( 3) frack_jan22.txt
( 4) manager.conf
( 5) messages_frack.txt
( 6) messages_log.txt
( 7) pjsip.conf
( 8) refs.txt
Description:The call scenario is as below.
A calls B. Then via AMI asterisk sends a missed call to B.

It was observed that memory usage by asterisk process was in increasing mode. Additionally after around 100000 calls following message is logged in messages file.

{quote}
[Jan  9 18:06:39] ERROR[68501] stasis.c: Excessive refcount 100000 reached on ao2 object 0x1b5fb00
[Jan  9 18:06:39] ERROR[68501] stasis.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x1b5fb00 (0)
{quote}

The error is observed after around 100000 calls after each restart.
Backtrace is attached.
Comments:By: Asterisk Team (asteriskteam) 2018-01-17 02:36:47.388-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Kevin Harwell (kharwell) 2018-01-17 10:14:22.667-0600

Please attach the dialplan used in the scenario and any other relevant conf files (sip.conf or pjsip.conf, manager.conf, etc...). Also as it sounds like by running the same scenario over again causes the issue can you post a debug log of at least one run of the scenario. More info on how to collect debug information can be found [here|https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information].

Thanks!
By: Corey Farrell (coreyfarrell) 2018-01-17 10:23:25.341-0600

If possible please also collect a reference debug log \[1\].  You would need to do a more limited test, thousands of calls would create a log so large that the refcounter.py script would run out of memory processing the results.  Please also specify the exact version you test with (and if you added any patches).  I have recently done some work on stasis which I'm pretty sure you are not running but want to confirm.

\[1\] https://wiki.asterisk.org/wiki/display/AST/Reference+Count+Debugging
By: Siruja Maharjan (sirujam) 2018-01-22 23:29:49.517-0600

Pleaes find the configuration files. I have attached debug information for one scenario. Also, refs.txt is uploaded. I changed 100000 limit to 5000 (astobj2.c) to get the error message.

P.S.
While uploading files, the issue went to triage process again.
By: George Joseph (gjoseph) 2018-01-23 08:22:01.403-0600

[~coreyfarrell] I assigned this to you to at least check if it's related to your recent work.  Put the issue back in triage if it isn't.

By: Corey Farrell (coreyfarrell) 2018-01-23 11:37:10.163-0600

It looks like none of the recent work I've done on stasis (core or res_stasis) has made it to a release yet.  Unfortunately the refs.txt only shows a minor leak from ODBC.  The line numbers in the latest FRACK matches 15.1.x but are slightly off from 15.2.0.
By: George Joseph (gjoseph) 2018-01-23 15:19:52.779-0600

Which version of asterisk were you running when you captured the logs?
"core show version"

By: Siruja Maharjan (sirujam) 2018-01-23 21:11:54.852-0600

Please find the version
{quote}
Asterisk 15.1.4 built by root @ localhost on a x86_64 running Linux on 2017-12-19 05:02:40 UTC
{quote}

By: Friendly Automation (friendly-automation) 2018-07-27 05:39:01.128-0500

Change 9660 merged by Joshua Colp:
devicestate: Don't create topic when change isn't cached.

[https://gerrit.asterisk.org/9660|https://gerrit.asterisk.org/9660]
By: Friendly Automation (friendly-automation) 2018-07-27 05:39:15.666-0500

Change 9659 merged by Joshua Colp:
devicestate: Don't create topic when change isn't cached.

[https://gerrit.asterisk.org/9659|https://gerrit.asterisk.org/9659]
By: Friendly Automation (friendly-automation) 2018-07-27 06:10:40.859-0500

Change 9658 merged by Jenkins2:
devicestate: Don't create topic when change isn't cached.

[https://gerrit.asterisk.org/9658|https://gerrit.asterisk.org/9658]
By: Friendly Automation (friendly-automation) 2018-07-27 06:11:14.060-0500

Change 9661 merged by Joshua Colp:
devicestate: Don't create topic when change isn't cached.

[https://gerrit.asterisk.org/9661|https://gerrit.asterisk.org/9661]
By: Siruja Maharjan (sirujam) 2018-08-09 10:10:10.323-0500

After patch upgrade, the memory utilization is more or less stable. Memory leakage has not been observed till now.

Thanks,
Siruja
By: Asterisk Team (asteriskteam) 2018-08-09 10:10:10.629-0500

This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable.