Summary: | ASTERISK-27591: Frack errors in stasis.c and memory leakage | ||
Reporter: | Siruja Maharjan (sirujam) | Labels: | pjsip |
Date Opened: | 2018-01-17 02:36:45.000-0600 | Date Closed: | 2018-07-27 05:38:58 |
Priority: | Major | Regression? | |
Status: | Closed/Complete | Components: | Core/Stasis |
Versions: | 14.3.0 15.1.2 15.1.4 | Frequency of Occurrence | Occasional |
Related Issues: | |||
Environment: | RHEL 7.4, Centos 7.3 | Attachments: | ( 0) backtrace_stasis.log ( 1) debug_log ( 2) extensions.conf ( 3) frack_jan22.txt ( 4) manager.conf ( 5) messages_frack.txt ( 6) messages_log.txt ( 7) pjsip.conf ( 8) refs.txt |
Description: | The call scenario is as below.
A calls B. Then via AMI asterisk sends a missed call to B. It was observed that memory usage by asterisk process was in increasing mode. Additionally after around 100000 calls following message is logged in messages file. {quote} [Jan 9 18:06:39] ERROR[68501] stasis.c: Excessive refcount 100000 reached on ao2 object 0x1b5fb00 [Jan 9 18:06:39] ERROR[68501] stasis.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x1b5fb00 (0) {quote} The error is observed after around 100000 calls after each restart. Backtrace is attached. | ||
Comments: | By: Asterisk Team (asteriskteam) 2018-01-17 02:36:47.388-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Kevin Harwell (kharwell) 2018-01-17 10:14:22.667-0600 Please attach the dialplan used in the scenario and any other relevant conf files (sip.conf or pjsip.conf, manager.conf, etc...). Also as it sounds like by running the same scenario over again causes the issue can you post a debug log of at least one run of the scenario. More info on how to collect debug information can be found [here|https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information]. Thanks! By: Corey Farrell (coreyfarrell) 2018-01-17 10:23:25.341-0600 If possible please also collect a reference debug log \[1\]. You would need to do a more limited test, thousands of calls would create a log so large that the refcounter.py script would run out of memory processing the results. Please also specify the exact version you test with (and if you added any patches). I have recently done some work on stasis which I'm pretty sure you are not running but want to confirm. \[1\] https://wiki.asterisk.org/wiki/display/AST/Reference+Count+Debugging By: Siruja Maharjan (sirujam) 2018-01-22 23:29:49.517-0600 Pleaes find the configuration files. I have attached debug information for one scenario. Also, refs.txt is uploaded. I changed 100000 limit to 5000 (astobj2.c) to get the error message. P.S. While uploading files, the issue went to triage process again. By: George Joseph (gjoseph) 2018-01-23 08:22:01.403-0600 [~coreyfarrell] I assigned this to you to at least check if it's related to your recent work. Put the issue back in triage if it isn't. By: Corey Farrell (coreyfarrell) 2018-01-23 11:37:10.163-0600 It looks like none of the recent work I've done on stasis (core or res_stasis) has made it to a release yet. Unfortunately the refs.txt only shows a minor leak from ODBC. The line numbers in the latest FRACK matches 15.1.x but are slightly off from 15.2.0. By: George Joseph (gjoseph) 2018-01-23 15:19:52.779-0600 Which version of asterisk were you running when you captured the logs? "core show version" By: Siruja Maharjan (sirujam) 2018-01-23 21:11:54.852-0600 Please find the version {quote} Asterisk 15.1.4 built by root @ localhost on a x86_64 running Linux on 2017-12-19 05:02:40 UTC {quote} By: Friendly Automation (friendly-automation) 2018-07-27 05:39:01.128-0500 Change 9660 merged by Joshua Colp: devicestate: Don't create topic when change isn't cached. [https://gerrit.asterisk.org/9660|https://gerrit.asterisk.org/9660] By: Friendly Automation (friendly-automation) 2018-07-27 05:39:15.666-0500 Change 9659 merged by Joshua Colp: devicestate: Don't create topic when change isn't cached. [https://gerrit.asterisk.org/9659|https://gerrit.asterisk.org/9659] By: Friendly Automation (friendly-automation) 2018-07-27 06:10:40.859-0500 Change 9658 merged by Jenkins2: devicestate: Don't create topic when change isn't cached. [https://gerrit.asterisk.org/9658|https://gerrit.asterisk.org/9658] By: Friendly Automation (friendly-automation) 2018-07-27 06:11:14.060-0500 Change 9661 merged by Joshua Colp: devicestate: Don't create topic when change isn't cached. [https://gerrit.asterisk.org/9661|https://gerrit.asterisk.org/9661] By: Siruja Maharjan (sirujam) 2018-08-09 10:10:10.323-0500 After patch upgrade, the memory utilization is more or less stable. Memory leakage has not been observed till now. Thanks, Siruja By: Asterisk Team (asteriskteam) 2018-08-09 10:10:10.629-0500 This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable. |