Summary:	ASTERISK-28084: app_queue: QueueMemberStatus Event flooding AMI
Reporter:	Andrej (tekach)	Labels:	patch
Date Opened:	2018-09-28 13:13:53	Date Closed:	2018-10-02 07:58:59
Priority:	Major	Regression?
Status:	Closed/Complete	Components:	Core/ManagerInterface Core/Stasis
Versions:	13.23.1	Frequency of Occurrence	Frequent
Related Issues:
Environment:	Centos 7	Attachments:	( 0) jira_asterisk_28084_v13.patch
Description:	Hi everybody, I’d kindly ask for your help - we have an asterisk 13 installation with approximately 100 queues and 30 agents -- all agents are dynamic queue members in multiple queues. When a call comes into any of the queues and asterisk cycles through available agents, we get an error: [2018-09-28 19:40:51] WARNING[107251]: taskprocessor.c:895 taskprocessor_push: The 'subm:manager_topic-00000006' task processor queue reached 3000 scheduled tasks again. Negative effect is that AMI dependent applications (like our CRM) start lagging. Same thing happens when agents login/logoff. I’ve first noticed this problem with asterisk version 13.19, but it’s also present in 13.20 and 13.21. and 13.23.1. Last working version was therefore 13.18. I can easily reproduce this with test system without any serious load. If this matters, we are using freepbx and dynamic queue members. I strongly suspect that problem is related to huge amount of AMI messages QueueMemberStatus (flooding the AMI? Stasis? I am no expert, not sure about this part). I've tried to filter QueueMemberStatus with eventfilter=!Event: QueueMemberStatus in all defined manager users, but that did not solve the problem, asterisk warning messages (and lag) was still there. Then I've tried to use stasis.conf to decline queue_member_status_type events and this did get rid of warning messages (and lag), but induces memory leak - with every call there is higher memory consumption until asterisk uses all the memory and OOM killer kicks in. Thanks, Andrej
Comments:	By: Asterisk Team (asteriskteam) 2018-09-28 13:13:54.931-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Richard Mudgett (rmudgett) 2018-09-28 14:02:52.090-0500 [^jira_asterisk_28084_v13.patch] - This patch should fix the memory leak when the stasis queue_member_status_type message is declined in stasis.conf. Please test and report back. By: Andrej (tekach) 2018-09-29 01:09:25.984-0500 Wow, that was super fast - thank you very much, Richard. I applied the patch on the test system and tested using sipp, leak seems to be gone. I've now used patch in production environment and will report back. Looks good. Best regards, Andrej By: Andrej (tekach) 2018-09-29 13:27:33.227-0500 I can confirm that this issue is solved. Big thanks for prompt response and well working fix. Best regards, Andrej By: Friendly Automation (friendly-automation) 2018-10-02 07:59:02.651-0500 Change 10338 merged by Joshua Colp: app_queue.c: Fix json ref leak [https://gerrit.asterisk.org/10338|https://gerrit.asterisk.org/10338] By: Friendly Automation (friendly-automation) 2018-10-02 07:59:13.920-0500 Change 10340 merged by Joshua Colp: app_queue.c: Fix json ref leak [https://gerrit.asterisk.org/10340|https://gerrit.asterisk.org/10340] By: Friendly Automation (friendly-automation) 2018-10-02 08:00:18.510-0500 Change 10339 merged by Joshua Colp: app_queue.c: Fix json ref leak [https://gerrit.asterisk.org/10339|https://gerrit.asterisk.org/10339] By: Friendly Automation (friendly-automation) 2018-10-02 08:47:19.495-0500 Change 10337 merged by Joshua Colp: app_queue.c: Fix json ref leak [https://gerrit.asterisk.org/10337|https://gerrit.asterisk.org/10337]