Summary:	ASTERISK-25844: app_queue: Ghost channels in "core show channels" output
Reporter:	Etienne Lessard (hexanol)	Labels:
Date Opened:	2016-03-14 15:10:21	Date Closed:	2020-05-05 10:10:49
Priority:	Minor	Regression?
Status:	Closed/Complete	Components:	Applications/app_queue Channels/chan_local Core/Channels
Versions:	13.7.2	Frequency of Occurrence	Constant
Related Issues:	causes ASTERISK-29020 Regression: Fix for app_queue: track masquerades in app_queue to avoid leaked stasis subscriptions broke queue log is related to ASTERISK-28829 app_queue: leaking stasis subscription when Redirecting call
Environment:	Debian 8.3 amd64	Attachments:
Description:	Given I have two 2 SIP users, Alice (SIP/alice) and Bob (SIP/bob) Given I have the following queues.conf: {noformat} [sales] strategy = ringall member => Local/52@default [wait] stategy = ringall {noformat} Given I have the following extensions.conf {noformat} [default] exten = 50,1,NoOp() same  =   n,Queue(sales) same  =   n,Hangup() exten = 51,1,NoOp() same  =   n,Queue(wait) same  =   n,Hangup() exten = 52,1,NoOp() same  =   n,Dial(SIP/bob) same  =   n,Hangup() exten = 53,1,NoOp() same  =   n,Bridge(SIP/alice-00000000) same  =   n,Hangup() {noformat} When Alice calls the 50 extension And Bob answers Then there is 1 temporary ghost channel "Local/52@default-00000000;1" that can be seen in the "core show channels concise" output When Alice's channel is redirected to the 51@default extension via the AMI "Redirect" command When a "channel originate SIP/bob extension 53@default" is done And Bob answers and is bridged with Alice's channel And Alice's channel is hanged up via the AMI "Hangup" command Then there are 2 permanent ghost channels in the output of core show channels: {noformat} el*CLI> core show channels Channel              Location             State   Application(Data)             Local/52@default-000 50@default:1         Up      AppQueue((Outgoing Line))     Surrogate/SIP/69jqad 51@default:1         Down    (None)                         0 active channels 0 active calls 4 calls processed {noformat} So this scenario is a bit complex, I've tried finding a simpler one but was not able to. With this scenario, I've been able to produce ghost channels systematically. This impact us because we have an external application (a switchboard application) that use Asterisk in this kind of way. The only impact I have found is that the output of "core show channels" becomes less and less useful (for administration purpose) as ghost channels adds up. When we try to hangup one of these ghost channels via "channel request hangup", it says "<channel> is not a known channel". It seems like these channels only exists in the stasis cache and not in the ao2_container of channels.
Comments:	By: Asterisk Team (asteriskteam) 2016-03-14 15:10:21.797-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Richard Mudgett (rmudgett) 2016-03-14 17:35:29.527-0500 I've seen a similar situation that had a temporary ghost channel that I'll couch using your configuration: # Alice calls the 50 exten # Bob answers # When the local channels between Alice and Bob optimize out the Local/52@default;1 channel still shows up in the "core show channels" output. # If the call is now hung up, the "ghost" channel also goes away. A ref leak of a stasis channel snapshot will cause the kind of permanent ghost channel you are seeing. FYI: I do hope your {{member => Local/52@default}} line in your actual {{queues.conf}} file specifies the state interface of Bob. {{syntax: member => interface,\[,penalty]\[,membername]\[,state_interface]\[,ringinuse]}} By: Robert Mordec (veilen) 2016-11-24 04:37:48.738-0600 Thanks to Reference Count Debugging wiki (https://wiki.asterisk.org/wiki/display/AST/Reference+Count+Debugging) I found the probable culprit. Info from git: {quote} SHA-1: 6409e7b11a2310196a9978b30a6b79e2760be592 * app_queue: Crash when transferring ASTERISK-25185 #close Reported by: Etienne Lessard (...) This patch adds a reference to both the member and caller channels that extends to the lifetime of the queue'd call, thus making sure the channels will always exist when retrieving the latest snapshots. Change-Id: Ic397fa68fb4ff35fbc378e745da9246a7b552128 {quote} {code:title=app_queue.c|borderStyle=solid} static struct queue_stasis_data *queue_stasis_data_alloc(struct queue_ent *qe, struct ast_channel *peer, struct member *mem, time_t holdstart, time_t starttime, int callcompletedinsl) { (...) /* * During transfers it's possible for both the member and/or caller * channel(s) to not be available. Adding a reference here ensures * that the channels remain until app_queue is completely done with * them. */ queue_data->member_channel = ao2_bump(peer); queue_data->caller_channel = ao2_bump(qe->chan); {code} The added reference holds (optimized away?) local channel which will be destroyed when the call ends. At this point I don't have a good idea how to fix this without reintroducing ASTERISK-25185. Maybe someone else will have more luck using this additional info. By: Friendly Automation (friendly-automation) 2020-05-05 10:10:50.276-0500 Change 14321 merged by Friendly Automation: app_queue: track masquerades in app_queue to avoid leaked stasis subscriptions [https://gerrit.asterisk.org/c/asterisk/+/14321|https://gerrit.asterisk.org/c/asterisk/+/14321] By: Friendly Automation (friendly-automation) 2020-05-05 10:10:58.985-0500 Change 14381 merged by Friendly Automation: app_queue: track masquerades in app_queue to avoid leaked stasis subscriptions [https://gerrit.asterisk.org/c/asterisk/+/14381|https://gerrit.asterisk.org/c/asterisk/+/14381] By: Friendly Automation (friendly-automation) 2020-05-05 10:19:31.742-0500 Change 14382 merged by Friendly Automation: app_queue: track masquerades in app_queue to avoid leaked stasis subscriptions [https://gerrit.asterisk.org/c/asterisk/+/14382|https://gerrit.asterisk.org/c/asterisk/+/14382] By: Friendly Automation (friendly-automation) 2020-05-06 04:11:59.956-0500 Change 14383 merged by Joshua Colp: app_queue: track masquerades in app_queue to avoid leaked stasis subscriptions [https://gerrit.asterisk.org/c/asterisk/+/14383|https://gerrit.asterisk.org/c/asterisk/+/14383]