| Summary: | ASTERISK-16713: potential channel/queue deadlock when setqueuevar enabled | ||
| Reporter: | Ben Winslow (rain) | Labels: | |
| Date Opened: | 2010-09-22 15:51:29 | Date Closed: | 2010-11-19 15:42:11.000-0600 |
| Priority: | Critical | Regression? | No |
| Status: | Closed/Complete | Components: | Applications/app_queue |
| Versions: | Frequency of Occurrence | ||
| Related Issues: | |||
| Environment: | Attachments: | ||
| Description: | When setqueuevar is enabled in app_queue, the potential exists for one thread to lock a call_queue structure with a channel lock held while another thread locks the channel with the call_queue lock held. The resulting deadlock will end up crippling Asterisk since quite a few parts of Asterisk will traverse the channel list. At several points in app_queue (in my case, end_bridge_callback), a queue will be locked before calling set_queue_variables (which pulls data directly out of the queue structure); this is fine, but when setqueuevar is actually enabled, the call to pbx_builtin_setvar_multiple will lock the channel. Meanwhile, another thread may be calling ast_do_masquerade() to clone the same channel, resulting in a lock on the same channel (clonechan.) Later in the masquerade, the datastore fixup functions are called, and queue_transfer_fixup's call to update_queue() will result in an attempt to lock the related queue (or all queues if shared_lastcall is enabled.) This deadlock occurred for me on 1.6.2.6 but appears to present in 1.6.2.13 as well. In my case, do_masquerade() got the channel lock while end_bridge_callback() got the queue lock. ****** ADDITIONAL INFORMATION ****** Queue config: setqueuevar = yes shared_lastcall = yes All agents in the queue were Local channels. Unfortunately, I lost the core I forced from this deadlock, so I can't provide backtraces for the two deadlocked threads. I can tell you that they were (roughly)... Thread A: ... -> ast_bridge_call -> app_queue.c end_bridge_callback [queue lock happens here] -> set_queue_variables -> pbx_builtin_setvar_multiple -> pbx_builtin_setvar_helper -> ast_channel_lock Thread B: ... -> ast_do_masquerade [channel lock happens here] -> queue_transfer_fixup -> update_queue -> ao2_lock | ||
| Comments: | By: Digium Subversion (svnbot) 2010-11-19 15:40:22.000-0600 Repository: asterisk Revision: 295670 U branches/1.8/apps/app_queue.c ------------------------------------------------------------------------ r295670 | bbryant | 2010-11-19 15:40:21 -0600 (Fri, 19 Nov 2010) | 8 lines Patch for deadlock from ordering issue between channel/queue locks in app_queue (set_queue_variables). (closes issue ASTERISK-16713) Reported by: rain Review: https://reviewboard.asterisk.org/r/1018/ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=295670 By: Digium Subversion (svnbot) 2010-11-19 15:42:11.000-0600 Repository: asterisk Revision: 295671 _U trunk/ U trunk/apps/app_queue.c ------------------------------------------------------------------------ r295671 | bbryant | 2010-11-19 15:42:11 -0600 (Fri, 19 Nov 2010) | 15 lines Merged revisions 295670 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.8 ........ r295670 | bbryant | 2010-11-19 16:40:21 -0500 (Fri, 19 Nov 2010) | 8 lines Patch for deadlock from ordering issue between channel/queue locks in app_queue (set_queue_variables). (closes issue ASTERISK-16713) Reported by: rain Review: https://reviewboard.asterisk.org/r/1018/ ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=295671 | ||