[Home]

Summary:ASTERISK-20552: Deadlock when DEBUG_THREADS enabled due to held lock inside of standard locks
Reporter:David Brillert (aragon)Labels:
Date Opened:2012-10-10 20:18:58Date Closed:2012-11-07 11:46:01.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Channels/General
Versions:SVN Frequency of
Occurrence		Occasional
Related
Issues:
Environment:SVN-branch-1.8-r374802Attachments:( 0) thread_apply_all_bt.txt
Description:First a bunch of these queue_frame warnings and then I checked core show locks and attached gdb.
GDB debug info attached.

[2012-10-10 21:09:19] WARNING[22800]: channel.c:1507 __ast_queue_frame: Exceptionally long voice queue length queuing to Local/6147@default-agent-00000006;1
[2012-10-10 21:09:19] WARNING[22800]: channel.c:1507 __ast_queue_frame: Exceptionally long voice queue length queuing to Local/6147@default-agent-00000006;1
[2012-10-10 21:09:19] WARNING[22873]: channel.c:1507 __ast_queue_frame: Exceptionally long voice queue length queuing to Local/6120@default-agent-0000001d;2
6000*CLI> core show locks

=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0x41387940 (pbx_thread           started at [ 5374] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 4845 ast_write chan 0x1fd3c4c0 (1)
Comments:By: Matt Jordan (mjordan) 2012-10-11 11:23:15.990-0500

Something got all hung up on the DEBUG_THREADs lock.  This most likely won't occur if you don't have DEBUG_THREADs enabled.

That being said, I have no idea why the DEBUG_THREADs re-entrancy lock would cause something to block (permanently).  Nothing has really been changed in the locking code recently, so its interesting that you would just now run into this.
By: David Brillert (aragon) 2012-10-11 11:40:38.167-0500

We no longer compile our installations by default with DEBUG_THREADS enabled.  The performance hit is too big.  It would be our preference to do so in 1.8 but this was only possible in 1.4 where we used to compile with DONT_OPTIMIZE and DEBUG_THREADS enabled to provide quicker bug resolution.  1.4 ran pretty well in production systems with those options compiled.  From our end that would explain why we only just ran into this scenario while soaking our lab servers to debug another issue.

By: David Brillert (aragon) 2012-11-07 11:46:01.129-0600

Compiling without debug_threads avoids the deadlock.
Closing as workaround available.