Summary:	ASTERISK-28173: Deadlock in chan_sip handling subscribe request during res_parking reload
Reporter:	Giuseppe Sucameli (gsucameli)	Labels:	pjsip
Date Opened:	2018-11-19 17:26:16.000-0600	Date Closed:	2019-02-05 11:38:44.000-0600
Priority:	Major	Regression?	No
Status:	Closed/Complete	Components:	Channels/chan_sip/Subscriptions Resources/res_parking
Versions:	13.1.1 13.18.3	Frequency of Occurrence	Occasional
Related Issues:
Environment:		Attachments:
Description:	There's a deadlock if you get a subscribe request for a parking lot during res_parking module reload. The subscribe request first locks the {{sip_pvt}} in {{handle_request_do()}} and then calls {{ast_rdlock_contexts()}} in {{ast_hint_extension()}}. The res_parking module reload instead first calls {{ast_wrlock_contexts()}} in {{ast_context_destroy()}} and then the {{sip_pvt_lock(p)}} in {{extensionstate_update()}}. The issue has been replicated on both *Asterisk 13.1-cert8* and *13.18-cert3*. Here's the backtrace of the issue replicated on Asterisk 13.1-cert8. {code} (gdb) info threads  Id   Target Id         Frame [...]  24   Thread 0xb1787b40 (LWP 24000) 0xb71ad27d in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread-2.19.so [...]  8    Thread 0xb464bb40 (LWP 23993) 0xb71ad27d in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread-2.19.so [...] {code} Thread 8 is running the *res_parking module reload*: {code} (gdb) t 8 [Switching to thread 8 (Thread 0xb464bb40 (LWP 23993))] #0  0xb71ad27d in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread-2.19.so (gdb) bt #0  0xb71ad27d in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread-2.19.so #1  0xb71a8fe7 in _L_lock_920 () from /lib/i386-linux-gnu/libpthread-2.19.so #2  0xb71a8e63 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread-2.19.so #3  0xb344a7cf in extensionstate_update (force=<optimized out>, p=<optimized out>, data=<optimized out>, exten=<optimized out>, context=<optimized out>) at chan_sip.c:16742 #4  cb_extensionstate (context=0xac60b844 "parkedcalls_10001199", exten=0xac60b894 "890", info=0xfffffe00, data=0xa2f50374) at chan_sip.c:16838 #5  0x0819a188 in ?? () #6  0x0819a225 in ?? () #7  0x080988d1 in __ao2_ref () #8  0x08197152 in ?? () #9  0x081b00f6 in __ast_context_destroy () #10 0x081b09ad in ast_context_destroy () #11 0xb2a15f6b in parking_lot_cfg_remove_extensions (lot_cfg=0xac6b6084) at res_parking.c:630 #12 0xb2a17099 in remove_all_configured_parking_lot_extensions () at res_parking.c:648 #13 configure_parking_extensions () at res_parking.c:1094 #14 config_parking_preapply () at res_parking.c:1135 #15 0x0811a4bc in aco_process_config () #16 0xb2a15a3d in reload_module () at res_parking.c:1259 #17 0x08164e1f in ast_module_reload () #18 0x0817f007 in ?? () #19 0x081761fd in ?? () #20 0x0817b307 in ?? () #21 0x081ff21b in ?? () #22 0x0820d22e in ?? () #23 0xb71a6d67 in start_thread () from /lib/i386-linux-gnu/libpthread-2.19.so #24 0xb6f9064e in clone () from /lib/i386-linux-gnu/libc-2.19.so {code} Thread 24 is the one handing the *subscribe request*: {code} (gdb) t 24 [Switching to thread 24 (Thread 0xb1787b40 (LWP 24000))] #0  0xb71ad27d in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread-2.19.so (gdb) bt #0  0xb71ad27d in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread-2.19.so #1  0xb71a8fe7 in _L_lock_920 () from /lib/i386-linux-gnu/libpthread-2.19.so #2  0xb71a8e63 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread-2.19.so #3  0x081a727c in ast_get_hint () #4  0xb343c495 in get_destination (p=p@entry=0xa2f50374, oreq=oreq@entry=0x0, cc_recall_core_id=0x0) at chan_sip.c:17739 #5  0xb3492b8c in handle_request_subscribe (p=p@entry=0xa2f50374, req=req@entry=0xb1786c7c, addr=0xb1786bf8, seqno=13246, e=0x989d0376 "sip:890@10.131.253.10:5060") at chan_sip.c:27438 #6  0xb3496c43 in handle_incoming (p=0xa2f50374, req=0xb1786c7c, addr=0xb1786bf8, recount=0xb1786bc8, nounlock=0xb1786bcc) at chan_sip.c:28049 #7  0xb3499709 in handle_request_do (req=0xb1786c7c, addr=0x8331d40, addr@entry=0xb1786bf8) at chan_sip.c:28220 #8  0xb349af1b in sipsock_read (id=0xb1400480, fd=12, events=1, ignore=0x0) at chan_sip.c:28151 #9  0x0815f4b1 in ast_io_wait () #10 0xb3475d98 in do_monitor (data=0x0) at chan_sip.c:28783 #11 0x0820d22e in ?? () #12 0xb71a6d67 in start_thread () from /lib/i386-linux-gnu/libpthread-2.19.so #13 0xb6f9064e in clone () from /lib/i386-linux-gnu/libc-2.19.so {code} I've also written down a patch which fixes the issue moving the call to extension state changed callback (thread 8 frame 6 aka {{destroy_hint()}} in {{main.pbx.c}}) to stasis. I'll provide it for revision ASAP.
Comments:	By: Asterisk Team (asteriskteam) 2018-11-19 17:26:17.154-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Ross Beer (rossbeer) 2018-12-15 07:59:20.614-0600 Could this also happen with chan_pjsip? By: Friendly Automation (friendly-automation) 2019-02-05 11:38:44.910-0600 Change 10959 merged by George Joseph: Fix deadlock handling subscribe req during res_parking reload [https://gerrit.asterisk.org/10959|https://gerrit.asterisk.org/10959] By: Friendly Automation (friendly-automation) 2019-02-05 11:39:02.999-0600 Change 10701 merged by George Joseph: Fix deadlock handling subscribe req during res_parking reload [https://gerrit.asterisk.org/10701|https://gerrit.asterisk.org/10701] By: Friendly Automation (friendly-automation) 2019-02-05 11:39:19.941-0600 Change 10958 merged by George Joseph: Fix deadlock handling subscribe req during res_parking reload [https://gerrit.asterisk.org/10958|https://gerrit.asterisk.org/10958]