Summary:	ASTERISK-29046: pbx: Deadlock when doing a reload, while simultaneously doing an ExtensionState on a pattern match hint that ends up adding an extension
Reporter:	Ramarajan (pramarajan)	Labels:
Date Opened:	2020-08-27 00:12:35	Date Closed:	2020-08-28 11:37:17
Priority:	Major	Regression?
Status:	Closed/Complete	Components:	PBX/General
Versions:	13.22.0	Frequency of Occurrence	Frequent
Related Issues:
Environment:	FreePBX V15 with asterisk 13.22.0, Running on Virtual machine. 32 GB RAM, 8 Core CPU.Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz	Attachments:	( 0) core-asterisk-running-2020-08-27T00-22-17-0400-brief.txt ( 1) core-asterisk-running-2020-08-27T00-22-17-0400-full.txt ( 2) core-asterisk-running-2020-08-27T00-22-17-0400-locks.txt ( 3) core-asterisk-running-2020-08-27T00-22-17-0400-thread1.txt
Description:	Asterisk hangs when doing the core reload (apply config in FreePBX). This issue occurs very frequently when we do multiple reloads at a time. During the problematic time, asterisk not accepting any calls, registrations, and not even responding to any SIP messages. The only solution to get it up and running again is to kill the asterisk process and start it over again. I have attached the core-dump files for your reference.
Comments:	By: Asterisk Team (asteriskteam) 2020-08-27 00:12:37.226-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: Joshua C. Colp (jcolp) 2020-08-27 04:49:21.435-0500 This deadlock is between two threads due to differing locking order: {noformat} Thread 114 (Thread 0x7f9400df4700 (LWP 2958)): -> Has hints locked (in ast_add_hint), but wants contexts #0  0x00007f9454e8e54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1  0x00007f9454e89eb6 in _L_lock_941 () from /lib64/libpthread.so.0 #2  0x00007f9454e89daf in pthread_mutex_lock () from /lib64/libpthread.so.0 #3  0x000000000053728e in __ast_pthread_mutex_lock () #4  0x000000000057fde6 in ast_rdlock_contexts () #5  0x0000000000570b28 in ast_hint_extension () #6  0x00000000005738e7 in ast_get_hint () #7  0x000000000054fac9 in manager_state_cb () #8  0x00000000005715cc in execute_state_callback () #9  0x0000000000573597 in ast_add_hint () #10 0x000000000057cb8b in add_priority () #11 0x000000000057d338 in ast_add_extension2_lockopt () #12 0x000000000057cbf9 in ast_add_extension2 () #13 0x00007f94504bf02f in pbx_load_config () from /usr/lib64/asterisk/modules/pbx_config.so #14 0x00007f94504bfed6 in pbx_load_module () from /usr/lib64/asterisk/modules/pbx_config.so #15 0x00007f94504c00ce in reload () from /usr/lib64/asterisk/modules/pbx_config.so #16 0x0000000000534ea4 in ast_module_reload () #17 0x00000000004ce4e0 in handle_core_reload () #18 0x00000000004d6703 in ast_cli_command_full () #19 0x00000000004d6846 in ast_cli_command_multiple_full () #20 0x0000000000453ca7 in netconsole () #21 0x00000000005fd81c in dummy_start () #22 0x00007f9454e87ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007f94541638dd in clone () from /lib64/libc.so.6 Thread 112 (Thread 0x7f940173b700 (LWP 923)): -> Has contexts read locked (in ast_add_extension), but wants hints #0  0x00007f9454e8e54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1  0x00007f9454e89eb6 in _L_lock_941 () from /lib64/libpthread.so.0 #2  0x00007f9454e89daf in pthread_mutex_lock () from /lib64/libpthread.so.0 #3  0x000000000053728e in __ast_pthread_mutex_lock () #4  0x000000000045b293 in __ao2_lock () #5  0x000000000057328c in ast_add_hint () #6  0x000000000057d625 in ast_add_extension2_lockopt () #7  0x000000000057cbf9 in ast_add_extension2 () #8  0x000000000057bf15 in ast_add_extension () #9  0x000000000057107e in internal_extension_state_extended () #10 0x00000000005710ee in ast_extension_state () #11 0x000000000054c081 in action_extensionstate () #12 0x000000000054df61 in process_message () #13 0x000000000054e889 in do_message () #14 0x000000000054eccb in session_do () #15 0x00000000005ea1e5 in handle_tcptls_connection () #16 0x00000000005fd81c in dummy_start () #17 0x00007f9454e87ea5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007f94541638dd in clone () from /lib64/libc.so.6 {noformat} By: Friendly Automation (friendly-automation) 2020-08-28 11:37:18.392-0500 Change 14847 merged by Friendly Automation: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14847|https://gerrit.asterisk.org/c/asterisk/+/14847] By: Friendly Automation (friendly-automation) 2020-08-28 12:13:19.065-0500 Change 14815 merged by Friendly Automation: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14815|https://gerrit.asterisk.org/c/asterisk/+/14815] By: Friendly Automation (friendly-automation) 2020-08-28 12:37:26.323-0500 Change 14833 merged by George Joseph: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14833|https://gerrit.asterisk.org/c/asterisk/+/14833] By: Friendly Automation (friendly-automation) 2020-08-28 12:37:50.788-0500 Change 14846 merged by George Joseph: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14846|https://gerrit.asterisk.org/c/asterisk/+/14846] By: Friendly Automation (friendly-automation) 2020-08-28 12:38:02.959-0500 Change 14834 merged by George Joseph: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14834|https://gerrit.asterisk.org/c/asterisk/+/14834] By: Friendly Automation (friendly-automation) 2020-08-28 12:38:15.171-0500 Change 14848 merged by George Joseph: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14848|https://gerrit.asterisk.org/c/asterisk/+/14848] By: Friendly Automation (friendly-automation) 2020-08-28 12:38:31.596-0500 Change 14836 merged by George Joseph: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14836|https://gerrit.asterisk.org/c/asterisk/+/14836] By: Friendly Automation (friendly-automation) 2020-08-28 12:38:46.284-0500 Change 14835 merged by George Joseph: pbx: Fix hints deadlock between reload and ExtensionState. [https://gerrit.asterisk.org/c/asterisk/+/14835|https://gerrit.asterisk.org/c/asterisk/+/14835]