ASTERISK-30157: chan_iax2: Deadlock with device state and channel locking

[Home]

Summary: ASTERISK-30157: chan_iax2: Deadlock with device state and channel locking

Reporter: N A (InterLinked) Labels:

Date Opened: 2022-07-26 06:19:43 Date Closed: 2022-10-03 12:18:04

Priority: Minor Regression?

Status: Closed/Complete Components: Channels/chan_iax2 Core/Channels

Versions: 18.9.0 Frequency of
Occurrence Frequent

Related
Issues:
duplicates ASTERISK-29821 Deadlock in bridge_channel_internal_join() on local channels.

Environment: Attachments: ( 0) 6bzz1npcct.txt

Description: This has happened several times now that I've begun to pick out a pattern of this deadlock consistently happening in the same place, so here we are:

The deadlock is always on a channel (which then blocks the entire channels container), which basically grinds Asterisk to a halt as existing calls may complete but no new channels can be created.

core show locks:
{noformat}
[2022-07-26 10:45:51] ERROR[7806]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f20923beeff]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:299 __ast_pthread_mutex_lock: channel.c line 922 (__ast_channel_alloc_ap): Deadlock? waited 16665 sec for mutex 'channels'?
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_bt_get_addresses+0x1d) [0x5603f1ac6716]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xb5) [0x5603f1b81517]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ao2_lock+0x10b) [0x5603f1ab9091]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0xb6c6e) [0x5603f1af8c6e]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_channel_alloc+0xde) [0x5603f1af8f56]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_unreal_new_channels+0x227) [0x5603f1b3d14c]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0xf7195) [0x5603f1b39195]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0xc8cfa) [0x5603f1b0acfa]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_request+0x51) [0x5603f1b0b3ca]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x103352) [0x5603f1b45352]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_dial_prerun+0x77) [0x5603f1b45845]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x17395e) [0x5603f1bb595e]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_pbx_outgoing_app_predial+0xb9) [0x5603f1bb64a6]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_pbx_outgoing_app+0x5d) [0x5603f1bb63e7]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/lib/asterisk/modules/pbx_spool.so(+0x4f24) [0x7f2083a3df24]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x215820) [0x5603f1c57820]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f209292afa3]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f20923beeff]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:307 __ast_pthread_mutex_lock: astobj2_container.c line 310 (internal_ao2_traverse): 'channels' was locked here.
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_bt_get_addresses+0x1d) [0x5603f1ac6716]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xb5) [0x5603f1b81517]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ao2_lock+0x10b) [0x5603f1ab9091]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x79982) [0x5603f1abb982]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ao2_callback_data+0x4d) [0x5603f1abbd37]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_channel_callback+0x54) [0x5603f1afa0c8]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_channel_get_by_name_prefix+0x46) [0x5603f1afa7f9]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_parse_device_state+0x4e) [0x5603f1b42caa]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x10102b) [0x5603f1b4302b]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x1014d1) [0x5603f1b434d1]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x1018aa) [0x5603f1b438aa]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x215820) [0x5603f1c57820]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f209292afa3]
[2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f20923beeff]
voip*CLI> core show locks

=======================================================================
=== 18.9.0
=== Currently Held Locks
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0x7f2083d11700 LWP:31458 (do_devstate_changes started at [ 621] devicestate.c ast_device_state_engine_init())
=== ---> Lock #0 (astobj2_container.c): MUTEX 310 internal_ao2_traverse self 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1abb982] asterisk astobj2_container.c:315 internal_ao2_traverse()
[0x5603f1abbd37] asterisk astobj2_container.c:421 __ao2_callback_data()
[0x5603f1afa0c8] asterisk channel.c:1275 ast_channel_callback()
[0x5603f1afa7f9] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
[0x5603f1b42caa] asterisk devicestate.c:295 ast_parse_device_state()
[0x5603f1b4302b] asterisk devicestate.c:376 _ast_device_state()
[0x5603f1b434d1] asterisk devicestate.c:464 do_state_change()
[0x5603f1b438aa] asterisk devicestate.c:540 do_devstate_changes()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 1290 ast_channel_by_name_cb chan 0x7f206c4c0060 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1afa189] asterisk channel.c:1291 ast_channel_by_name_cb()
[0x5603f1abba0a] asterisk astobj2_container.c:326 internal_ao2_traverse()
[0x5603f1abbd37] asterisk astobj2_container.c:421 __ao2_callback_data()
[0x5603f1afa0c8] asterisk channel.c:1275 ast_channel_callback()
[0x5603f1afa7f9] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
[0x5603f1b42caa] asterisk devicestate.c:295 ast_parse_device_state()
[0x5603f1b4302b] asterisk devicestate.c:376 _ast_device_state()
[0x5603f1b434d1] asterisk devicestate.c:464 do_state_change()
[0x5603f1b438aa] asterisk devicestate.c:540 do_devstate_changes()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067d04700 LWP:31499 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f62ba660 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067c87700 LWP:31500 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070ae46e0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067c0a700 LWP:31501 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f20614274c0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067b8d700 LWP:31502 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2078e8a4a0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067b10700 LWP:31503 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2076c18840 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067a93700 LWP:31504 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207d5d6f10 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067a16700 LWP:31505 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2088b673f0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067999700 LWP:31506 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2085c8de80 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206791c700 LWP:31507 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f208cf92ef0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206789f700 LWP:31508 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f748d6d0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f20697c4700 LWP:443 (worker_start started at [ 1111] threadpool.c worker_thread_start())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207c82fce0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
[0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
[0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
[0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
[0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
[0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
[0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
[0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
[0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
[0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2067ffa700 LWP:3871 (worker_start started at [ 1111] threadpool.c worker_thread_start())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f206d746c80 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
[0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
[0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
[0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
[0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
[0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
[0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
[0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
[0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
[0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f209008c700 LWP:21787 (worker_start started at [ 1111] threadpool.c worker_thread_start())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207e9f3d30 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
[0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
[0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
[0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
[0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
[0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
[0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
[0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
[0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
[0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f20650b1700 LWP:2130 (default_tps_processing_function started at [ 226] taskprocessor.c default_listener_start())
=== ---> Waiting for Lock #0 (astobj2_container.c): MUTEX 308 internal_ao2_traverse self 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2065228700 LWP:6593 (pbx_thread started at [ 4736] pbx.c ast_pbx_start())
=== ---> Lock #0 (bridge_channel.c): MUTEX 2877 bridge_channel_internal_join bridge_channel->bridge 0x7f206d6426c0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1ae3793] asterisk bridge.h:484 _ast_bridge_lock()
[0x5603f1aeb2c8] asterisk bridge_channel.c:2879 bridge_channel_internal_join()
[0x5603f1acca08] asterisk bridge.c:1725 ast_bridge_join()
[0x5603f1c9cf55] asterisk features.c:684 ast_bridge_call_with_flags()
[0x5603f1c9d113] asterisk features.c:715 ast_bridge_call()
[0x7f20812bd146] app_dial.so app_dial.c:3413 dial_exec_full()
[0x7f20812bda11] app_dial.so app_dial.c:3482 dial_exec()
[0x5603f1bbbc19] asterisk pbx_app.c:492 pbx_exec()
[0x5603f1ba42f6] asterisk pbx.c:2948 pbx_extension_helper()
[0x5603f1ba868f] asterisk pbx.c:4210 ast_spawn_extension()
[0x5603f1ba949f] asterisk pbx.c:4386 __ast_pbx_run()
[0x5603f1baae48] asterisk pbx.c:4711 pbx_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (bridge_channel.c): MUTEX 2879 bridge_channel_internal_join bridge_channel->chan 0x7f206d539fc0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1aeb2fb] asterisk bridge_channel.c:2881 bridge_channel_internal_join()
[0x5603f1acca08] asterisk bridge.c:1725 ast_bridge_join()
[0x5603f1c9cf55] asterisk features.c:684 ast_bridge_call_with_flags()
[0x5603f1c9d113] asterisk features.c:715 ast_bridge_call()
[0x7f20812bd146] app_dial.so app_dial.c:3413 dial_exec_full()
[0x7f20812bda11] app_dial.so app_dial.c:3482 dial_exec()
[0x5603f1bbbc19] asterisk pbx_app.c:492 pbx_exec()
[0x5603f1ba42f6] asterisk pbx.c:2948 pbx_extension_helper()
[0x5603f1ba868f] asterisk pbx.c:4210 ast_spawn_extension()
[0x5603f1ba949f] asterisk pbx.c:4386 __ast_pbx_run()
[0x5603f1baae48] asterisk pbx.c:4711 pbx_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (bridge_channel.c): MUTEX 2885 bridge_channel_internal_join peer 0x7f206c4c0060 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1afa189] asterisk channel.c:1291 ast_channel_by_name_cb()
[0x5603f1abba0a] asterisk astobj2_container.c:326 internal_ao2_traverse()
[0x5603f1abbd37] asterisk astobj2_container.c:421 __ao2_callback_data()
[0x5603f1afa0c8] asterisk channel.c:1275 ast_channel_callback()
[0x5603f1afa7f9] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
[0x5603f1b42caa] asterisk devicestate.c:295 ast_parse_device_state()
[0x5603f1b4302b] asterisk devicestate.c:376 _ast_device_state()
[0x5603f1b434d1] asterisk devicestate.c:464 do_state_change()
[0x5603f1b438aa] asterisk devicestate.c:540 do_devstate_changes()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206541c700 LWP:6598 (bridge_channel_ind_thread started at [ 1892] bridge.c bridge_impart_internal())
=== ---> Waiting for Lock #0 (bridge_channel.c): MUTEX 235 ast_bridge_channel_lock_bridge bridge 0x7f206d6426c0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1ae3793] asterisk bridge.h:484 _ast_bridge_lock()
[0x5603f1ae3c77] asterisk bridge_channel.c:236 ast_bridge_channel_lock_bridge()
[0x5603f1ada376] asterisk bridge_basic.c:296 basic_hangup_hook()
[0x5603f1ae50b3] asterisk bridge_channel.c:613 ast_bridge_channel_kick()
[0x5603f1aea8b9] asterisk bridge_channel.c:2595 bridge_handle_trip()
[0x5603f1aeaf78] asterisk bridge_channel.c:2803 bridge_channel_wait()
[0x5603f1aeb9de] asterisk bridge_channel.c:2975 bridge_channel_internal_join()
[0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2877 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f20669fa700 LWP:6599 (bridge_channel_ind_thread started at [ 1892] bridge.c bridge_impart_internal())
=== ---> Lock #0 (bridge_channel.c): MUTEX 2877 bridge_channel_internal_join bridge_channel->bridge 0x7f208d2dcef0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1ae3793] asterisk bridge.h:484 _ast_bridge_lock()
[0x5603f1aeb2c8] asterisk bridge_channel.c:2879 bridge_channel_internal_join()
[0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (bridge_channel.c): MUTEX 2879 bridge_channel_internal_join bridge_channel->chan 0x7f206c4c0060 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1aeb2fb] asterisk bridge_channel.c:2881 bridge_channel_internal_join()
[0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (bridge_channel.c): MUTEX 2885 bridge_channel_internal_join peer 0x7f206d539fc0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1aeb349] asterisk bridge_channel.c:2886 bridge_channel_internal_join()
[0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2066082700 LWP:6632 (worker_start started at [ 1111] threadpool.c worker_thread_start())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207deb3e30 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
[0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
[0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
[0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
[0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
[0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
[0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
[0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
[0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
[0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2069747700 LWP:6633 (worker_start started at [ 1111] threadpool.c worker_thread_start())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2088ca6da0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
[0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
[0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
[0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
[0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
[0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
[0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
[0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
[0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
[0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
[0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f20643ce700 LWP:7806 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f207f2d2898 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb61c4] asterisk pbx.c:8444 ast_pbx_outgoing_exten_predial()
[0x5603f1bb6129] asterisk pbx.c:8423 ast_pbx_outgoing_exten()
[0x7f2083a3e052] pbx_spool.so pbx_spool.c:443 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207d0d3590 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb61c4] asterisk pbx.c:8444 ast_pbx_outgoing_exten_predial()
[0x5603f1bb6129] asterisk pbx.c:8423 ast_pbx_outgoing_exten()
[0x7f2083a3e052] pbx_spool.so pbx_spool.c:443 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2065516700 LWP:10658 (iax2_process_thread started at [ 1626] chan_iax2.c find_idle_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f208d1bf620 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206670c700 LWP:10667 (iax2_process_thread started at [ 1626] chan_iax2.c find_idle_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070ac7120 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206668f700 LWP:10689 (iax2_process_thread started at [ 1626] chan_iax2.c find_idle_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2078eb8cf0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f20680f4700 LWP:10690 (iax2_process_thread started at [ 1626] chan_iax2.c find_idle_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f205ca4c040 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206444b700 LWP:20245 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f205647e9f8 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f20567c5740 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2064fb7700 LWP:20580 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x5603f73717a8 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f6d92510 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2066883700 LWP:20849 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2078e3ef88 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2078dfb650 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206641e700 LWP:21042 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f208886ffa8 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2088b7cde0 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2066324700 LWP:21379 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2070621b68 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070bfce20 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2066518700 LWP:21625 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f205735b548 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f20574cfa00 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2066789700 LWP:21839 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2053cd3658 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2048f8ac00 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f206622a700 LWP:22010 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2070b2b8c8 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070bf6880 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2065f88700 LWP:22282 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f207cf88398 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207cc72a50 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2064ebd700 LWP:22553 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x5603f7a5bf08 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f6dbfa10 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
[0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
[0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
[0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
[0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
[0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
[0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
[0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
[0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
[0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
[0x5603f1c57820] asterisk utils.c:1572 dummy_start()
[0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f20923beeff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=======================================================================
{noformat}

Comments: By: Asterisk Team (asteriskteam) 2022-07-26 06:19:46.571-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/].
By: N A (InterLinked) 2022-07-26 06:21:22.446-0500

Backtrace from coredump of --RUNNING is attached.

(Asterisk doesn't crash, just stays deadlock until forcibly restarted)
By: Joshua C. Colp (jcolp) 2022-07-26 07:17:27.761-0500

This looks like an interaction between bridging, Local channels, IAX2, and devicestate. I question whether the problem would occur if IAX2 was not used.
By: N A (InterLinked) 2022-07-26 08:57:13.504-0500

I think the only reason IAX2 is in here is after a point trunk calls started arriving and then failed because new channel set up failed, but I believe this has happened without IAX2 channels involved.

Also, this backtrace must be several hours after the deadlock started, so it includes all the channel setup that attempted to occur and failed during that time.

Right now I don't have a good automated process in place for automatically detecting deadlocks. For example, an AMI event when a deadlock starts that could go out to a process that automatically gets a core dump immediately, which I could then export. Would that make sense, or would there be a better way to go about this?
By: Joshua C. Colp (jcolp) 2022-07-26 09:02:12.157-0500

I'd want to see a backtrace without IAX2 involved if you believe that's the case. One of the threads deadlocked and holding a lock is as a result of IAX2 device state, when going through the channels since it doesn't natively support it.
By: Joshua C. Colp (jcolp) 2022-07-26 09:05:24.400-0500

I don't have any thoughts regarding automating deadlock collection like that. When a deadlock occurs it may or may not be possible. Ultimately depends, safest thing is to collect before restarting Asterisk.
By: N A (InterLinked) 2022-07-26 09:13:57.198-0500

Collect before restart is what I'm doing now, but often a significant time after the deadlock has started, particularly since this particular deadlock doesn't impact existing channels.

In that case, would you object to adding an AMI action when compiled with DETECT_DEADLOCKS that would emit "DeadlockStarted" AMI events?

It would be possible since an event is emitted every 5 seconds, and we can simply emit the event in the code when a deadlock has just started.

That would make it much easier to stay on top of these and get clean backtraces.
By: Joshua C. Colp (jcolp) 2022-07-26 09:23:01.614-0500

Would I? I don't think so. Would others? I can't speak for them. And to be specific it would make it easier for you and potentially a few others to get clean backtraces. It would not be commonly used.
By: Joshua C. Colp (jcolp) 2022-07-27 17:38:46.396-0500

A second backtrace without chan_iax2 involvement is needed for this to see if it is indeed an interaction there as I suspect or not.
By: N A (InterLinked) 2022-08-01 20:18:15.157-0500

All right, it just happened again.

IAX2 is still present in this one, because at the time of the deadlock, I was actually on a call with somebody so inherently IAX2 would be in the picture due to that. The deadlock started about 11 minutes before I noticed it - all of a sudden a reorder tone dropped into the call (the call itself still went on fine, just with the reorder on top), and I opened the CLI to find a deadlock in progress. Was certainly interesting, to say the least.

core show locks seems similar to the last one... :

{noformat}
voip*CLI> core show locks

=======================================================================
=== 18.9.0
=== Currently Held Locks
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0x7f4573e88700 LWP:18003 (do_devstate_changes started at [ 621] devicestate.c ast_device_state_engine_init())
=== ---> Lock #0 (astobj2_container.c): MUTEX 310 internal_ao2_traverse self 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b151c5] asterisk astobj2_container.c:315 internal_ao2_traverse()
[0x557c73b1557a] asterisk astobj2_container.c:421 __ao2_callback_data()
[0x557c73b53934] asterisk channel.c:1275 ast_channel_callback()
[0x557c73b54065] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
[0x557c73b9c516] asterisk devicestate.c:295 ast_parse_device_state()
[0x557c73b9c86f] asterisk devicestate.c:371 _ast_device_state()
[0x557c73b9cd3d] asterisk devicestate.c:464 do_state_change()
[0x557c73b9d116] asterisk devicestate.c:540 do_devstate_changes()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 1290 ast_channel_by_name_cb chan 0x7f458167ab20 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b539f5] asterisk channel.c:1291 ast_channel_by_name_cb()
[0x557c73b1524d] asterisk astobj2_container.c:326 internal_ao2_traverse()
[0x557c73b1557a] asterisk astobj2_container.c:421 __ao2_callback_data()
[0x557c73b53934] asterisk channel.c:1275 ast_channel_callback()
[0x557c73b54065] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
[0x557c73b9c516] asterisk devicestate.c:295 ast_parse_device_state()
[0x557c73b9c86f] asterisk devicestate.c:371 _ast_device_state()
[0x557c73b9cd3d] asterisk devicestate.c:464 do_state_change()
[0x557c73b9d116] asterisk devicestate.c:540 do_devstate_changes()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568dc7700 LWP:18044 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f458cfe8be0 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568d4a700 LWP:18045 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x557c7b82aca0 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568ccd700 LWP:18046 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f456c9fb180 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568c50700 LWP:18047 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f457b747880 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568bd3700 LWP:18048 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f4576a44e00 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568b56700 LWP:18049 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f4581723b30 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568ad9700 LWP:18050 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f457f6d6d80 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568a5c700 LWP:18051 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f4588634470 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f45689df700 LWP:18052 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f4585663a00 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714fbbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4568962700 LWP:18053 (iax2_process_thread started at [12720] chan_iax2.c start_network_thread())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f458d0ae590 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4566530700 LWP:875 (worker_start started at [ 1111] threadpool.c worker_thread_start())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f457dd7b5f0 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f4570426016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
[0x7f4570432e0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
[0x7f45704e14ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
[0x7f45704de446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
[0x7f45704decec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
[0x7f45704dfee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
[0x7f45932ef58f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
[0x7f45705a8e23] res_pjsip.so pjsip_distributor.c:956 distribute()
[0x557c73c9b8a2] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x557c73ca56c0] asterisk threadpool.c:1354 execute_tasks()
[0x557c73c9b8a2] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x557c73ca3226] asterisk threadpool.c:367 threadpool_execute()
[0x557c73ca4e68] asterisk threadpool.c:1137 worker_active()
[0x557c73ca4bd2] asterisk threadpool.c:1057 worker_start()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4565d60700 LWP:23497 (pbx_thread started at [ 4736] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (bridge_channel.c): MUTEX 235 ast_bridge_channel_lock_bridge bridge 0x7f456c233000 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x7f45718c6b07] bridge_softmix.so bridge.h:484 _ast_bridge_lock()
[0x7f45718cc748] bridge_softmix.so bridge_softmix.c:1976 softmix_mixing_loop()
[0x7f45718ccada] bridge_softmix.so bridge_softmix.c:2052 softmix_mixing_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2877 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f45667a1700 LWP:23498 (softmix_mixing_thread started at [ 2116] bridge_softmix.c softmix_bridge_create())
=== ---> Waiting for Lock #0 (bridge_softmix.c): MUTEX 1973 softmix_mixing_loop bridge 0x7f456c233000 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x7f45718c6b07] bridge_softmix.so bridge.h:484 _ast_bridge_lock()
[0x7f45718cc748] bridge_softmix.so bridge_softmix.c:1976 softmix_mixing_loop()
[0x7f45718ccada] bridge_softmix.so bridge_softmix.c:2052 softmix_mixing_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2877 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4564aa1700 LWP:31390 (worker_start started at [ 1111] threadpool.c worker_thread_start())
=== ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x557c7b56a6c0 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f4570426016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
[0x7f4570432e0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
[0x7f45704e14ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
[0x7f45704de446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
[0x7f45704decec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
[0x7f45704dfee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
[0x7f45932ef58f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
[0x7f45705a8e23] res_pjsip.so pjsip_distributor.c:956 distribute()
[0x557c73c9b8a2] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x557c73ca56c0] asterisk threadpool.c:1354 execute_tasks()
[0x557c73c9b8a2] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
[0x557c73ca3226] asterisk threadpool.c:367 threadpool_execute()
[0x557c73ca4e68] asterisk threadpool.c:1137 worker_active()
[0x557c73ca4bd2] asterisk threadpool.c:1057 worker_start()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f4590261700 LWP:4731 (pbx_thread started at [ 4736] pbx.c ast_pbx_start())
=== ---> Lock #0 (bridge_channel.c): MUTEX 2877 bridge_channel_internal_join bridge_channel->bridge 0x7f456c233000 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b3cfd6] asterisk bridge.h:484 _ast_bridge_lock()
[0x557c73b44b0b] asterisk bridge_channel.c:2879 bridge_channel_internal_join()
[0x557c73b2624b] asterisk bridge.c:1725 ast_bridge_join()
[0x7f4570781571] app_confbridge.so app_confbridge.c:2910 confbridge_exec()
[0x557c73c1553b] asterisk pbx_app.c:492 pbx_exec()
[0x557c73bfdc18] asterisk pbx.c:2948 pbx_extension_helper()
[0x557c73c01fb1] asterisk pbx.c:4210 ast_spawn_extension()
[0x557c73c02dc1] asterisk pbx.c:4386 __ast_pbx_run()
[0x557c73c0476a] asterisk pbx.c:4711 pbx_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (bridge_channel.c): MUTEX 2879 bridge_channel_internal_join bridge_channel->chan 0x7f458167ab20 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b44b3e] asterisk bridge_channel.c:2881 bridge_channel_internal_join()
[0x557c73b2624b] asterisk bridge.c:1725 ast_bridge_join()
[0x7f4570781571] app_confbridge.so app_confbridge.c:2910 confbridge_exec()
[0x557c73c1553b] asterisk pbx_app.c:492 pbx_exec()
[0x557c73bfdc18] asterisk pbx.c:2948 pbx_extension_helper()
[0x557c73c01fb1] asterisk pbx.c:4210 ast_spawn_extension()
[0x557c73c02dc1] asterisk pbx.c:4386 __ast_pbx_run()
[0x557c73c0476a] asterisk pbx.c:4711 pbx_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (bridge_channel.c): MUTEX 2885 bridge_channel_internal_join peer 0x7f458178ed60 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b44b8c] asterisk bridge_channel.c:2886 bridge_channel_internal_join()
[0x557c73b2624b] asterisk bridge.c:1725 ast_bridge_join()
[0x7f4570781571] app_confbridge.so app_confbridge.c:2910 confbridge_exec()
[0x557c73c1553b] asterisk pbx_app.c:492 pbx_exec()
[0x557c73bfdc18] asterisk pbx.c:2948 pbx_extension_helper()
[0x557c73c01fb1] asterisk pbx.c:4210 ast_spawn_extension()
[0x557c73c02dc1] asterisk pbx.c:4386 __ast_pbx_run()
[0x557c73c0476a] asterisk pbx.c:4711 pbx_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f456706b700 LWP:4732 (bridge_channel_ind_thread started at [ 1892] bridge.c bridge_impart_internal())
=== ---> Lock #0 (bridge_channel.c): MUTEX 2877 bridge_channel_internal_join bridge_channel->bridge 0x7f458154bd90 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b3cfd6] asterisk bridge.h:484 _ast_bridge_lock()
[0x557c73b44b0b] asterisk bridge_channel.c:2879 bridge_channel_internal_join()
[0x557c73b26618] asterisk bridge.c:1798 bridge_channel_ind_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (bridge_channel.c): MUTEX 2879 bridge_channel_internal_join bridge_channel->chan 0x7f458178ed60 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b44b3e] asterisk bridge_channel.c:2881 bridge_channel_internal_join()
[0x557c73b26618] asterisk bridge.c:1798 bridge_channel_ind_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (bridge_channel.c): MUTEX 2885 bridge_channel_internal_join peer 0x7f458167ab20 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b539f5] asterisk channel.c:1291 ast_channel_by_name_cb()
[0x557c73b1524d] asterisk astobj2_container.c:326 internal_ao2_traverse()
[0x557c73b1557a] asterisk astobj2_container.c:421 __ao2_callback_data()
[0x557c73b53934] asterisk channel.c:1275 ast_channel_callback()
[0x557c73b54065] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
[0x557c73b9c516] asterisk devicestate.c:295 ast_parse_device_state()
[0x557c73b9c86f] asterisk devicestate.c:371 _ast_device_state()
[0x557c73b9cd3d] asterisk devicestate.c:464 do_state_change()
[0x557c73b9d116] asterisk devicestate.c:540 do_devstate_changes()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f456913a700 LWP:4733 (pbx_thread started at [ 4736] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2_container.c): MUTEX 308 internal_ao2_traverse self 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f45664b3700 LWP:5410 (attempt_thread started at [ 475] pbx_spool.c launch_service())
=== ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f457564d1a8 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b9f08b] asterisk dial.c:440 ast_dial_prerun()
[0x557c73c0f280] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x557c73c0fae6] asterisk pbx.c:8444 ast_pbx_outgoing_exten_predial()
[0x557c73c0fa4b] asterisk pbx.c:8423 ast_pbx_outgoing_exten()
[0x7f459000c078] pbx_spool.so pbx_spool.c:443 attempt_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f4576079de0 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524ab] asterisk channel.c:922 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x557c73b969b8] asterisk core_unreal.c:1204 ast_unreal_new_channels()
[0x557c73b92a01] asterisk core_local.c:1027 local_request_with_stream_topology()
[0x557c73b64566] asterisk channel.c:6347 request_channel()
[0x557c73b64c36] asterisk channel.c:6442 ast_request()
[0x557c73b9ebbe] asterisk dial.c:368 begin_dial_prerun()
[0x557c73b9f0b1] asterisk dial.c:441 ast_dial_prerun()
[0x557c73c0f280] asterisk pbx.c:8264 pbx_outgoing_attempt()
[0x557c73c0fae6] asterisk pbx.c:8444 ast_pbx_outgoing_exten_predial()
[0x557c73c0fa4b] asterisk pbx.c:8423 ast_pbx_outgoing_exten()
[0x7f459000c078] pbx_spool.so pbx_spool.c:443 attempt_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x557c7608d500 (1)
[0x557c73bdae18] asterisk lock.c:272 __ast_pthread_mutex_lock()
[0x557c73b128c1] asterisk astobj2.c:241 __ao2_lock()
[0x557c73b524da] asterisk channel.c:924 __ast_channel_alloc_ap()
[0x557c73b527c2] asterisk channel.c:966 __ast_channel_alloc()
[0x7f45714d9bbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
[0x7f45714f3ec9] chan_iax2.so chan_iax2.c:10612 socket_process_helper()
[0x7f457150119b] chan_iax2.so chan_iax2.c:12073 socket_process()
[0x7f4571501851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
[0x557c73cb11a5] asterisk utils.c:1572 dummy_start()
[0x7f4592b21fa3] libpthread.so.0 pthread_create.c:487 start_thread()
[0x7f45925b5eff] libc.so.6 clone.S:97 clone()
=== --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
=== -------------------------------------------------------------------
===
{noformat}
By: N A (InterLinked) 2022-08-05 16:02:02.601-0500

Okay, just happened again. IAX2 and device state are there again. I happened to catch this about as soon as it started (within 15-20 seconds). Not to mention, the system had crashed due to PJSIP just 40 minutes before, so it was a pretty "new" run.

So it looks like this may be an IAX2/device state issue as you suggested, so it can opened as such I suppose.

I should clarify there were no IAX2 calls actually up when this happened - just two PJSIP calls (one in, one out). So maybe IAX2 related but it's not related to an actual call, per se - possibly just device state.
By: N A (InterLinked) 2022-08-24 19:13:15.989-0500

Okay, upon looking into this more, I don't actually think this has anything to do with IAX2, although IAX2 exposes it.
As you said, since IAX2 doesn't natively do device state, ast_parse_device_state gets called, which doesn't happen for all channel drivers.

Since that calls ast_channel_get_by_name_prefix, that tries to lock the channel container and iterate channels, in turn locking each channel. The device state update could be for anything. I have backtraces where the channel is IAX2, but others where it's CBAnn, for instance. So that doesn't matter.

The deadlock seems to specifically happen when Asterisk is executing bridge_channel_internal_join on a channel (for example, this can be seen in the attached backtrace, and bridge_channel_wait specifically). There is a part of this function where the channel is locked. I'm not sure if this has to be the same channel or not.

So, this isn't a bug with IAX2, it's a bug with the device state core, and more specifically, I think it's a bug exposed through the device state core, but this is really a problem with bridge_channel_internal_join. It seems if chan is locked there, then we can have deadlock.
By: N A (InterLinked) 2022-10-03 12:18:04.741-0500

ASTERISK-29821