[Home]

Summary:ASTERISK-18142: unresponsive to sip requests
Reporter:Ole Kaas (ole.kaas)Labels:
Date Opened:2011-07-15 04:00:21Date Closed:2011-08-17 15:27:19
Priority:MajorRegression?
Status:Closed/CompleteComponents:Channels/chan_sip/General
Versions:1.8.5.0 Frequency of
Occurrence
Frequent
Related
Issues:
is related toASTERISK-18166 Deadlock: asterisk isn't responding to any sip package anymore
Environment:Linux (Debian 6.0) 2.6.32-5-686 #1 SMP Mon Jun 13 04:13:06 UTC 2011 i686 GNU/LinuxAttachments:( 0) backtrace_crash_11-08-11.txt
( 1) backtrace2.txt
( 2) bt.2011-07-15T08-15-05-0200.txt
( 3) lock.2011-07-29.txt
( 4) singler_backtrace-threads_11-08-05.txt
( 5) singler_core-show-locks_11-08-05.txt
( 6) timerfd_fix.diff
Description:All of a sudden * wont respond to sip requests (examined with ngrep). It is possible to login to the consoe with 'asterisk -r' and 'core show channels' shows no active calls - wich is never the case during normal office hours. So far the fix is to 'kill -9 pidof*' and wait for safe_asterisk to bring it up again. Happens 2-3 times a day.

Attached a backtrace from a core dump produced with 'kill -11'

The issue seems to happen on version 1.8.4.4 too. That system, however, is only under light load - so it might happen once a week.

Comments:By: Ole Kaas (ole.kaas) 2011-07-15 04:01:36.150-0500

backtrace

By: Leif Madsen (lmadsen) 2011-07-26 09:55:28.969-0500

This is likely a problem with res_timing_timerfd, so for a work around you should use res_timing_dahdi for now. Hopefully we'll get res_timing_timerfd fixed up here shortly.

By: caspy (caspy) 2011-07-27 07:48:03.371-0500

i'm getting a deadlock like this with res_timing_dahdi.
next lock i'll try to suply bt.

By: KC (kchan1028) 2011-07-28 12:51:44.551-0500

I am having the exact same problem all of the sudden on Asterisk 1.6.2.19 and Asterisk 1.8.4 but not on Asterisk 1.4. There is no ERROR show in the log file other than the normal ERROR "Dropping this SIP message with Call-ID". My setup is very simple and use FASTAGI. I recompiled with DEBUG_THREADS and DONT_OPTIMIZE.

I tired to get the backtrace but the backtrace is not available.It say No Stack.

Here is the first core show locks

cat core-show-locks.txt

=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 140357420693248 (do_monitor           started at [24746] chan_sip.c restart_monitor())
=== ---> Lock #0 (chan_sip.c): MUTEX 24235 handle_request_do &netlock 0x7fa78adc9c60 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7b9c1) [0x7fa78ab7e9c1]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6]
       /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== ---> Lock #1 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445658]
       /usr/sbin/asterisk(__ao2_callback+0x5f) [0x445b7d]
       /usr/sbin/asterisk(__ao2_find+0x31) [0x445cef]
       /usr/sbin/asterisk() [0x46bc60]
       /usr/sbin/asterisk(ast_channel_get_by_name+0x27) [0x46bc91]
       /usr/lib/asterisk/modules/func_channel.so(+0x46a6) [0x7fa7945676a6]
       /usr/sbin/asterisk(ast_func_write+0x15e) [0x503b87]
       /usr/sbin/asterisk(pbx_builtin_setvar_helper+0xba) [0x51800c]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7a30b) [0x7fa78ab7d30b]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7bf3f) [0x7fa78ab7ef3f]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6]
       /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== ---> Waiting for Lock #2 (channel.c): MUTEX 1641 ast_channel_cmp_cb chan 0x7fa749058f50 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x46b872]
       /usr/sbin/asterisk() [0x445749]
       /usr/sbin/asterisk(__ao2_callback+0x5f) [0x445b7d]
       /usr/sbin/asterisk(__ao2_find+0x31) [0x445cef]
       /usr/sbin/asterisk() [0x46bc60]
       /usr/sbin/asterisk(ast_channel_get_by_name+0x27) [0x46bc91]
       /usr/lib/asterisk/modules/func_channel.so(+0x46a6) [0x7fa7945676a6]
       /usr/sbin/asterisk(ast_func_write+0x15e) [0x503b87]
       /usr/sbin/asterisk(pbx_builtin_setvar_helper+0xba) [0x51800c]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7a30b) [0x7fa78ab7d30b]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7bf3f) [0x7fa78ab7ef3f]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6]
       /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: channel.c line 3661 (__ast_read)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357415556864 (netconsole           started at [ 1344] asterisk.c listener())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 842 internal_ao2_iterator_next a->c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357410477824 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351656638208 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357400827648 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357402351360 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351665272576 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357405906688 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351658161920 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357386098432 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357390669568 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357404382976 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357406922496 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357403875072 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 3661 __ast_read chan 0x7fa749058f50 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_trylock+0xd9) [0x4dd202]
       /usr/sbin/asterisk(__ao2_trylock+0x5a) [0x444a0a]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7baf0) [0x7fa78ab7eaf0]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6]
       /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106]
       /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== -------------------------------------------------------------------
===
=== Thread ID: 140357401843456 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351674414848 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351673906944 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357393209088 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351671875328 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351662733056 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357387622144 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357387114240 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351658669824 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357397272320 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357409462016 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351663748864 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357392193280 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351664256768 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140351660193536 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 140357384574720 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1)
       /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40]
       /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47]
       /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948]
       /usr/sbin/asterisk() [0x445e45]
       /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275]
       /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815]
       /usr/sbin/asterisk() [0x487b2d]
       /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551]
       /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad]
       /usr/sbin/asterisk() [0x43c266]
       /usr/sbin/asterisk() [0x55dca9]
       /lib64/libpthread.so.0() [0x33d9006ccb]
       /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d]
=== --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=======================================================================


By: Richard Mudgett (rmudgett) 2011-07-28 19:13:00.446-0500

KC: Your core show locks looks like the timerfd problem.

Also please do not put files like a core show locks capture inline.  Attach them and describe the contents.

By: KC (kchan1028) 2011-07-29 03:20:51.331-0500

Richard: I apologized for that.

I tried res_timing_dahdi instead timerrfd. This time asterisk didn't hang but very long SIP response time take causing PDD issue. I tested it with sipsak on the same server as asterisk.
received last message 5809.219 ms after first request (test duration).
biggest delay between request and response was 5809.219 ms
2 time(s) the timeout of 500 ms exceeded and request was retransmitted.
SIP warning

There is no backtrace again (Maybe i did something wrong. I use gbd -se asterisk <pid> | tee /tmp/backtrace.txt). I am going to attach the core show locks now. Thank You

By: KC (kchan1028) 2011-07-29 03:22:21.792-0500

core show locks

By: KC (kchan1028) 2011-07-29 11:49:00.823-0500

I recompiled again without DEBUG_THREADS and DONT_OPTIMIZE and it worked fine for the past few hours now.

By: Ole Kaas (ole.kaas) 2011-08-08 04:34:25.978-0500

After noload => timerfd the system have been rock stable. So I've gone ahead and upgraded a much more busy server. This is turning into a nightmare - deadlock within an hour or even shorter. This system is 64bit.

Again - you can connect to the console and "core show channels" is empty. Howner - some times there are calls running - but sip is unresponsive (no backtrace yet).

Linux (Debian) 2.6.32-5-amd64 #1 SMP Tue Jun 14 09:42:28 UTC 2011 x86_64 GNU/Linux

By: Ole Kaas (ole.kaas) 2011-08-08 06:57:09.322-0500

Investigating this issue further (on the 64bit system). All breakdowns seems to happen right after a reload. We have a script that checks for config changes every 10 minutes and it sends a "reload" to asterisk if it is required. All alerts so far are 0 or 1 minute behind auto reload (sipsak connect attempt every minute)

By: singler (singler) 2011-08-08 07:36:03.224-0500

I am experiencing similar problem (32bit system). I am attaching my backtraces, later asterisk debug log will follow

By: singler (singler) 2011-08-11 01:17:01.513-0500

After using dahdi timer I did not receive "threadlock" in two days, but today asterisk crashed.
In syslog I can see: kernel: [9552804.194007] asterisk[10583]: segfault at 4 ip b57db4d2 sp b17fe230 error 4 in chan_sip.so[b5768000+b3000]
I am attaching backtrace_crash_11-08-11.txt

Size of backtrace was 3GB, so I suspect a memory leak (I also reported at ASTERISK-18247 issue).
Could you guys check if your systems do not experience memory leak?

By: Ole Kaas (ole.kaas) 2011-08-15 10:26:16.682-0500

After upgrading to 1.8.6.0-rc1 this issue seems to have vanished. We have issued several full reloads today when needed due to config changes. Not a single deadlock so far.

By: Terry Wilson (twilson) 2011-08-15 13:58:46.267-0500

timerfd_fix.diff is the patch from https://reviewboard.asterisk.org/r/1361/ for 1.8 which should fix this issue. Please test!

By: Ole Kaas (ole.kaas) 2011-08-15 17:46:08.155-0500

Patch applied to 1.8.5.0 on server with arch i386.
Compiled with DONT_OPTIMIZE DEBUG_THREADS MALLOC_DEBUG
noload => res_timer_timerfd.so commented out in config.
verified module in use with "modules show"


By: Terry Wilson (twilson) 2011-08-17 10:33:16.657-0500

Ole: And did it work?

By: Terry Wilson (twilson) 2011-08-17 10:37:13.085-0500

Also, I'm guessing that 1.8.6.0-rc1 has an earlier version of the patch applied which was reverted because it caused some other issues.

By: Ole Kaas (ole.kaas) 2011-08-17 10:48:28.671-0500

Yes. I decided to wait one more day to be sure. There have not been any deadlocks at all.

The server (amd64) running 1.8.6.0-rc1 is currently using res_timing_pthread.so. However those deadlocks was caused by reloads and the timing wasn't reverted to dahdi after upgrade from 1.8.5.0. We will probably revert to dahdi or even fd, but it seems to be stable and we need some quiet time now :)

By: Terry Wilson (twilson) 2011-08-17 15:28:14.407-0500

Fixed in 1.8 r332320