[Home]

Summary:ASTERISK-21129: Lock on do_monitor
Reporter:c0rnoTa (c0rnota)Labels:
Date Opened:2013-02-18 03:08:59.000-0600Date Closed:2013-03-07 11:32:53.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Channels/chan_sip/General
Versions:1.8.20.1 Frequency of
Occurrence		Occasional
Related
Issues:
Environment:Gentoo Linux 3.2.12-gentoo #2 SMP Mon Jun 4 14:49:23 MSK 2012 x86_64 Intel(R) Xeon(R) CPU X5660 @ 2.80GHz GenuineIntel GNU/Linux, based on VMWare virtual system, Phones: Cisco SPA502, Cisco SPA525 with SPA 500S panels. Using SIP realtime. Attachments:
Description:After a while of working, system stop to process calls from sip subscribers. If I call to someone from console, I didn't set that system is 'Calling SIP/...' and INVITE pockets are not sent.
Asterisk was compiled without optimization.

But here is only 'show locks' output:

{noformat}
=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0x7f2d5495e700 (do_monitor           started at [26511] chan_sip.c restart_monitor())
=== ---> Lock #0 (chan_sip.c): MUTEX 26481 do_monitor &monlock 0x7f2d40ac2020 (1)
asterisk(ast_bt_get_addresses+0x1d) [0x4ee0c0]
asterisk(__ast_pthread_mutex_lock+0xd9) [0x4e6f16]
/usr/lib/asterisk/modules/chan_sip.so(+0x83312) [0x7f2d40875312]
asterisk() [0x56ca8c]
/lib64/libpthread.so.0(+0x7aea) [0x7f2d57de5aea]
/lib64/libc.so.6(clone+0x6d) [0x7f2d574481bd]
=== ---> Waiting for Lock #1 (chan_sip.c): MUTEX 8425 sip_pvt_lock_full pvt 0x1b47c70 (1)
asterisk(ast_bt_get_addresses+0x1d) [0x4ee0c0]
asterisk(__ast_pthread_mutex_lock+0xd9) [0x4e6f16]
asterisk(__ao2_lock+0x5a) [0x447ff1]
/usr/lib/asterisk/modules/chan_sip.so(+0x2b09d) [0x7f2d4081d09d]
/usr/lib/asterisk/modules/chan_sip.so(+0x183b9) [0x7f2d4080a3b9]
/usr/lib/asterisk/modules/chan_sip.so(+0x1859b) [0x7f2d4080a59b]
asterisk(ast_sched_runq+0x18e) [0x54fe18]
/usr/lib/asterisk/modules/chan_sip.so(+0x83324) [0x7f2d40875324]
asterisk() [0x56ca8c]
/lib64/libpthread.so.0(+0x7aea) [0x7f2d57de5aea]
/lib64/libc.so.6(clone+0x6d) [0x7f2d574481bd]
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f2d38808700 (pbx_thread           started at [ 5597] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (chan_sip.c): MUTEX 26500 restart_monitor &monlock 0x7f2d40ac2020 (1)
asterisk(ast_bt_get_addresses+0x1d) [0x4ee0c0]
asterisk(__ast_pthread_mutex_lock+0xd9) [0x4e6f16]
/usr/lib/asterisk/modules/chan_sip.so(+0x8344e) [0x7f2d4087544e]
/usr/lib/asterisk/modules/chan_sip.so(+0x875de) [0x7f2d408795de]
asterisk(ast_request+0x27e) [0x47b7b6]
/usr/lib/asterisk/modules/app_dial.so(+0xc7f7) [0x7f2d447d97f7]
/usr/lib/asterisk/modules/app_dial.so(+0x1107b) [0x7f2d447de07b]
asterisk(pbx_exec+0x1fb) [0x507b47]
asterisk() [0x511ae5]
asterisk(ast_spawn_extension+0x65) [0x513723]
asterisk() [0x51417a]
asterisk() [0x515e8d]
asterisk() [0x56ca8c]
/lib64/libpthread.so.0(+0x7aea) [0x7f2d57de5aea]
/lib64/libc.so.6(clone+0x6d) [0x7f2d574481bd]
=== --- ---> Locked Here: chan_sip.c line 26481 (do_monitor)
=== -------------------------------------------------------------------
===
=======================================================================

{noformat}
Comments:By: Matt Jordan (mjordan) 2013-02-18 06:10:54.999-0600

Debugging deadlocks: Please select DEBUG_THREADS and DONT_OPTIMIZE in the Compiler Flags section of menuselect. Recompile and install Asterisk (i.e. make install).  This will then give you the console command "core show locks." When the symptoms of the deadlock present themselves again, please provide output of the deadlock via:

# asterisk -rx "core show locks" | tee /tmp/core-show-locks.txt
# gdb -se "asterisk" <pid of asterisk> | tee /tmp/backtrace.txt
gdb> bt
gdb> bt full
gdb> thread apply all bt

Then attach the core-show-locks.txt and backtrace.txt files to this issue. Thanks!

Please attach the full output of {{core show locks}} as a separate file on the issue.
By: Matt Jordan (mjordan) 2013-03-07 11:32:45.570-0600

Since there's no feedback on this issue, I'll note the following:
* The {{core show locks}} output doesn't indicate a deadlock so much as something locked a pvt and never released it.
* Walking through the current 1.8 codebase, I don't see an incident where a pvt is locked and not unlocked appropriately.

Since there's no backtrace or other information available, there isn't anything that points to what didn't release the lock. As such, I'm closing out the issue as suspended. If more information becomes available, please contact a bug marshal in #asterisk-bugs and we can reopen this.
By: c0rnoTa (c0rnota) 2013-03-11 08:25:24.498-0500

Yes, sorry for long response.
The issue was because Asterisk was installed in Gentoo Linux on VMWare. We have tried to install Asterisk with Gentoo (or Gentoo with Asterisk) on real hardware and there was no problem. Then we'v tried to install Asterisk on CentOS 6.3 on VMWare and problem was not detected as well. We have experienced problem with network on Gentoo and VMWare, that's because we changed different environment. And deadlocks were not detected since locating PBX on real hardware.