| Summary: | ASTERISK-22903: SIP Hangs, Asterisk becomes unresponsive when compiled with DEBUG_THREADS | ||||||
| Reporter: | Craig Arno (craigarno) | Labels: | |||||
| Date Opened: | 2013-11-25 12:27:02.000-0600 | Date Closed: | 2014-06-25 11:12:23 | ||||
| Priority: | Major | Regression? | |||||
| Status: | Closed/Complete | Components: | Core/General | ||||
| Versions: | 11.6.0 | Frequency of Occurrence | Occasional | ||||
| Related Issues: |
| ||||||
| Environment: | OpenSUSE 12.2 x64 (uname -a): Linux pluto 3.4.63-2.44-desktop #1 SMP PREEMPT Wed Oct 2 11:18:32 UTC 2013 (d91a619) x86_64 x86_64 x86_64 GNU/Linux AMD FX-8120 processor. | Attachments: | ( 0) backtrace5.txt ( 1) core-show-locks5.txt ( 2) start-msgs-11.5.1.txt ( 3) start-msgs-11.6.0.txt | ||||
| Description: | I'm here looking for a solution to a SIP/Asterisk hang issue which showed first in Asterisk version 11.6.0. I rolled back to 11.5.1 which works fine on my server.
When I figure out my Asterisk 11.6.0 installation isn't accepting calls, I execute "core stop now", and am told Asterisk is stopping, and everything looks/behaves normal. There is no running Asterisk process. I then restart Asterisk from the command prompt, no options and it looks like it boots normally, except Asterisk will then not respond to commands, like "core stop now". Asterisk also won't quit with a normal "kill" command. The only way to stop Asterisk 11.6.0 is to "kill -9". At this point, starting Asterisk 11.6.0 yet another time and it will execute normally for about another day before sliding silently into this "dead" behavior again. My call volume is very low, only about 1-5/day. All my Sipura (2000, 2100, 3000, etc) devices are connected using SIP. My hardware is AMD FX-8120, ASUS Sabertooth 990FX, 8GB RAM, 9 TB HD... running under OpenSUSE 12.2 x64; Linux pluto 3.4.63-2.44-desktop #1 SMP PREEMPT Wed Oct 2 11:18:32 UTC 2013 (d91a619) x86_64. Asterisk 11.6.0 is compiled with optimization off and thread debugging on. There are no resource issues and all parts are running to factory specifications. This Asterisk is exposed to the Internet for 2 SIP users and incoming/outgoing calls. This is a stable system (not a new installation). I've been silent hoping for another release soon. This looks like a difficult problem to solve, so if I can perform some actions to help without risking an undetectable loss of phone service, please let me know how. 11.5.1 is continuing to run well for me. After reviewing 11.6.0 issues, ASTERISK-22854 looks the closest to what I may be experiencing. I tried the forums http://forums.asterisk.org/viewtopic.php?f=1&t=88593 | ||||||
| Comments: | By: Matt Jordan (mjordan) 2013-11-25 13:02:42.705-0600 Debugging deadlocks: Please select DEBUG_THREADS and DONT_OPTIMIZE in the Compiler Flags section of menuselect. Recompile and install Asterisk (i.e. make install). This will then give you the console command "core show locks." When the symptoms of the deadlock present themselves again, please provide output of the deadlock via: # asterisk -rx "core show locks" | tee /tmp/core-show-locks.txt # gdb -se "asterisk" <pid of asterisk> | tee /tmp/backtrace.txt gdb> bt gdb> bt full gdb> thread apply all bt Then attach the core-show-locks.txt and backtrace.txt files to this issue. Thanks! Note that if this is happening on shutdown, the 'core show locks' will probably not be possible. However, you can still attach gdb to the asterisk instance shutting down and get a backtrace. That will tell us whether or not it is the CEL issue with PostgreSQL, or something else. By: Craig Arno (craigarno) 2013-11-25 13:15:00.553-0600 Oh, I see I've already confused things. This Asterisk installation is stand-alone. There is no database backend, no GUI front-end. Just plain Asterisk 11.6.0. I'll switch back to 11.6.0 to repeat, it's already compiled with Debug Threads and Don't optimize (my default installation setup). This issue repeated enough times already. What's scary is there is no trace, no indication, just Asterisk becoming semi-dead, silently, no longer answering phone calls. It takes about a day to die. I usually get an email from someone saying "what's wrong with your phone system?". I'll be back when I have data. By: Craig Arno (craigarno) 2013-11-25 13:45:13.309-0600 As fortune would have it, I fired up 11.6.0 and it immediately displayed the behavior I described above. Here is some data from this bizarre state: {code} pluto:/home/install/linux/asterisk/asterisk-11.6.0 # asterisk -rx "core show locks" | tee /tmp/core-show-locks.txt ======================================================================= === 11.6.0 === Currently Held Locks ======================================================================= === === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked) === ======================================================================= # asterisk -vvvr Asterisk 11.6.0, Copyright (C) 1999 - 2013 Digium, Inc. and others. Created by Mark Spencer <markster@digium.com> Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. This is free software, with components licensed under the GNU General Public License version 2 and other licenses; you are welcome to redistribute it under certain conditions. Type 'core show license' for details. ========================================================================= Connected to Asterisk 11.6.0 currently running on pluto (pid = 16042) pluto*CLI> pluto*CLI> pluto*CLI> == Using SIP RTP TOS bits 184 == Using SIP RTP CoS mark 5 [Nov 25 11:18:59] NOTICE[16061][C-00000000]: chan_sip.c:25399 handle_request_invite: Call from 'LINE2' (192.168.90.15:5061) to extension '2062035630' rejected because extension not found in context 'home'. -- Remote UNIX connection -- Remote UNIX connection disconnected pluto*CLI> pluto*CLI> == Using SIP RTP TOS bits 184 == Using SIP RTP CoS mark 5 [Nov 25 11:20:22] NOTICE[16061][C-00000001]: chan_sip.c:25399 handle_request_invite: Call from 'LINE2' (192.168.90.15:5061) to extension '2062035630' rejected because extension not found in context 'home'. pluto*CLI> == Using SIP RTP TOS bits 184 == Using SIP RTP CoS mark 5 [Nov 25 11:20:40] NOTICE[16061][C-00000002]: chan_sip.c:25399 handle_request_invite: Call from '472002' (192.168.90.15:5060) to extension '2062035640' rejected because extension not found in context 'home'. pluto*CLI> == Using SIP RTP TOS bits 184 == Using SIP RTP CoS mark 5 [Nov 25 11:23:36] NOTICE[16061][C-00000003]: chan_sip.c:25399 handle_request_invite: Call from 'IPKall' (66.54.140.46:5060) to extension 'LINE1' rejected because extension not found in context 'from-sip'. pluto*CLI> core show locks ======================================================================= === 11.6.0 === Currently Held Locks ======================================================================= === === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked) === ======================================================================= pluto:/home/install/linux/asterisk/asterisk-11.6.0 # ps aux|grep asterisk root 16042 1.1 2.7 2725200 221828 ? Ssl 11:17 0:06 asterisk root 16170 0.0 0.0 39592 3352 pts/0 S+ 11:26 0:00 rasterisk vvvr root 16182 0.0 0.0 8260 868 pts/1 S+ 11:27 0:00 grep --color=auto asterisk pluto:/home/install/linux/asterisk/asterisk-11.6.0 # gdb -se "asterisk" 16042 | tee /tmp/backtrace.txt GNU gdb (GDB) SUSE (7.5.1-2.5.1) Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... 16042: No such file or directory. (gdb) bt No stack. (gdb) bt full No stack. (gdb) thread apply all bt (gdb) quit pluto:/home/install/linux/asterisk/asterisk-11.6.0 # kill 16042 pluto:/home/install/linux/asterisk/asterisk-11.6.0 # ps aux|grep asterisk root 16042 1.0 2.7 2725200 221828 ? Ssl 11:17 0:08 asterisk root 16170 0.0 0.0 39592 3352 pts/0 S+ 11:26 0:00 rasterisk vvvr root 16592 0.0 0.0 8264 872 pts/1 S+ 11:31 0:00 grep --color=auto asterisk pluto:/home/install/linux/asterisk/asterisk-11.6.0 # kill -9 16042 You have mail in /var/mail/craig pluto:/home/install/linux/asterisk/asterisk-11.6.0 # ps aux|grep asterisk root 16599 0.0 0.0 8260 872 pts/1 S+ 11:31 0:00 grep --color=auto asterisk {code} After Asterisk 11.6.0 is killed, then another start and it appears to run correctly. Past experience indicates this will last for about a day. I'm reinstalling 11.5.1 until we decide to try something else. By: Matt Jordan (mjordan) 2013-11-25 16:00:07.048-0600 So, the core show locks didn't show anything, which isn't terribly surprising. Unfortunately, the gdb backtrace didn't work because you forgot to pass -p to gdb: {noformat} For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... 16042: No such file or directory. (gdb) bt No stack. {noformat} All of that being said, I don't think this is a locking issue. {noformat} == Using SIP RTP TOS bits 184 == Using SIP RTP CoS mark 5 [Nov 25 11:20:22] NOTICE[16061][C-00000001]: chan_sip.c:25399 handle_request_invite: Call from 'LINE2' (192.168.90.15:5061) to extension '2062035630' rejected because extension not found in context 'home'. pluto*CLI> {noformat} This looks more like a general configuration problem: how are these peers/friends configured? By: Craig Arno (craigarno) 2013-11-25 17:12:35.281-0600 Here are the two basic peer configurations which have mostly worked as is since Asterisk 1.0 (with a few syntax changes). And as I said, kill Asterisk process, then rerun and everything works again with no configuration changes. {code} [LINE2] type=peer host=dynamic localnet=192.168.90.0/24 secret=mySecret callerid="Craig" <52> dtmfmode=rfc2833 dtmf=rfc2833 relaxdtmf=yes ;YES=Interpret DTMF tones during a call (for "outside" RFC2833 channel) toneduration=150 insecure=port,invite canreinvite=no ;Asterisk must stay in the middle because SPA2100 is behind NAT nat=no ;At home, so no NAT between SPA2100 and Asterisk register => 4255551212@sip.broadvoice.com:myBVsecret:4255551212@sip.broadvoice.com/4255551212 [sip.broadvoice.com] ;Following is new with Asterisk 1.8 defaultuser=4255551212 ;End of new 1.8 stuff type=peer user=phone qualify=yes ;NAT=yes - only if Asterisk is behind a NAT (myDomain.com is not) nat=no ;Disable canreinvite if SIP device is behind a NAT, BV doesn't allow reinvite canreinvite=no host=sip.broadvoice.com fromdomain=sip.broadvoice.com fromuser=4255551212 secret=0v01hwstyk username=4255551212 insecure=port,invite ;"port,invite" works. "very" is another setting. context=from-broadvoice authname=4255551212 {code} By: Craig Arno (craigarno) 2013-11-25 17:47:26.772-0600 Saw your note about the -p option... I reinstalled 11.6.0 again. It appeared to boot and was again in altered states displaying same symptoms (...extension not found in context 'home'). These symptoms disappear and the system runs normally (probably for about a day) after killing the Asterisk 11.6.0 process and restarting Asterisk 11.6.0. Since I can't see an easy way to attach the back-trace here, I'll just put it inline wrapped with "code" statements: {code} pluto:/home/install/linux/asterisk/asterisk-11.6.0 # cat backtrace3.txt GNU gdb (GDB) SUSE (7.5.1-2.5.1) Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/asterisk...done. Attaching to program: /usr/sbin/asterisk, process 24891 Reading symbols from /usr/lib/libasteriskssl.so.1...done. Loaded symbols for /usr/lib/libasteriskssl.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxml2.so.2 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /usr/lib64/libsqlite3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsqlite3.so.0 Reading symbols from /lib64/libssl.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libssl.so.1.0.0 Reading symbols from /lib64/libcrypto.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypto.so.1.0.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [New LWP 24924] [New LWP 24923] [New LWP 24922] [New LWP 24921] [New LWP 24920] [New LWP 24919] [New LWP 24918] [New LWP 24917] [New LWP 24916] [New LWP 24915] [New LWP 24914] [New LWP 24913] [New LWP 24912] [New LWP 24911] [New LWP 24910] [New LWP 24909] [New LWP 24908] [New LWP 24907] [New LWP 24905] [New LWP 24904] [New LWP 24903] [New LWP 24902] [New LWP 24901] [New LWP 24900] [New LWP 24899] [New LWP 24898] [New LWP 24897] [New LWP 24896] [New LWP 24895] [New LWP 24894] [New LWP 24893] [New LWP 24892] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done. Loaded symbols for /lib64/libtinfo.so.5 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/liblzma.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/liblzma.so.5 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /usr/lib64/asterisk/modules/res_smdi.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_smdi.so Reading symbols from /usr/lib64/asterisk/modules/res_crypto.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_crypto.so Reading symbols from /usr/lib64/asterisk/modules/res_stun_monitor.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_stun_monitor.so Reading symbols from /usr/lib64/asterisk/modules/res_monitor.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_monitor.so Reading symbols from /usr/lib64/asterisk/modules/res_speech.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_speech.so Reading symbols from /usr/lib64/asterisk/modules/res_fax.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_fax.so Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib64/asterisk/modules/res_http_websocket.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_http_websocket.so Reading symbols from /usr/lib64/asterisk/modules/res_calendar.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_calendar.so Reading symbols from /usr/lib64/asterisk/modules/res_ael_share.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_ael_share.so Reading symbols from /usr/lib64/asterisk/modules/res_agi.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_agi.so Reading symbols from /usr/lib64/asterisk/modules/res_musiconhold.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_musiconhold.so Reading symbols from /usr/lib64/asterisk/modules/pbx_config.so...done. Loaded symbols for /usr/lib64/asterisk/modules/pbx_config.so Reading symbols from /usr/lib64/asterisk/modules/codec_alaw.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_alaw.so Reading symbols from /usr/lib64/asterisk/modules/app_sms.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_sms.so Reading symbols from /usr/lib64/asterisk/modules/app_milliwatt.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_milliwatt.so Reading symbols from /usr/lib64/asterisk/modules/app_channelredirect.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_channelredirect.so Reading symbols from /usr/lib64/asterisk/modules/codec_g726.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_g726.so Reading symbols from /usr/lib64/asterisk/modules/func_presencestate.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_presencestate.so Reading symbols from /usr/lib64/asterisk/modules/bridge_builtin_features.so...done. Loaded symbols for /usr/lib64/asterisk/modules/bridge_builtin_features.so Reading symbols from /usr/lib64/asterisk/modules/func_dialgroup.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_dialgroup.so Reading symbols from /usr/lib64/asterisk/modules/res_adsi.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_adsi.so Reading symbols from /usr/lib64/asterisk/modules/app_voicemail.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_voicemail.so Reading symbols from /usr/lib64/asterisk/modules/app_alarmreceiver.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_alarmreceiver.so Reading symbols from /usr/lib64/asterisk/modules/format_jpeg.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_jpeg.so Reading symbols from /usr/lib64/asterisk/modules/codec_ulaw.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_ulaw.so Reading symbols from /usr/lib64/asterisk/modules/app_dumpchan.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_dumpchan.so Reading symbols from /usr/lib64/asterisk/modules/app_macro.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_macro.so Reading symbols from /usr/lib64/asterisk/modules/app_dial.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_dial.so Reading symbols from /usr/lib64/asterisk/modules/app_mp3.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_mp3.so Reading symbols from /usr/lib64/asterisk/modules/app_playtones.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_playtones.so Reading symbols from /usr/lib64/asterisk/modules/func_realtime.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_realtime.so Reading symbols from /usr/lib64/asterisk/modules/func_extstate.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_extstate.so Reading symbols from /usr/lib64/asterisk/modules/app_morsecode.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_morsecode.so Reading symbols from /usr/lib64/asterisk/modules/bridge_softmix.so...done. Loaded symbols for /usr/lib64/asterisk/modules/bridge_softmix.so Reading symbols from /usr/lib64/asterisk/modules/func_frame_trace.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_frame_trace.so Reading symbols from /usr/lib64/asterisk/modules/codec_ilbc.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_ilbc.so Reading symbols from /usr/lib64/asterisk/modules/app_chanisavail.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_chanisavail.so Reading symbols from /usr/lib64/asterisk/modules/pbx_spool.so...done. Loaded symbols for /usr/lib64/asterisk/modules/pbx_spool.so Reading symbols from /usr/lib64/asterisk/modules/cel_manager.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cel_manager.so Reading symbols from /usr/lib64/asterisk/modules/app_db.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_db.so Reading symbols from /usr/lib64/asterisk/modules/cdr_manager.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cdr_manager.so Reading symbols from /usr/lib64/asterisk/modules/app_echo.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_echo.so Reading symbols from /usr/lib64/asterisk/modules/app_ices.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_ices.so Reading symbols from /usr/lib64/asterisk/modules/app_confbridge.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_confbridge.so Reading symbols from /usr/lib64/asterisk/modules/res_timing_pthread.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_timing_pthread.so Reading symbols from /usr/lib64/asterisk/modules/chan_local.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_local.so Reading symbols from /usr/lib64/asterisk/modules/cel_sqlite3_custom.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cel_sqlite3_custom.so Reading symbols from /usr/lib64/asterisk/modules/codec_gsm.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_gsm.so Reading symbols from /usr/lib64/asterisk/modules/app_cdr.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_cdr.so Reading symbols from /usr/lib64/asterisk/modules/app_while.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_while.so Reading symbols from /usr/lib64/asterisk/modules/chan_bridge.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_bridge.so Reading symbols from /usr/lib64/asterisk/modules/codec_a_mu.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_a_mu.so Reading symbols from /usr/lib64/asterisk/modules/func_sprintf.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_sprintf.so Reading symbols from /usr/lib64/asterisk/modules/func_cdr.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_cdr.so Reading symbols from /usr/lib64/asterisk/modules/app_externalivr.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_externalivr.so Reading symbols from /usr/lib64/asterisk/modules/res_convert.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_convert.so Reading symbols from /usr/lib64/asterisk/modules/format_g726.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_g726.so Reading symbols from /usr/lib64/asterisk/modules/app_amd.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_amd.so Reading symbols from /usr/lib64/asterisk/modules/app_celgenuserevent.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_celgenuserevent.so Reading symbols from /usr/lib64/asterisk/modules/app_queue.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_queue.so Reading symbols from /usr/lib64/asterisk/modules/app_image.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_image.so Reading symbols from /usr/lib64/asterisk/modules/app_verbose.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_verbose.so Reading symbols from /usr/lib64/asterisk/modules/cdr_custom.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cdr_custom.so Reading symbols from /usr/lib64/asterisk/modules/app_privacy.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_privacy.so Reading symbols from /usr/lib64/asterisk/modules/app_speech_utils.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_speech_utils.so Reading symbols from /usr/lib64/asterisk/modules/bridge_multiplexed.so...done. Loaded symbols for /usr/lib64/asterisk/modules/bridge_multiplexed.so Reading symbols from /usr/lib64/asterisk/modules/app_url.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_url.so Reading symbols from /usr/lib64/asterisk/modules/func_strings.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_strings.so Reading symbols from /usr/lib64/asterisk/modules/app_waitforring.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_waitforring.so Reading symbols from /usr/lib64/asterisk/modules/chan_multicast_rtp.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_multicast_rtp.so Reading symbols from /usr/lib64/asterisk/modules/func_aes.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_aes.so Reading symbols from /usr/lib64/asterisk/modules/res_config_sqlite3.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_config_sqlite3.so Reading symbols from /usr/lib64/asterisk/modules/func_math.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_math.so Reading symbols from /usr/lib64/asterisk/modules/func_hangupcause.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_hangupcause.so Reading symbols from /usr/lib64/asterisk/modules/func_vmcount.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_vmcount.so Reading symbols from /usr/lib64/asterisk/modules/app_directed_pickup.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_directed_pickup.so Reading symbols from /usr/lib64/asterisk/modules/app_followme.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_followme.so Reading symbols from /usr/lib64/asterisk/modules/func_callcompletion.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_callcompletion.so Reading symbols from /usr/lib64/asterisk/modules/res_realtime.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_realtime.so Reading symbols from /usr/lib64/asterisk/modules/chan_agent.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_agent.so Reading symbols from /usr/lib64/asterisk/modules/app_chanspy.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_chanspy.so Reading symbols from /usr/lib64/asterisk/modules/app_waitforsilence.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_waitforsilence.so Reading symbols from /usr/lib64/asterisk/modules/func_dialplan.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_dialplan.so Reading symbols from /usr/lib64/asterisk/modules/bridge_simple.so...done. Loaded symbols for /usr/lib64/asterisk/modules/bridge_simple.so Reading symbols from /usr/lib64/asterisk/modules/app_playback.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_playback.so Reading symbols from /usr/lib64/asterisk/modules/pbx_ael.so...done. Loaded symbols for /usr/lib64/asterisk/modules/pbx_ael.so Reading symbols from /usr/lib64/asterisk/modules/res_format_attr_silk.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_format_attr_silk.so Reading symbols from /usr/lib64/asterisk/modules/app_dictate.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_dictate.so Reading symbols from /usr/lib64/asterisk/modules/func_base64.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_base64.so Reading symbols from /usr/lib64/asterisk/modules/func_timeout.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_timeout.so Reading symbols from /usr/lib64/asterisk/modules/app_forkcdr.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_forkcdr.so Reading symbols from /usr/lib64/asterisk/modules/cdr_sqlite3_custom.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cdr_sqlite3_custom.so Reading symbols from /usr/lib64/asterisk/modules/app_transfer.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_transfer.so Reading symbols from /usr/lib64/asterisk/modules/format_sln.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_sln.so Reading symbols from /usr/lib64/asterisk/modules/func_lock.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_lock.so Reading symbols from /usr/lib64/asterisk/modules/res_rtp_asterisk.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_rtp_asterisk.so Reading symbols from /usr/lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libuuid.so.1 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /usr/lib64/asterisk/modules/chan_iax2.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_iax2.so Reading symbols from /usr/lib64/asterisk/modules/format_vox.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_vox.so Reading symbols from /usr/lib64/asterisk/modules/res_format_attr_celt.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_format_attr_celt.so Reading symbols from /usr/lib64/asterisk/modules/app_parkandannounce.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_parkandannounce.so Reading symbols from /usr/lib64/asterisk/modules/format_g729.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_g729.so Reading symbols from /usr/lib64/asterisk/modules/res_clialiases.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_clialiases.so Reading symbols from /usr/lib64/asterisk/modules/app_waituntil.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_waituntil.so Reading symbols from /usr/lib64/asterisk/modules/func_channel.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_channel.so Reading symbols from /usr/lib64/asterisk/modules/res_timing_timerfd.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_timing_timerfd.so Reading symbols from /usr/lib64/asterisk/modules/func_sha1.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_sha1.so Reading symbols from /usr/lib64/asterisk/modules/func_audiohookinherit.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_audiohookinherit.so Reading symbols from /usr/lib64/asterisk/modules/func_cut.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_cut.so Reading symbols from /usr/lib64/asterisk/modules/func_pitchshift.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_pitchshift.so Reading symbols from /usr/lib64/asterisk/modules/func_env.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_env.so Reading symbols from /usr/lib64/asterisk/modules/func_shell.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_shell.so Reading symbols from /usr/lib64/asterisk/modules/app_zapateller.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_zapateller.so Reading symbols from /usr/lib64/asterisk/modules/app_page.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_page.so Reading symbols from /usr/lib64/asterisk/modules/res_phoneprov.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_phoneprov.so Reading symbols from /usr/lib64/asterisk/modules/func_db.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_db.so Reading symbols from /usr/lib64/asterisk/modules/func_config.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_config.so Reading symbols from /usr/lib64/asterisk/modules/app_sendtext.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_sendtext.so Reading symbols from /usr/lib64/asterisk/modules/app_nbscat.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_nbscat.so Reading symbols from /usr/lib64/asterisk/modules/codec_lpc10.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_lpc10.so Reading symbols from /usr/lib64/asterisk/modules/app_exec.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_exec.so Reading symbols from /usr/lib64/asterisk/modules/func_srv.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_srv.so Reading symbols from /usr/lib64/asterisk/modules/func_jitterbuffer.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_jitterbuffer.so Reading symbols from /usr/lib64/asterisk/modules/func_md5.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_md5.so Reading symbols from /usr/lib64/asterisk/modules/res_rtp_multicast.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_rtp_multicast.so Reading symbols from /usr/lib64/asterisk/modules/res_limit.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_limit.so Reading symbols from /usr/lib64/asterisk/modules/cdr_csv.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cdr_csv.so Reading symbols from /usr/lib64/asterisk/modules/app_readexten.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_readexten.so Reading symbols from /usr/lib64/asterisk/modules/format_pcm.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_pcm.so Reading symbols from /usr/lib64/asterisk/modules/app_minivm.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_minivm.so Reading symbols from /usr/lib64/asterisk/modules/chan_skinny.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_skinny.so Reading symbols from /usr/lib64/asterisk/modules/func_module.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_module.so Reading symbols from /usr/lib64/asterisk/modules/func_sysinfo.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_sysinfo.so Reading symbols from /usr/lib64/asterisk/modules/res_security_log.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_security_log.so Reading symbols from /usr/lib64/asterisk/modules/format_wav.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_wav.so Reading symbols from /usr/lib64/asterisk/modules/app_festival.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_festival.so Reading symbols from /usr/lib64/asterisk/modules/func_groupcount.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_groupcount.so Reading symbols from /usr/lib64/asterisk/modules/app_test.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_test.so Reading symbols from /usr/lib64/asterisk/modules/codec_adpcm.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_adpcm.so Reading symbols from /usr/lib64/asterisk/modules/pbx_loopback.so...done. Loaded symbols for /usr/lib64/asterisk/modules/pbx_loopback.so Reading symbols from /usr/lib64/asterisk/modules/format_siren7.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_siren7.so Reading symbols from /usr/lib64/asterisk/modules/app_mixmonitor.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_mixmonitor.so Reading symbols from /usr/lib64/asterisk/modules/format_gsm.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_gsm.so Reading symbols from /usr/lib64/asterisk/modules/res_format_attr_h263.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_format_attr_h263.so Reading symbols from /usr/lib64/asterisk/modules/app_authenticate.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_authenticate.so Reading symbols from /usr/lib64/asterisk/modules/app_userevent.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_userevent.so Reading symbols from /usr/lib64/asterisk/modules/func_version.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_version.so Reading symbols from /usr/lib64/asterisk/modules/app_stack.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_stack.so Reading symbols from /usr/lib64/asterisk/modules/app_read.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_read.so Reading symbols from /usr/lib64/asterisk/modules/app_directory.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_directory.so Reading symbols from /usr/lib64/asterisk/modules/res_mutestream.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_mutestream.so Reading symbols from /usr/lib64/asterisk/modules/app_getcpeid.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_getcpeid.so Reading symbols from /usr/lib64/asterisk/modules/format_ilbc.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_ilbc.so Reading symbols from /usr/lib64/asterisk/modules/res_clioriginate.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_clioriginate.so Reading symbols from /usr/lib64/asterisk/modules/codec_g722.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_g722.so Reading symbols from /usr/lib64/asterisk/modules/func_callerid.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_callerid.so Reading symbols from /usr/lib64/asterisk/modules/cel_custom.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cel_custom.so Reading symbols from /usr/lib64/asterisk/modules/format_siren14.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_siren14.so Reading symbols from /usr/lib64/asterisk/modules/chan_phone.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_phone.so Reading symbols from /usr/lib64/asterisk/modules/app_disa.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_disa.so Reading symbols from /usr/lib64/asterisk/modules/chan_unistim.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_unistim.so Reading symbols from /usr/lib64/asterisk/modules/func_logic.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_logic.so Reading symbols from /usr/lib64/asterisk/modules/app_originate.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_originate.so Reading symbols from /usr/lib64/asterisk/modules/cdr_syslog.so...done. Loaded symbols for /usr/lib64/asterisk/modules/cdr_syslog.so Reading symbols from /usr/lib64/asterisk/modules/pbx_dundi.so...done. Loaded symbols for /usr/lib64/asterisk/modules/pbx_dundi.so Reading symbols from /usr/lib64/asterisk/modules/app_senddtmf.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_senddtmf.so Reading symbols from /usr/lib64/asterisk/modules/chan_sip.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_sip.so Reading symbols from /usr/lib64/asterisk/modules/app_controlplayback.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_controlplayback.so Reading symbols from /usr/lib64/asterisk/modules/func_blacklist.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_blacklist.so Reading symbols from /usr/lib64/asterisk/modules/func_global.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_global.so Reading symbols from /usr/lib64/asterisk/modules/app_system.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_system.so Reading symbols from /usr/lib64/asterisk/modules/func_uri.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_uri.so Reading symbols from /usr/lib64/asterisk/modules/app_softhangup.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_softhangup.so Reading symbols from /usr/lib64/asterisk/modules/chan_oss.so...done. Loaded symbols for /usr/lib64/asterisk/modules/chan_oss.so Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libX11.so.6 Reading symbols from /usr/lib64/libxcb.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxcb.so.1 Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXau.so.6 Reading symbols from /usr/lib64/asterisk/modules/format_g719.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_g719.so Reading symbols from /usr/lib64/asterisk/modules/res_config_ldap.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_config_ldap.so Reading symbols from /usr/lib64/libldap-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libldap-2.4.so.2 Reading symbols from /usr/lib64/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/liblber-2.4.so.2 Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /usr/lib64/asterisk/modules/app_adsiprog.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_adsiprog.so Reading symbols from /usr/lib64/asterisk/modules/format_g723.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_g723.so Reading symbols from /usr/lib64/asterisk/modules/format_h263.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_h263.so Reading symbols from /usr/lib64/asterisk/modules/app_talkdetect.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_talkdetect.so Reading symbols from /usr/lib64/asterisk/modules/res_format_attr_h264.so...done. Loaded symbols for /usr/lib64/asterisk/modules/res_format_attr_h264.so Reading symbols from /usr/lib64/asterisk/modules/codec_resample.so...done. Loaded symbols for /usr/lib64/asterisk/modules/codec_resample.so Reading symbols from /usr/lib64/asterisk/modules/func_iconv.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_iconv.so Reading symbols from /usr/lib64/asterisk/modules/pbx_realtime.so...done. Loaded symbols for /usr/lib64/asterisk/modules/pbx_realtime.so Reading symbols from /usr/lib64/asterisk/modules/format_h264.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_h264.so Reading symbols from /usr/lib64/asterisk/modules/func_rand.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_rand.so Reading symbols from /usr/lib64/asterisk/modules/app_sayunixtime.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_sayunixtime.so Reading symbols from /usr/lib64/asterisk/modules/func_devstate.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_devstate.so Reading symbols from /usr/lib64/asterisk/modules/format_wav_gsm.so...done. Loaded symbols for /usr/lib64/asterisk/modules/format_wav_gsm.so Reading symbols from /usr/lib64/asterisk/modules/func_enum.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_enum.so Reading symbols from /usr/lib64/asterisk/modules/app_record.so...done. Loaded symbols for /usr/lib64/asterisk/modules/app_record.so Reading symbols from /usr/lib64/asterisk/modules/func_volume.so...done. Loaded symbols for /usr/lib64/asterisk/modules/func_volume.so 0x00007f5796f1600c in __lll_lock_wait () from /lib64/libpthread.so.0 Missing separate debuginfos, use: zypper install cyrus-sasl-debuginfo-2.1.25-21.2.1.x86_64 glibc-debuginfo-2.15-22.17.1.x86_64 libX11-6-debuginfo-1.5.0-2.7.1.x86_64 libXau6-debuginfo-1.0.7-2.1.2.x86_64 libgcc47-debuginfo-4.7.1_20120723-1.1.1.x86_64 libldap-2_4-2-debuginfo-2.4.31-2.1.3.x86_64 liblzma5-debuginfo-5.0.3-12.2.2.x86_64 libncurses5-debuginfo-5.9-12.5.1.x86_64 libopenssl1_0_0-debuginfo-1.0.1e-2.12.1.x86_64 libsqlite3-0-debuginfo-3.7.12.1-2.1.2.x86_64 libuuid1-debuginfo-2.21.2-4.5.1.x86_64 libxcb1-debuginfo-1.8.1-2.8.1.x86_64 libxml2-2-debuginfo-2.7.8+git20120223-8.30.1.x86_64 zlib-debuginfo-1.2.7-2.1.2.x86_64 (gdb) bt #0 0x00007f5796f1600c in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f5796f11ee7 in _L_lock_1018 () from /lib64/libpthread.so.0 #2 0x00007f5796f11d79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000500ba7 in ast_reentrancy_lock (lt=0xde66f0) at /home/install/linux/asterisk/asterisk-11.6.0/include/asterisk/lock.h:420 #4 0x00000000005014b8 in __ast_pthread_mutex_lock (filename=0x5d8414 "logger.c", lineno=1522, func=0x5d92f1 <__PRETTY_FUNCTION__.14206> "ast_log_full", mutex_name=0x5d8efd "&(&logmsgs)->lock", t=0x831210 <logmsgs+16>) at lock.c:218 #5 0x0000000000508c73 in ast_log_full (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s", ap=0x7fffaf3a5e58) at logger.c:1522 #6 0x0000000000508f46 in ast_log_callid (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s") at logger.c:1561 #7 0x000000000050966a in __ast_verbose_ap (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, callid=0x0, fmt=0xd9b2aa4 "", ap=0x7fffaf3a6080) at logger.c:1841 #8 0x000000000050974f in __ast_verbose (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, fmt=0x5d78b2 "%s => (%s)\n") at logger.c:1853 #9 0x00000000004ff66b in start_resource (mod=0x15a78b0) at loader.c:850 #10 0x0000000000500060 in load_resource_list (load_order=0x7fffaf3a6390, global_symbols=0, mod_count=0x7fffaf3a637c) at loader.c:1047 #11 0x00000000005006a5 in load_modules (preload_only=0) at loader.c:1200 #12 0x000000000044e9f0 in main (argc=1, argv=0x7fffaf3a79c8) at asterisk.c:4239 (gdb) #0 0x00007f5796f1600c in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f5796f11ee7 in _L_lock_1018 () from /lib64/libpthread.so.0 #2 0x00007f5796f11d79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000500ba7 in ast_reentrancy_lock (lt=0xde66f0) at /home/install/linux/asterisk/asterisk-11.6.0/include/asterisk/lock.h:420 #4 0x00000000005014b8 in __ast_pthread_mutex_lock (filename=0x5d8414 "logger.c", lineno=1522, func=0x5d92f1 <__PRETTY_FUNCTION__.14206> "ast_log_full", mutex_name=0x5d8efd "&(&logmsgs)->lock", t=0x831210 <logmsgs+16>) at lock.c:218 #5 0x0000000000508c73 in ast_log_full (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s", ap=0x7fffaf3a5e58) at logger.c:1522 #6 0x0000000000508f46 in ast_log_callid (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s") at logger.c:1561 #7 0x000000000050966a in __ast_verbose_ap (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, callid=0x0, fmt=0xd9b2aa4 "", ap=0x7fffaf3a6080) at logger.c:1841 #8 0x000000000050974f in __ast_verbose (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, fmt=0x5d78b2 "%s => (%s)\n") at logger.c:1853 #9 0x00000000004ff66b in start_resource (mod=0x15a78b0) at loader.c:850 #10 0x0000000000500060 in load_resource_list (load_order=0x7fffaf3a6390, global_symbols=0, mod_count=0x7fffaf3a637c) at loader.c:1047 #11 0x00000000005006a5 in load_modules (preload_only=0) at loader.c:1200 #12 0x000000000044e9f0 in main (argc=1, argv=0x7fffaf3a79c8) at asterisk.c:4239 (gdb) bt full #0 0x00007f5796f1600c in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f5796f11ee7 in _L_lock_1018 () from /lib64/libpthread.so.0 No symbol table info available. #2 0x00007f5796f11d79 in pthread_mutex_lock () from /lib64/libpthread.so.0 No symbol table info available. #3 0x0000000000500ba7 in ast_reentrancy_lock (lt=0xde66f0) at /home/install/linux/asterisk/asterisk-11.6.0/include/asterisk/lock.h:420 res = 32767 #4 0x00000000005014b8 in __ast_pthread_mutex_lock (filename=0x5d8414 "logger.c", lineno=1522, func=0x5d92f1 <__PRETTY_FUNCTION__.14206> "ast_log_full", mutex_name=0x5d8efd "&(&logmsgs)->lock", t=0x831210 <logmsgs+16>) at lock.c:218 tmp = {addresses = {0x508fe8 <ast_bt_get_addresses+29>, 0x5014ac <__ast_pthread_mutex_lock+201>, 0x508c73 <ast_log_full+2762>, 0x508f46 <ast_log_callid+198>, 0x50966a <__ast_verbose_ap+1003>, 0x50974f <__ast_verbose+211>, 0x4ff66b <start_resource+260>, 0x500060 <load_resource_list+632>, 0x5006a5 <load_modules+1507>, 0x44e9f0 <main+8183>, 0x7f57984d9455 <__libc_start_main+245>, 0x429ed9 <_start+41>, 0x7fffaf3a79c0, 0x613b0000, 0x7fffaf3a5b50, 0x429eb0 <_start>, 0x7fffaf3a79c0, 0x0, 0x0, 0x57d850 <ast_strftime_locale+607>, 0x5293dc4a, 0x0, 0x7fffaf3a5d10, 0x830f68 <dateformat+8>, 0x100, 0x7fffaf3a5c00, 0xf00dbb548, 0x8a0057a66b, 0x8a0000008a, 0x5ef243, 0xffffffffaf3a5f38, 0xd963118}, num_frames = 12, alloced = 0} res = 0 lt = 0xde66f0 canlog = 0 bt = 0x0 __PRETTY_FUNCTION__ = "__ast_pthread_mutex_lock" #5 0x0000000000508c73 in ast_log_full (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s", ap=0x7fffaf3a5e58) at logger.c:1522 logmsg = 0xd9bfb60 buf = 0xdb7930 tm = {tm_sec = 58, tm_min = 24, tm_hour = 15, tm_mday = 25, tm_mon = 10, tm_year = 113, tm_wday = 1, tm_yday = 328, tm_isdst = 0, tm_gmtoff = -28800, tm_zone = 0x0, tm_usec = 872134} now = {tv_sec = 1385421898, tv_usec = 872134} res = 46 datestring = "Nov 25 15:24:58\000\bw\333\000\000\000\000\000\345\240R\230W\177\000\000\001\200\255\373\000\000\000\000\bw\333\000\000\000\000\000\bw\333\000\000\000\000\000\bw\333\000\000\000\000\000\bw\333\000\000\000\000\000\nw\333\000\000\000\000\000\ax\333\000\000\000\000\000\bw\333\000\000\000\000\000\ax\333", '\000' <repeats 54 times>, "g^\000\000\000\000\000\000\000]", '\000' <repeats 13 times>, "\030\000\000\000\060\000\000\000\060\000\000\000\060\000\000\000\220]:\257\377\177", '\000' <repeats 18 times>, " ]:\257\377\177\000\000\377\377\377\377\000\000\000\000\020^:\257\377\177\000\000\262x]\000\000\000\000\000 d\205\230W\177\000" __PRETTY_FUNCTION__ = "ast_log_full" #6 0x0000000000508f46 in ast_log_callid (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s") at logger.c:1561 ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7fffaf3a5f30, reg_save_area = 0x7fffaf3a5e70}} #7 0x000000000050966a in __ast_verbose_ap (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, callid=0x0, fmt=0xd9b2aa4 "", ap=0x7fffaf3a6080) at logger.c:1841 p = 0xd9b2aa4 "" prefixed = 0xdb76f0 buf = 0xd9b2a60 res = 44 prefix = 0x5d90e9 " " magic = -2 '\376' #8 0x000000000050974f in __ast_verbose (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, fmt=0x5d78b2 "%s => (%s)\n") at logger.c:1853 callid = 0x0 ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7fffaf3a6160, reg_save_area = 0x7fffaf3a60a0}} #9 0x00000000004ff66b in start_resource (mod=0x15a78b0) at loader.c:850 tmp = "\033[0;33mSMS/PSTN handler\033[0m\000uration\033[0m\000\060m\000\033[0m\000he current call\033\300e[\001\000\000\000\000\220\350c\001\000\000\000\000\020b:\257\000\000\000\000\002\000\000\000\000\000\000\000\002\000\000\000 \000\200\200\300e[\001\000\000\000\000\220\350c\001\000\000\000\000 =W\001\000\000\000\000Pb:\257\377\177\000\000C:O\000\000\000\000\000\220Bg\001\001\000\000\000 =W\001\000\000\000\000\240b:\257\003\000\000\000\002\000\000\000\001\000\000\000\000\000;a\000\000\000\000\260\236B\000\000\000\000\000\200b:\257\377\177\000\000\071<O\000\000\000\000\000\240b:\257\001\000\000\000 =W\001\000\000\000\000\360"... res = AST_MODULE_LOAD_SUCCESS __PRETTY_FUNCTION__ = "start_resource" #10 0x0000000000500060 in load_resource_list (load_order=0x7fffaf3a6390, global_symbols=0, mod_count=0x7fffaf3a637c) at loader.c:1047 resource_heap = 0x1573d20 order = 0x0 mod = 0x15a78b0 count = 110 res = 0 __PRETTY_FUNCTION__ = "load_resource_list" #11 0x00000000005006a5 in load_modules (preload_only=0) at loader.c:1200 cfg = 0x1570b50 mod = 0x0 order = 0x0 v = 0x0 load_count = 191 load_order = {first = 0x0, last = 0x0} res = 0 config_flags = {flags = 0} modulecount = 9 dirent = 0x0 dir = 0x1573b90 __PRETTY_FUNCTION__ = "load_modules" #12 0x000000000044e9f0 in main (argc=1, argv=0x7fffaf3a79c8) at asterisk.c:4239 c = -1 filename = "/root/.asterisk_history", '\000' <repeats 56 times> hostname = "pluto", '\000' <repeats 58 times> tmp = "\000\000\000\000\000\000\000\000\225\371X\000\000\000\000\000H\020\000\000\000\000\000\000@S\203\000\000\000\000\000\365\330\003\000\000\000\000\000pB\333\000\000\000\000\000`w:\257\377\177\000\000\272\032Y\000\000\000\000\000p1\333\000\000\000\000\000P\242\202\000\000\000\000" xarg = 0x0 x = 1 f = 0xdb74b0 sigs = {__val = {134238211, 0 <repeats 15 times>}} num = 0 isroot = 1 rundir_exists = 1 buf = 0xdb3170 "\350\217P" runuser = 0x0 rungroup = 0x0 remotesock = 0x0 moduleresult = 0 l = {rlim_cur = 1024, rlim_max = 4096} __FUNCTION__ = "main" __PRETTY_FUNCTION__ = "main" (gdb) thread apply all bt Thread 33 (Thread 0x7f5798c54700 (LWP 24892)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5efc24 "taskprocessor.c", lineno=313, func=0x5f0330 <__PRETTY_FUNCTION__.8075> "tps_processing_function", cond_name=0x5eff91 "&i->poll_cond", mutex_name=0x5eff7a "&i->taskprocessor_lock", cond=0xddef40, t=0xddef78) at lock.c:565 #2 0x000000000057fd42 in tps_processing_function (data=0xddef38) at taskprocessor.c:313 #3 0x0000000000592421 in dummy_start (data=0xde0770) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f5798bd8700 (LWP 24893)): #0 0x00007f579859414f in poll () from /lib64/libc.so.6 #1 0x0000000000446773 in listener (unused=0x0) at asterisk.c:1401 #2 0x0000000000592421 in dummy_start (data=0xde6540) at utils.c:1162 #3 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #4 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f5798b5c700 (LWP 24894)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5d8414 "logger.c", lineno=1157, func=0x5d93b2 <__PRETTY_FUNCTION__.14111> "logger_thread", cond_name=0x5d8f1d "&logcond", mutex_name=0x5d8f0f "&logmsgs.lock", cond=0x84b460 <logcond>, t=0x831210 <logmsgs+16>) at lock.c:565 #2 0x00000000005078a4 in logger_thread (data=0x0) at logger.c:1157 #3 0x0000000000592421 in dummy_start (data=0xde65b0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f5796677700 (LWP 24895)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5c84b4 "db.c", lineno=959, func=0x5c9446 <__PRETTY_FUNCTION__.14449> "db_sync_thread", cond_name=0x5c93f0 "&dbcond", mutex_name=0x5c8719 "&dblock", cond=0x84a560 <dbcond>, t=0x82d880 <dblock>) at lock.c:565 #2 0x00000000004b52da in db_sync_thread (data=0x0) at db.c:959 #3 0x0000000000592421 in dummy_start (data=0xf0e250) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f57965fb700 (LWP 24896)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5efc24 "taskprocessor.c", lineno=313, func=0x5f0330 <__PRETTY_FUNCTION__.8075> "tps_processing_function", cond_name=0x5eff91 "&i->poll_cond", mutex_name=0x5eff7a "&i->taskprocessor_lock", cond=0x13a0840, t=0x13a0878) at lock.c:565 #2 0x000000000057fd42 in tps_processing_function (data=0x13a0838) at taskprocessor.c:313 #3 0x0000000000592421 in dummy_start (data=0x136d1f0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f579657f700 (LWP 24897)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5c9574 "devicestate.c", lineno=549, func=0x5c9ca0 <__PRETTY_FUNCTION__.13391> "do_devstate_changes", cond_name=0x5c9827 "&change_pending", mutex_name=0x5c9837 "&state_changes.lock", cond=0x84a600 <change_pending>, t=0x82deb0 <state_changes+16>) at lock.c:565 #2 0x00000000004b63d7 in do_devstate_changes (data=0x0) at devicestate.c:549 #3 0x0000000000592421 in dummy_start (data=0x13ccf10) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f5796503700 (LWP 24898)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5e6e34 "presencestate.c", lineno=299, func=0x5e7060 <__PRETTY_FUNCTION__.13344> "do_presence_changes", cond_name=0x5e6fb0 "&change_pending", mutex_name=0x5e6fc0 "&state_changes.lock", cond=0x84b640 <change_pending>, t=0x834370 <state_changes+16>) at lock.c:565 #2 0x000000000054ab41 in do_presence_changes (data=0x0) at presencestate.c:299 #3 0x0000000000592421 in dummy_start (data=0x13f5860) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f5796487700 (LWP 24899)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5efc24 "taskprocessor.c", lineno=313, func=0x5f0330 <__PRETTY_FUNCTION__.8075> "tps_processing_function", cond_name=0x5eff91 "&i->poll_cond", mutex_name=0x5eff7a "&i->taskprocessor_lock", cond=0x1403960, t=0x1403998) at lock.c:565 #2 0x000000000057fd42 in tps_processing_function (data=0x1403958) at taskprocessor.c:313 #3 0x0000000000592421 in dummy_start (data=0x1403e40) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f579640b700 (LWP 24900)): #0 0x00007f579859414f in poll () from /lib64/libc.so.6 #1 0x00000000004d7538 in do_parking_thread (ignore=0x0) at features.c:5324 #2 0x0000000000592421 in dummy_start (data=0x140c200) at utils.c:1162 #3 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #4 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f579638f700 (LWP 24901)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5efc24 "taskprocessor.c", lineno=313, func=0x5f0330 <__PRETTY_FUNCTION__.8075> "tps_processing_function", cond_name=0x5eff91 "&i->poll_cond", mutex_name=0x5eff7a "&i->taskprocessor_lock", cond=0x1429b20, t=0x1429b58) at lock.c:565 #2 0x000000000057fd42 in tps_processing_function (data=0x1429b18) at taskprocessor.c:313 #3 0x0000000000592421 in dummy_start (data=0x142b540) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f5796313700 (LWP 24902)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5ecde4 "sched.c", lineno=120, func=0x5ed295 <__PRETTY_FUNCTION__.11422> "sched_run", cond_name=0x5ecdf7 "&con->sched_thread->cond", mutex_name=0x5ecdec "&con->lock", cond=0x1410ce8, t=0x142acd0) at lock.c:565 #2 0x0000000000573552 in sched_run (data=0x142acd0) at sched.c:120 #3 0x0000000000592421 in dummy_start (data=0x1429d00) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f5794b97700 (LWP 24903)): #0 0x00007f5796f13c61 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502b3c in __ast_cond_timedwait (filename=0x7f5795233054 "res_calendar.c", lineno=1783, func=0x7f579523434c <__PRETTY_FUNCTION__.13978> "do_refresh", cond_name=0x7f579523372a "&refresh_condition", mutex_name=0x7f57952335f5 "&refreshlock", cond=0x7f5795436b60 <refresh_condition>, t=0x7f5795436b20 <refreshlock>, abstime=0x7f5794b96dd0) at lock.c:646 #2 0x00007f5795232b13 in do_refresh (data=0x0) at res_calendar.c:1783 #3 0x0000000000592421 in dummy_start (data=0x15927e0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f57940d7700 (LWP 24904)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57642a24b4 "res_timing_pthread.c", lineno=476, func=0x7f57642a2883 <__PRETTY_FUNCTION__.9216> "do_timing", cond_name=0x7f57642a24f4 "&timing_thread.cond", mutex_name=0x7f57642a24e0 "&timing_thread.lock", cond=0x7f57644a42e0 <timing_thread+64>, t=0x7f57644a42a8 <timing_thread+8>) at lock.c:565 #2 0x00007f57642a21ce in do_timing (arg=0x0) at res_timing_pthread.c:476 #3 0x0000000000592421 in dummy_start (data=0x15cd0d0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f5747fff700 (LWP 24905)): #0 0x00007f579856dcad in nanosleep () from /lib64/libc.so.6 #1 0x00007f57985966b4 in usleep () from /lib64/libc.so.6 #2 0x00007f57599dc223 in pj_thread_sleep () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so #3 0x00007f57599db2bc in pj_ioqueue_poll () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so #4 0x00007f57599aaaaf in ice_worker_thread (data=0x0) at res_rtp_asterisk.c:1208 #5 0x00007f57599dbe1f in thread_main () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so #6 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #7 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f576408f700 (LWP 24907)): #0 0x00007f579859414f in poll () from /lib64/libc.so.6 #1 0x00000000004463cc in netconsole (vconsole=0x865b00 <consoles>) at asterisk.c:1323 #2 0x0000000000592421 in dummy_start (data=0x7f5788001990) at utils.c:1162 #3 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #4 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f574c1e7700 (LWP 24908)): #0 0x00007f579859414f in poll () from /lib64/libc.so.6 #1 0x00000000004fb2c3 in ast_io_wait (ioc=0x17006c0, howlong=1000) at io.c:279 #2 0x00007f5750346e9d in do_monitor (data=0x0) at chan_sip.c:28925 #3 0x0000000000592421 in dummy_start (data=0x1705520) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f57477fe700 (LWP 24909)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5efc24 "taskprocessor.c", lineno=313, func=0x5f0330 <__PRETTY_FUNCTION__.8075> "tps_processing_function", cond_name=0x5eff91 "&i->poll_cond", mutex_name=0x5eff7a "&i->taskprocessor_lock", cond=0x7a0ec10, t=0x7a0ec48) at lock.c:565 #2 0x000000000057fd42 in tps_processing_function (data=0x7a0ec08) at taskprocessor.c:313 #3 0x0000000000592421 in dummy_start (data=0x7a10440) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f5747782700 (LWP 24910)): #0 0x00007f5796f13c61 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502b3c in __ast_cond_timedwait (filename=0x5ecde4 "sched.c", lineno=126, func=0x5ed295 <__PRETTY_FUNCTION__.11422> "sched_run", cond_name=0x5ecdf7 "&con->sched_thread->cond", mutex_name=0x5ecdec "&con->lock", cond=0xd713368, t=0xd710620, abstime=0x7f5747781dc0) at lock.c:646 #2 0x000000000057367a in sched_run (data=0xd710620) at sched.c:126 #3 0x0000000000592421 in dummy_start (data=0x7a10440) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f5747706700 (LWP 24911)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd7555d8, t=0xd7555a0) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd7544e0) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd74def0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f574768a700 (LWP 24912)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd758a88, t=0xd758a50) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd757990) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd74def0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f574760e700 (LWP 24913)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd75b398, t=0xd75b360) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd75a2a0) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd7507a0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f5747592700 (LWP 24914)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd75dca8, t=0xd75dc70) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd75cbb0) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd750450) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f5747516700 (LWP 24915)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd7605b8, t=0xd760580) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd75f4c0) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd7507a0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f574749a700 (LWP 24916)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd762ec8, t=0xd762e90) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd761dd0) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd750450) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f574741e700 (LWP 24917)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd7657d8, t=0xd7657a0) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd7646e0) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd7507a0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f57473a2700 (LWP 24918)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd7680e8, t=0xd7680b0) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd766ff0) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd750450) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f5747326700 (LWP 24919)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd76a9f8, t=0xd76a9c0) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd769900) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd7507a0) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f57472aa700 (LWP 24920)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x7f57590e4bb4 "chan_iax2.c", lineno=11948, func=0x7f57590ec330 <__PRETTY_FUNCTION__.20118> "iax2_process_thread", cond_name=0x7f57590e4ee9 "&thread->cond", mutex_name=0x7f57590e4edb "&thread->lock", cond=0xd76d438, t=0xd76d400) at lock.c:565 #2 0x00007f57590ccd38 in iax2_process_thread (data=0xd76c340) at chan_iax2.c:11948 #3 0x0000000000592421 in dummy_start (data=0xd750450) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f574722e700 (LWP 24921)): #0 0x00007f579859414f in poll () from /lib64/libc.so.6 #1 0x00000000004fb2c3 in ast_io_wait (ioc=0xd713550, howlong=1000) at io.c:279 #2 0x00007f57590cf39d in network_thread (ignore=0x0) at chan_iax2.c:12435 #3 0x0000000000592421 in dummy_start (data=0xd750450) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f57471b2700 (LWP 24922)): #0 0x00007f579859cfcd in accept () from /lib64/libc.so.6 #1 0x00007f5754d4641f in accept_thread (ignore=0x0) at chan_skinny.c:7001 #2 0x0000000000592421 in dummy_start (data=0xd7c2dd0) at utils.c:1162 #3 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #4 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f5747136700 (LWP 24923)): #0 0x00007f5796f138f4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000502572 in __ast_cond_wait (filename=0x5ecde4 "sched.c", lineno=120, func=0x5ed295 <__PRETTY_FUNCTION__.11422> "sched_run", cond_name=0x5ecdf7 "&con->sched_thread->cond", mutex_name=0x5ecdec "&con->lock", cond=0xd7c4ba8, t=0xd7c4b20) at lock.c:565 #2 0x0000000000573552 in sched_run (data=0xd7c4b20) at sched.c:120 #3 0x0000000000592421 in dummy_start (data=0xd7c2c50) at utils.c:1162 #4 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #5 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f57470ba700 (LWP 24924)): #0 0x00007f579856dcad in nanosleep () from /lib64/libc.so.6 #1 0x00007f5759c09c48 in lock_broker (unused=0x0) at func_lock.c:196 #2 0x0000000000592421 in dummy_start (data=0xd91e650) at utils.c:1162 #3 0x00007f5796f0fe0e in start_thread () from /lib64/libpthread.so.0 #4 0x00007f579859c2cd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f5798c56700 (LWP 24891)): #0 0x00007f5796f1600c in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f5796f11ee7 in _L_lock_1018 () from /lib64/libpthread.so.0 #2 0x00007f5796f11d79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000500ba7 in ast_reentrancy_lock (lt=0xde66f0) at /home/install/linux/asterisk/asterisk-11.6.0/include/asterisk/lock.h:420 #4 0x00000000005014b8 in __ast_pthread_mutex_lock (filename=0x5d8414 "logger.c", lineno=1522, func=0x5d92f1 <__PRETTY_FUNCTION__.14206> "ast_log_full", mutex_name=0x5d8efd "&(&logmsgs)->lock", t=0x831210 <logmsgs+16>) at lock.c:218 #5 0x0000000000508c73 in ast_log_full (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s", ap=0x7fffaf3a5e58) at logger.c:1522 #6 0x0000000000508f46 in ast_log_callid (level=5, file=0x5d7344 "loader.c", line=850, function=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", callid=0x0, fmt=0x5d87ff "%s") at logger.c:1561 #7 0x000000000050966a in __ast_verbose_ap (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, callid=0x0, fmt=0xd9b2aa4 "", ap=0x7fffaf3a6080) at logger.c:1841 #8 0x000000000050974f in __ast_verbose (file=0x5d7344 "loader.c", line=850, func=0x5d7c80 <__PRETTY_FUNCTION__.25219> "start_resource", level=1, fmt=0x5d78b2 "%s => (%s)\n") at logger.c:1853 #9 0x00000000004ff66b in start_resource (mod=0x15a78b0) at loader.c:850 #10 0x0000000000500060 in load_resource_list (load_order=0x7fffaf3a6390, global_symbols=0, mod_count=0x7fffaf3a637c) at loader.c:1047 #11 0x00000000005006a5 in load_modules (preload_only=0) at loader.c:1200 #12 0x000000000044e9f0 in main (argc=1, argv=0x7fffaf3a79c8) at asterisk.c:4239 (gdb) quit A debugging session is active. Inferior 1 [process 24891] will be detached. Quit anyway? (y or n) Detaching from program: /usr/sbin/asterisk, process 24891 {code} The only thing I can think to try next is a "make clean" followed by a full rebuild of Asterisk 11.6.0. Then review the 11.6.0 build output carefully for issues. Nothing struck me as being "wrong" when I built (I've done this hundreds of times since Asterisk 0.x), but maybe I did miss something? It's possible. By: Craig Arno (craigarno) 2013-11-28 13:38:20.247-0600 Output after clean rebuild of Asterisk 11.6.0 and "core stop now" of asterisk 11.5.1, then start of asterisk 11.6.0. Asterisk is in a non-functioning state: # asterisk -rx "core show locks" | tee core-show-locks5.txt # gdb -se "asterisk" -p 7427 | tee backtrace5.txt By: Craig Arno (craigarno) 2013-11-28 14:25:55.534-0600 More debug... output of asterisk -vvvc | tee start-msgs-11.5.1.txt and same for 11.6.0. It looks like Asterisk 11.6.0 is hanging right around IAX2 load, chan_iax2.so => (Inter Asterisk eXchange (Ver 2)). 4 Lines at the end on 11.6.0 are me attempting to make a call and getting the same results as before. (repeatable) As I said earlier, this is a "stable" system meaning it rarely changes configuration. The configuration in use works fine with 11.5.1. 11.6.0 hangs. The line "chan_multicast_rtp.so => (Multicast RTP Paging Channel)" in 11.5.1 startup never shows up in 11.6.0 startup. By: Matt Jordan (mjordan) 2013-12-30 19:59:47.029-0600 I'm not able to reproduce this on either Ubuntu, Fedora, or CentOS. I highly suspect that others aren't going to be able to reproduce this at will either. If you disable DEBUG_THREADS, do you get the same behaviour? By: Craig Arno (craigarno) 2014-01-02 10:57:09.506-0600 I recompiled 11.6.0 with No Optimization, this time without DEBUG_THREADS and installed 30-Dec-2013 8pm. The system is still running and accepting/placing calls as of 2-Jan-2014 9am. The only difference between the installation of 11.6.0 which works and 11.6.0 which eventually hangs is Thread Debugging in the build. With Thread Debugging off, I'm not able to reproduce this issue in about 2.5 days. It usually will have reasserted itself by now. I see 11.7.0 is released, so maybe I should try 11.7.0 to address another issue I experience (affecting call quality) but can't quantify yet. This is why I'm trying new versions. By: Craig Arno (craigarno) 2014-01-02 11:20:25.193-0600 Matt, Thank you for checking all those different system types. A recap; I use external Broadvoice and IPKall services with Asterisk 11.6.0. My hardware is (ASUS Sabertooth 990FX, AMD FX-8120, 8GB RAM) with software distribution (OpenSUSE 12.2 x64). The only other comment I can think to add is Dan Berninger (formerly FreeWorld Dialup) once commented my Asterisk configuration is the most complex (non-commercial) setup he knows about. My system is built around a standard Asterisk distribution with no add-ons. With Thread debug disabled, no optimization, this system is continuing to operate after 2.5 days. Previous failures with Thread debug enabled usually stopped responding to calls in about 1-1.5 days. By: Rusty Newton (rnewton) 2014-01-21 09:00:29.542-0600 I've run various versions of Asterisk many times with DEBUG_THREADS and haven't run into this. We don't have enough information to narrow down the issue at this time. Craig if you can narrow down the issue further by minimalizing the configuration, that is removing many parts (modules, config files, etc) unnecessary to basic operation and attempting to reproduce until the problem no longer occurs with DEBUG_THREADS running, that could lead to at least identifying which area of Asterisk may harbor the issue. Ideally if you could provide a few config files that would guarantee we could reproduce the issue with DEBUG_THREADS enabled then we would be able to move forward with further diagnosis. Most people do not run in production with DEBUG_THREADS enabled, which is likely why we haven't seen this issue before. However, since we use DEBUG_THREADS for debugging issues, it would be nice to resolve bugs within its use. [Edit - I take back what I said about not seeing others reporting this issue. I do see some others, I'll link them] By: Rusty Newton (rnewton) 2014-01-21 09:08:22.927-0600 Linking ASTERISK-22851 and ASTERISK-22857 Craig, can you test with the patch from here https://reviewboard.asterisk.org/r/2826/ and see if you can reproduce the issue with DEBUG_THREADS enabled? By: Craig Arno (craigarno) 2014-01-21 11:39:16.271-0600 After disabling "Debug Threads" and leaving the "No Optimization" I have not seen this problem repeat on my installation with 11.6.0. I've since upgraded to 11.7.0 and again don't experience this issue. The best clue I have to what is going on is the console start-up messages from 11.5.1(which worked) and 11.6.0(which would silently hang) I posted here 28/Nov/13 12:25 PM. The 11.6.0 messages stopped where I suspect the hang is occurring. I am willing to share my config files with the Asterisk team if we can handle these files securely. i.e. they have multiple passwords which can be used to gain access to my system. I see regular attempts to gain this sort of access to my system coming in from around the world, so I need these files to be handled securely. How would you like to proceed? By: Matteo (mpiazzatnetbug) 2014-03-06 04:08:46.177-0600 Just to let you know that we have notice the same behavior on our 11.6 installation. We didn't find the time to produce debug log and test etc. So we didn't open an issue and simply we are still using 11.2. I will try to apply the patch https://reviewboard.asterisk.org/r/2826/ By: Matt Jordan (mjordan) 2014-06-25 11:12:23.876-0500 This issue has been reported a lot under a number of different names. To keep the issue tracker sane - and so everyone who runs into this problem can find the same solution - I'm closing this out as a duplicate of ASTERISK-22851. | ||||||