[Home]

Summary:ASTERISK-17129: [patch] Asterisk Crashing/Hangs
Reporter:Harpreet (vois)Labels:
Date Opened:2010-12-17 13:16:09.000-0600Date Closed:2011-03-01 10:22:30.000-0600
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) backtrace-threads.txt
( 1) core-show-locks-1.txt
( 2) core-show-locks-2.txt
( 3) core-show-locks-new.txt
( 4) issues_18497.diff
Description:Asterisk 1.8 is crashing/hangs, running on Ubuntu. I was using Asterisk 1.4.36 and it was running fine. 600 peers(400 TLS and 200 UDP). Full debug is enabled, but not able to find cause of crash. Wat else i can do to figure out why crash is happening?

When asterisk crashes, i am unable to stop asterisk with core stop now, i have to stop it from system using /etc/init.d/asterisk stop and it takes few min to stop.
Comments:By: Paul Belanger (pabelanger) 2010-12-17 15:20:48.000-0600

This is not the appropriate place to ask for help, if the future your best to use http://www.asterisk.org/support

--
Thank you for your bug report. In order to move your issue forward, we require a backtrace[1] from the core file produced after the crash.

Also, be sure you have DONT_OPTIMIZE enabled in menuselect within the Compiler Flags section, then:

make install

after enabling, reproduce the crash, and then execute the backtrace[1] instructions.

When complete, attach that file to this issue report.

[1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace
By: Harpreet (vois) 2010-12-18 17:33:40.000-0600

Asterisk is not producing core file. Asterisk is not exactly crashing(shutting down), it just freezes we cannot make any sip calls. But if i run any CLI Command for example core show channels, sip show peers it work fine. I tried sip reload but it did not worked, only this did not worked. I enable sip debug which asterisk was frozen and it also worked. I even tried to register new sip account at that time and asterisk is accepting registration also at the time of freeze. If i make any call i see following in cli:

VERBOSE[29826] pbx.c:     -- Executing [6040000000@internal:1] Dial("SIP/username-00000685", "SIP/6040000000@SIPSERVER2") in new stack
VERBOSE[29826] netsock.c:   == Using UDPTL CoS mark 5
VERBOSE[29826] netsock2.c:   == Using SIP RTP CoS mark 5

If i stop asterisk, i does not stop instantaneously. I see following messages in cli and then after 30-50 seconds asterisk shutdown.

[Dec 17 11:30:37] WARNING[18274] channel.c: Channel allocation failed: Refusing due to active shutdown
[Dec 17 11:30:37] WARNING[18274] chan_sip.c: Unable to allocate AST channel structure for SIP channel
[Dec 17 11:30:37] NOTICE[18274] chan_sip.c: Unable to create/find SIP channel for this INVITE

I have full log saved for this crash if that can help.
Note: Asterisk 1.4.36 was working fine with current situation i only faced this problem when i upgraded to 1.8.1.  Asterisk 1.8.1 and 1.8.2-rc1 both doing exactly same. It only happens in day time when call volume is high, i have not seen asterisk freezing when call volume is low.


By: Ronald Chan (loloski) 2010-12-18 20:31:27.000-0600

just curious what does ulimit -n show ?
By: Harpreet (vois) 2010-12-18 21:36:30.000-0600

ulimit -n
1000000
ulimit -c
unlimited


By: Paul Belanger (pabelanger) 2010-12-19 12:43:07.000-0600

It might be a deadlock.
---
Debugging deadlocks:

Please select DEBUG_THREADS and DONT_OPTIMIZE in the Compiler Flags section of menuselect. Recompile and install Asterisk (i.e. make install)

This will then give you the console command:

core show locks

When the symptoms of the deadlock present themselves again, please provide output of the deadlock via:

# asterisk -rx "core show locks" | tee /tmp/core-show-locks.txt

# gdb -se "asterisk" <pid of asterisk> | tee /tmp/backtrace.txt

gdb> bt
gdb> bt full
gdb> thread apply all bt

Then attach the core-show-locks.txt and backtrace.txt files to this issue. Thanks!
By: Harpreet (vois) 2010-12-19 14:46:12.000-0600

I have attached core-show-locks.txt. When i run gdb -se "asterisk" <pid of asterisk> | tee /tmp/backtrace.txt i get following messages. I am unable to produce backtrace.txt for you because of this.

gdb -se "asterisk" 8335 | tee /tmp/backtrace.txt
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
8335: No such file or directory.
(gdb) quit

Let me know if their is another way to get backtrace.txt.
By: Paul Belanger (pabelanger) 2010-12-19 16:47:07.000-0600

try:

$ sudo gdb -ex "bt" -ex "bt full" -ex "thread apply all bt" --batch /usr/sbin/asterisk `pidof asterisk` > /tmp/backtrace-threads.txt
By: Harpreet (vois) 2010-12-19 18:08:01.000-0600

i uploaded backtrace-threads.txt and core-show-locks-new.txt.
By: Harpreet (vois) 2010-12-22 07:03:36.000-0600

hi team, do you have any ETR for this issue?


By: David Vossel (dvossel) 2010-12-23 12:07:12.000-0600

try the patch i uploaded
By: Harpreet (vois) 2011-01-15 10:02:39.000-0600

Issue is fixed after applying the patch. Thanks Guys.


By: Harpreet (vois) 2011-02-24 08:29:36.000-0600

i have tried 1.8.2.3 and 1.8.2.4 but i am facing same problem without the patch.
By: Ross Beer (rossbeer) 2011-02-24 08:36:41.000-0600

I am also having the same issue with Asterisk 1.6.2.16.2 on CentOS, has the patch been rolled out into the main code?

Also accurs on 1.6.2.17-rc3.
By: Marcin Kowalczyk (kowalma) 2011-02-24 13:53:01.000-0600

Today I hit same issue with 1.8.2.4
I've just applied patch and I will update Mantis in couple of days
By: Freddi Hansen (freddi_fonet) 2011-02-27 05:57:51.000-0600

I had 2-4 sip deadlocks per day that typically lasted around 10 minutes then it would recover by itself to process sip signalling again.
I did install this patch and we haven't had any sip-deadlocks since.
version is 1.6.2. rev 308525
By: David Vossel (dvossel) 2011-03-01 09:55:27.000-0600

It looks like this resolves the issue for people. I'll commit the patch.
By: Digium Subversion (svnbot) 2011-03-01 10:05:27.000-0600

Repository: asterisk
Revision: 309083

U   branches/1.6.2/channels/chan_sip.c

------------------------------------------------------------------------
r309083 | dvossel | 2011-03-01 10:05:26 -0600 (Tue, 01 Mar 2011) | 9 lines

Fixes thread blocking issue in the sip TCP/TLS implementation.

(closes issue ASTERISK-17129)
Reported by: vois
Patches:
     issues_18497.diff uploaded by dvossel (license 671)
Tested by: vois, rossbeer, kowalma, Freddi_Fonet


------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=309083
By: Digium Subversion (svnbot) 2011-03-01 10:09:13.000-0600

Repository: asterisk
Revision: 309084

_U  branches/1.8/
U   branches/1.8/channels/chan_sip.c

------------------------------------------------------------------------
r309084 | dvossel | 2011-03-01 10:09:12 -0600 (Tue, 01 Mar 2011) | 15 lines

Merged revisions 309083 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.6.2

........
 r309083 | dvossel | 2011-03-01 10:05:25 -0600 (Tue, 01 Mar 2011) | 9 lines
 
 Fixes thread blocking issue in the sip TCP/TLS implementation.
 
 (closes issue ASTERISK-17129)
 Reported by: vois
 Patches:
       issues_18497.diff uploaded by dvossel (license 671)
 Tested by: vois, rossbeer, kowalma, Freddi_Fonet
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=309084
By: Digium Subversion (svnbot) 2011-03-01 10:22:28.000-0600

Repository: asterisk
Revision: 309090

_U  trunk/
U   trunk/channels/chan_sip.c

------------------------------------------------------------------------
r309090 | dvossel | 2011-03-01 10:22:28 -0600 (Tue, 01 Mar 2011) | 22 lines

Merged revisions 309084 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.8

................
 r309084 | dvossel | 2011-03-01 10:09:11 -0600 (Tue, 01 Mar 2011) | 15 lines
 
 Merged revisions 309083 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.6.2
 
 ........
   r309083 | dvossel | 2011-03-01 10:05:25 -0600 (Tue, 01 Mar 2011) | 9 lines
   
   Fixes thread blocking issue in the sip TCP/TLS implementation.
   
   (closes issue ASTERISK-17129)
   Reported by: vois
   Patches:
         issues_18497.diff uploaded by dvossel (license 671)
   Tested by: vois, rossbeer, kowalma, Freddi_Fonet
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=309090