[Home]

Summary:ASTERISK-20394: Asterisk crashes in ast_generic_bridge()/ast_frame_free()
Reporter:Jon Christopherson (m00ster)Labels:
Date Opened:2012-09-08 12:02:12Date Closed:2017-12-12 19:38:28.000-0600
Priority:CriticalRegression?
Status:Closed/CompleteComponents:Core/General
Versions:1.8.15.1 1.8.16.0 Frequency of
Occurrence
Related
Issues:
Environment:CentOS 6.2 64-bit on a DELL poweredge R710 with 2x six core Xeon CPU's with 32G RAMAttachments:( 0) asterisk-backtrace.txt
( 1) asterisk-backtrace.txt
( 2) asterisk-backtrace.txt
( 3) asterisk-core-show-locks.txt
( 4) dt.txt
( 5) valgrind.txt
Description:when under load asterisk has often crashed with this type of trace. ~150 channels active, and about 90 calls being handled.

ast_generic_bridge()/ast_frame_free()
Comments:By: Jon Christopherson (m00ster) 2012-09-08 12:04:43.745-0500

Backtrace
By: Jon Christopherson (m00ster) 2012-09-08 12:07:44.203-0500

Asterisk often is using ~12.6G virtual ram and 100M res like this:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25124 asterisk  20   0 12.6g 100m  12m S 17.7  0.3  24:41.17 asterisk

Version details:

FreePBX Version          = 2.9.0.12
Running Asterisk Version = 1.8.15.1
Asterisk Source Version  = 1.8.15.1
Dahdi Source Version     = 2.6.1+2.6.1
Libpri Source Version    = 1.4.12    
Operating System         = CentOS release 6.2 (Final)
Kernel Version           = 2.6.32-220.23.1.el6.x86_64 - 64 Bit
By: Matt Jordan (mjordan) 2012-09-10 07:46:57.333-0500

Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks!


By: Jon Christopherson (m00ster) 2012-09-10 10:00:33.233-0500

I have attached a bit of valgrind output, i wasnt able to run it under valgrind for long due to performance issues when running under the debugger, asterisk performed too slowly to keep up with the load, I do still have it compiled with the debug options, so it it shows signs of sip deadlocks or crashes again ill attach those.
By: Jon Christopherson (m00ster) 2012-09-10 11:30:56.482-0500

Hello,

I uploaded a small valgrind output, I wasn't able to run asterisk under valgrind for lng due to performance issues. It slowed asterisk down to the point it couldn't keep up with call volume.
By: Jon Christopherson (m00ster) 2012-09-10 13:51:54.570-0500

Just had asterisk stop responding and hog cpu cycles again, if it doesn't crash it will do this instead and just stop processing calls.
By: Richard Mudgett (rmudgett) 2012-09-10 17:43:50.584-0500

Strange, the core show locks file shows three threads waiting for a lock that no one seems to have locked.
By: Jon Christopherson (m00ster) 2012-09-15 17:01:29.504-0500

I am still getting crashes on 1.8.16.0, still pointing to memory corruption. I tried running with valgrind, but the performance hit made it unusable. I did attach what little output i got to this report earlier.Here is the latest core dump from today when asterisk got busy.


asterisk[1848]: segfault at 7f8f0000001a ip 00000030a1279115 sp 00007f90eac9b720 error 4 in libc-2.12.so[30a1200000+189000]


By: Jon Christopherson (m00ster) 2012-09-15 18:52:35.778-0500

Another core a couple hours later, looks diffrent to me. the timing for these cores is odd.

asterisk[5018] general protection ip:30a12478de sp:7f488043a070 error:0 in libc-2.12.so[30a1200000+189000]


By: Matt Jordan (mjordan) 2012-09-17 08:30:39.831-0500

In each of the backtraces you've uploaded, at least one thread has been in multiple nested macros.  The usage of macros is deprecated for a reason - nesting macro calls can cause Asterisk to crash in interesting ways.  While the hard limit of 7 is noted on the [wiki|https://wiki.asterisk.org/wiki/display/AST/Application_Macro], it is also possible to hit a limit much sooner than that given the appropriate conditions (such as having a stack intensive application execute).

If you replace the nested macros with GoSubs, does this problem still occur?
By: Jon Christopherson (m00ster) 2012-09-17 08:50:58.215-0500

Hello,

We pretty much use the dialplan that ships with freepbx 2.9, the only changes we made to it involve placing recordings in a diffrent directory structure. I do see macros used alot on the dialplan, but not sure if i see alot of nested macros.
By: Corey Farrell (coreyfarrell) 2017-12-12 19:38:28.702-0600

Bridging has been rewritten since 1.8 and the {{ast_generic_bridge}} function no longer exists.

----

Per the Asterisk versions page \[1\], the maintenance (bug fix) support for the Asterisk branch you are using has ended. For continued maintenance support please move to a supported branch of Asterisk. After testing with a supported branch, if you find this problem has not been resolved, please open a new issue against the latest version of that Asterisk branch.

Thanks!

\[1\] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions