[Home]

Summary:ASTERISK-28050: Asterisk crash with Abort error
Reporter:Abhay Gupta (agupta)Labels:pjsip webrtc
Date Opened:2018-09-12 02:37:22Date Closed:2020-01-14 11:13:28.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:Core/Stasis
Versions:15.5.0 Frequency of
Occurrence		Occasional
Related
Issues:
Environment:ubuntu 16.04 with Asterisk 15.5 using stasis for call bridging Attachments:( 0) gdb.txt
Description:Asterisk has crash generating a core dump

#0  0x00007f8da0c95428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f8da0c9702a in __GI_abort () at abort.c:89
#2  0x00007f8da0cd77ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f8da0df0ed8 "*** Error in `%s': %s: 0x%s ***\n")
   at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f8da0ce037a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7f8da0dedcaf "free(): invalid pointer", action=3) at malloc.c:5006
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3867
#5  0x00007f8da0ce453c in __GI___libc_free (mem=mem@entry=0x7f8bc8198ae0) at malloc.c:2968
#6  0x000000000045bffd in __ao2_ref (user_data=0x7f8bc8198b08, delta=-1, tag=0x61bdc0 "", file=0x66a189 "stasis_message.c", line=105,
   func=0x66a1a0 <__PRETTY_FUNCTION__.10549> "stasis_message_dtor") at astobj2.c:607
#7  0x000000000045bd63 in __ao2_ref (user_data=user_data@entry=0x7f8bc8197c78, delta=delta@entry=-1, tag=tag@entry=0x61bdc0 "", file=file@entry=0x667cd7 "stasis.c",
   line=line@entry=732, func=func@entry=0x6683b0 <__PRETTY_FUNCTION__.16959> "dispatch_exec_async") at astobj2.c:576
#8  0x000000000045c09b in __ao2_cleanup_debug (obj=obj@entry=0x7f8bc8197c78, tag=tag@entry=0x61bdc0 "", file=file@entry=0x667cd7 "stasis.c", line=line@entry=732,
   function=function@entry=0x6683b0 <__PRETTY_FUNCTION__.16959> "dispatch_exec_async") at astobj2.c:627
#9  0x00000000005d971d in dispatch_exec_async (local=<optimized out>) at stasis.c:732
#10 0x00000000005f6bde in ast_taskprocessor_execute (tps=tps@entry=0x1866538) at taskprocessor.c:968
#11 0x00000000005f6c90 in default_tps_processing_function (data=data@entry=0x1865fd8) at taskprocessor.c:189
#12 0x000000000060932d in dummy_start (data=<optimized out>) at utils.c:1258
#13 0x00007f8da178d6ba in start_thread (arg=0x7f8d9d74a700) at pthread_create.c:333
#14 0x00007f8da0d6741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Comments:By: Asterisk Team (asteriskteam) 2018-09-12 02:37:24.493-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Abhay Gupta (agupta) 2018-09-12 02:38:08.861-0500

File with bt , bt full and thread apply all bt
By: Richard Mudgett (rmudgett) 2018-09-12 07:30:14.023-0500

Your backtrace appears to contain a memory corruption. We need one or both of the following items to continue investigation of the issue:
1. Valgrind output. See https://wiki.asterisk.org/wiki/display/AST/Valgrind for instructions on how to use Valgrind with Asterisk.
2. MALLOC_DEBUG output. See https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag for instructions on how to use the MALLOC_DEBUG option.

Note that MALLOC_DEBUG and Valgrind are mutually exclusive options. Valgrind output is preferable, but will be more system resource intensive and may be difficult to get on a production system. In such a case, you may have better luck getting the necessary output from MALLOC_DEBUG.


By: Abhay Gupta (agupta) 2018-09-13 02:27:06.441-0500

It is a heavily loaded server will see if MALLOC_DEBUG load can be handled by the server .
By: Joshua C. Colp (jcolp) 2018-09-13 04:06:50.905-0500

Information about what the system is used for, load, etc, would also be useful in addition to what Richard asked for.
By: Abhay Gupta (agupta) 2018-09-14 01:32:56.445-0500

System is used for incoming calls on PJSIP and lands on Agents who use webrtc using statis/ari for call bridge . Load normally hovers at around 2-3 .

Since we are using ARI and i saw on this issue tracker where it was suggested to use 15.6 with libjansson bundled as earlier version has some thread safety issue , i plan to upgrade to 15.6 with bundled libjansson and see if that is causing the memory issue else will try MALLOC_DEBUG if problem reappears .

By: Asterisk Team (asteriskteam) 2018-09-28 12:00:01.731-0500

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines