ASTERISK-24754: Asterisk 13 crashes every few minutes and does not generate core

[Home]

Summary: ASTERISK-24754: Asterisk 13 crashes every few minutes and does not generate core

Reporter: Private Name (falves11) Labels:

Date Opened: 2015-02-02 16:28:40.000-0600 Date Closed: 2015-02-06 09:32:21.000-0600

Priority: Critical Regression?

Status: Closed/Complete Components: Channels/chan_pjsip

Versions: 13.1.1 Frequency of
Occurrence

Related
Issues:

Environment: Fedora 20 Attachments:

Description: I get the error below every few minutes, but there is no core generated.
I compiled Asterisk with don't optimize, and better traces.
Asterisk command line is
27193 ? Sl 0:11 /usr/sbin/asterisk -gf

and my ulimits are
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1048576
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

etwork packet size (bytes): 4096
Clock Time (ms.) Total : 3280 Average : (264.0 rows per sec.)
Feb 2 16:45:46 fedora-1 kernel: [ 7321.477271] asterisk[11196]: segfault at aa00a0280a1 ip 00000aa00a0280a1 sp 00007f9fe7644830 error 14
Feb 2 16:46:27 fedora-1 kernel: [ 7362.309150] asterisk[14930]: segfault at 2a2aa0a220a2 ip 00002a2aa0a220a2 sp 00007f7d6bc15250 error 14
Feb 2 16:46:59 fedora-1 kernel: [ 7393.858324] asterisk[15287]: segfault at 554401414011 ip 0000554401414011 sp 00007fe308fab870 error 14
Feb 2 16:49:00 fedora-1 kernel: [ 7514.636854] asterisk[27595]: segfault at 550551501010 ip 0000550551501010 sp 00007f249353a250 error 14
Feb 2 16:55:11 fedora-1 kernel: [ 7886.065473] asterisk[26490]: segfault at 550051511011 ip 0000550051511011 sp 00007f874b89d250 error 14
Feb 2 17:01:00 fedora-1 kernel: [ 8234.490093] asterisk[20864]: segfault at 554541541011 ip 0000554541541011 sp 00007fc631ef1830 error 14
Feb 2 17:08:10 fedora-1 kernel: [ 8664.265411] asterisk[21557]: segfault at 555000100011 ip 0000555000100011 sp 00007f70a1b8e250 error 14
Feb 2 17:08:59 fedora-1 kernel: [ 8712.908734] asterisk[25960]: segfault at 554511355011 ip 0000554511355011 sp 00007f6e8ccfa830 error 14
Feb 2 17:12:10 fedora-1 kernel: [ 8904.312829] asterisk[13835]: segfault at 555054145011 ip 0000555054145011 sp 00007f6e734d1250 error 14
Feb 2 17:13:05 fedora-1 kernel: [ 8959.770402] asterisk[17369]: segfault at 555501500011 ip 0000555501500011 sp 00007fd4a4a98f50 error 14

Comments: By: Matt Jordan (mjordan) 2015-02-02 16:55:47.764-0600

Without a core file and a properly generated backtrace, we aren't going to be able to help you. You'll need to either determine why Asterisk is not generating a core file, or you'll need to attach gdb to the process and capture the backtrace in that fashion.

Given your previous issue - ASTERISK-24753 - where some module is complaining about optimization being disabled while {{FORTIFY_SOURCE}} is enabled, I'd suspect that you have some other build issues at play on your system.
By: Private Name (falves11) 2015-02-03 14:54:51.881-0600

I installed version 13 on brand new machine with Centos 7, not Fedora, and i keep getting the same exact issue. It crashes every few minutes and no trace is being generated. Is there any developer who I can hire via paypal and log into my box and capture the trace ans submit the trace to this case. I need the functionality in version13, application: Transfer, which is not fixed in earlier versions. There was a bug that just got fixed for the app Transfer and PJSIP. Please contact me via venefax aat good old gmail and let's us fix version13. It will take only a few minutes for a developer. I am not a developer, but a business man.

/usr/sbin/safe_asterisk: line 164: 14773 Segmentation fault (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -fg ${CLIARGS} ${ASTARGS} > /dev/null 2>&1 < /dev/null
/usr/sbin/safe_asterisk: line 164: 29544 Segmentation fault (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -fg ${CLIARGS} ${ASTARGS} > /dev/null 2>&1 < /dev/null
/usr/sbin/safe_asterisk: line 164: 9279 Segmentation fault (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -fg ${CLIARGS} ${ASTARGS} > /dev/null 2>&1 < /dev/null
/usr/sbin/safe_asterisk: line 164: 20117 Segmentation fault (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -fg ${CLIARGS} ${ASTARGS} > /dev/null 2>&1 < /dev/null
/usr/sbin/safe_asterisk: line 164: 23780 Segmentation fault (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -fg ${CLIARGS} ${ASTARGS} > /dev/null 2>&1 < /dev/null
/usr/sbin/safe_asterisk: line 164: 1153 Segmentation fault (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -fg ${CLIARGS} ${ASTARGS} > /dev/null 2>&1 < /dev/null

By: Private Name (falves11) 2015-02-04 12:45:31.136-0600

I found out what the issue is. The make file of Asterisk needs to enable fortify_source only when optimization is enabled, or else the compilation issues hundreds of warnings. Everybody is having the same exact issue.
Kindly so a search in Google for
warning _fortify_source requires compiling with optimization
and kindly issue a fix for the issue.
It is impossible right now to compile Asterisk with don't optimize in any if the latest distributions like Centos 7.
After that I will try to get the gdb trace
By: Matt Jordan (mjordan) 2015-02-04 13:44:48.683-0600

{quote}
I found out what the issue is. The make file of Asterisk needs to enable fortify_source only when optimization is enabled, or else the compilation issues hundreds of warnings. Everybody is having the same exact issue.
Kindly so a search in Google for
warning _fortify_source requires compiling with optimization
and kindly issue a fix for the issue.
{quote}

# In its configure scripts, Asterisk will detect to see if the compiler supports FORTIFY_SOURCE and - if it does - it will append it to the to the {{AST_FORTIFY_SOURCE}} flags.
# In the Makefile, we then look to see if {{DONT_OPTIMIZE}} is enabled. If so, we enable {{COMPILE_DOUBLE}} - if not, we append the {{AST_FORTIFY_SOURCE}} flags to the {{_ASTCFLAGS}}.
{code}
# determine whether to double-compile so that the optimizer can report code path problems
# this is only done when developer mode and DONT_OPTIMIZE are both enabled
# in that case, we run the preprocessor to produce a .i or .ii file from the source
# code, then compile once with optimizer enabled (and the output to /dev/null),
# and if that doesn't fail then compile again with optimizer disabled
ifeq ($(findstring DONT_OPTIMIZE,$(MENUSELECT_CFLAGS))$(AST_DEVMODE),DONT_OPTIMIZEyes)
COMPILE_DOUBLE=yes
else
_ASTCFLAGS+=$(AST_FORTIFY_SOURCE)
endif
{code}
That is, we *do not* enable FORTIFY_SOURCE if optimization is turned off.

So, I'm not sure how you're producing this problem, but it is not a problem that I'm able to reproduce. It is also not what this issue is related to.

By: Matt Jordan (mjordan) 2015-02-04 13:45:50.743-0600

{quote}
I installed version 13 on brand new machine with Centos 7, not Fedora, and i keep getting the same exact issue. It crashes every few minutes and no trace is being generated. Is there any developer who I can hire via paypal and log into my box and capture the trace ans submit the trace to this case. I need the functionality in version13, application: Transfer, which is not fixed in earlier versions. There was a bug that just got fixed for the app Transfer and PJSIP. Please contact me via venefax aat good old gmail and let's us fix version13. It will take only a few minutes for a developer. I am not a developer, but a business man.
{quote}

You can find instructions on offering a bounty on a bug here:

https://wiki.asterisk.org/wiki/display/AST/Asterisk+Bug+Bounties
By: Private Name (falves11) 2015-02-04 14:48:27.939-0600

I am using SVN version 13 and the same issue happens in Fedora 20 and Centos7, the later is a brand-new installation

Please contact me and I can give you the credential so you can see with your own eyes
Sample:
{noformat}
[CC] chan_sip.c -> chan_sip.o
In file included from /usr/include/inttypes.h:25:0,
from /usr/src1/asterisk/include/asterisk/compat.h:29,
from /usr/src1/asterisk/include/asterisk.h:27,
from chan_sip.c:222:
/usr/include/features.h:330:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
# warning _FORTIFY_SOURCE requires compiling with optimization (-O)
^
[LD] chan_sip.o sip/security_events.o sip/dialplan_functions.o sip/reqresp_parser.o sip/utils.o sip/config_parser.o sip/route.o -> chan_sip.so
[CC] cdr_mysql.c -> cdr_mysql.o
In file included from /usr/include/inttypes.h:25:0,
from /usr/src1/asterisk/include/asterisk/compat.h:29,
from /usr/src1/asterisk/include/asterisk.h:27,
from cdr_mysql.c:43:
/usr/include/features.h:330:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
# warning _FORTIFY_SOURCE requires compiling with optimization (-O)
^
[LD] cdr_mysql.o -> cdr_mysql.so
[CC] format_mp3.c -> format_mp3.o
In file included from /usr/include/inttypes.h:25:0,
from /usr/src1/asterisk/include/asterisk/compat.h:29,
from /usr/src1/asterisk/include/asterisk.h:27,
from format_mp3.c:35:
/usr/include/features.h:330:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
# warning _FORTIFY_SOURCE requires compiling with optimization (-O)
^
{noformat}

By: Matt Jordan (mjordan) 2015-02-05 08:58:39.186-0600

This is not a support channel. If you need support, I highly suggest you e-mail the [asterisk-biz|http://lists.digium.com/mailman/listinfo/asterisk-biz] mailing list and hire a consultant.

If you cannot provide explicit steps to reproduce the problem, this issue will be closed.

By: Matt Jordan (mjordan) 2015-02-05 13:42:57.111-0600

Update:

I have managed to reproduce this on CentOS 7. The trick is to not {{configure}} with {{--enable-dev-mode}}. As it turns out, {{features.h}} was changed between CentOS 6 and CentOS 7, such that the logic for how it checks for {{FORTIFY_SOURCE}} was modified. This probably was a "problem" in CentOS 6 as well, except that no warning was spit out.

As a workaround, you can configure Asterisk with {{--enable-dev-mode}} and the warning will go away.

Now, that being said, this isn't actually preventing compilation or {{DONT_OPTIMIZE}} from working. While it spammed warnings away, Asterisk still compiled just fine, installed just fine, and ran just fine. Whatever underlying issue you are having, this is not causing it, nor is it preventing the ability to get the requested information.
By: Private Name (falves11) 2015-02-05 14:57:01.507-0600

I followed your instructions and ./configured with --enable-dev-mode
The warnings are gone, but now this
[CPP] res_srtp.c -> res_srtp.i
[CCi] res_srtp.i -> res_srtp.o
res_srtp.c: In function 'res_srtp_shutdown':
res_srtp.c:551:2: error: implicit declaration of function 'srtp_shutdown' [-Werror=implicit-function-declaration]
srtp_shutdown();
^
cc1: all warnings being treated as errors
make[1]: *** [res_srtp.o] Error 1
make: *** [res] Error 2

I did an "svn" up about 5 minutes ago.
Updating '.':
At revision 431574.

By: Matt Jordan (mjordan) 2015-02-05 15:19:46.011-0600

That's not a bug. You've misconfigured PJSIP and Asterisk. See [https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject] for instructions on how to build pjproject - particularly where you have to instruct pjproject that it has an external SRTP library.

I'm not going to turn this issue into an ongoing support request. If you can reproduce the crash and attach the necessary information, then we'll keep this issue open. Until then, if you have further issues, please use the {{asterisk-users}} mailing list.
By: Private Name (falves11) 2015-02-05 20:01:48.086-0600

I have another issue that maybe you can tell me how to troubleshoot
I have an Asterisk 13 that only processes app Transfer with PJSIP, to the tune of 60 per second. No voice calls.
After like 2 hours, I can no longer get into Asterisk. This command, asterisk -r, fails, and also "asterisk -rx core show channels", etc. I am returned to the bash prompt. I checked the handles and

lsof | grep asterisk |wc -l
7098126

I think there is a kind of handle leak here. Nothing else runs in the box
If there is a way to find out what happens, let me know. The dialplan is confidential, for it belongs to my customer,but I can give you access to the box.
In short , the app receives a call, checks the number against a database and calls app_transfer. That is it.

This is what I see when the command fails:

asterisk -r
Asterisk SVN-branch-13-r431555M, Copyright (C) 1999 - 2014, Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
[root@centos7 /]#
this command shows the issue, thousands of lines
lsof | grep asterisk

asterisk 4077 root *450w FIFO 0,8 0t0 110430221 pipe
asterisk 4077 root *451r FIFO 0,8 0t0 110429239 pipe
asterisk 4077 root *452w FIFO 0,8 0t0 110429239 pipe
asterisk 4077 root *453r FIFO 0,8 0t0 110417598 pipe
asterisk 4077 root *454w FIFO 0,8 0t0 110417598 pipe
asterisk 4077 root *455r FIFO 0,8 0t0 110426507 pipe
asterisk 4077 root *456w FIFO 0,8 0t0 110426507 pipe^

It looks like
https://issues.asterisk.org/jira/browse/ASTERISK-823
but in fact I am using PJSIP.

It is definitely PJSIP, for I replaced the dialplan with plain SIP, and there is no issue, ceteris paribus.

By: Matt Jordan (mjordan) 2015-02-06 09:32:02.193-0600

Thanks for your comments. This does not appear to be a bug report and we are closing it. We appreciate the difficulties you are facing, but it would make more sense to raise your question in the support tracker, http://www.asterisk.org/support.

By: Private Name (falves11) 2015-02-06 09:38:54.234-0600

The obvious handle leak is not a bug?
Should I open another bug report or look for support on the 700K handles?
How can support help a low-level leak handle?
I will stop using PJSIP for now.
Thanks for your help.