[Home]

Summary:ASTERISK-24399: Crashes - res_rtp_asterisk with DTLS-SRTP - in libssl
Reporter:Tobias Gunkel (tgunkel)Labels:
Date Opened:2014-10-07 14:36:31Date Closed:2014-11-18 09:47:38.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:Resources/res_rtp_asterisk
Versions:12.4.0 12.5.0 12.5.1 12.6.0 Frequency of
Occurrence		Frequent
Related
Issues:
Environment:Debian Linux jessie/sid, kernel: 3.16.3 preemptive x86_64Attachments:( 0) backtrace_2.txt
( 1) backtrace_3.txt
( 2) backtrace_4.txt
( 3) backtrace_5.txt
( 4) backtrace.txt
( 5) gdb_errors.txt
Description:Hi everyone,

we're using Asterisk with {{res_pjsip}} and since version 12.4.0 it crashes once a day with a segmentation fault:

{code}kernel: [665291.790543] asterisk[2160]: segfault at 7fda5c16cdf0 ip 00007fdb0f7ed4de sp 00007fd9c78d7c18 error 4 in libc-2.19.so[7fdb0f75d000+19f000]{code}

We used {{libpjsip2}} (2.1.0.0.ast20130823-1) from Debian packages and also compiled version 2.2.1, 2.3 and git (master@2014-10-06) from source. This didn't fix the problem.

The issue is not bound to system load. The problem occurs even with 0 open channels.

Any help would be greatly appreciated!

Best regards,
Tobi
Comments:By: Tobias Gunkel (tgunkel) 2014-10-07 14:40:24.488-0500

Attached backtrace. I got some errors while running gdb, so I uploaded them too. Maybe this is relevant.
By: Tobias Gunkel (tgunkel) 2014-10-07 15:01:06.572-0500

Ah, the gdb errors were a result of using a slightly older core dump. Because after every crash we're trying out new libpjsip versions and/or compiler flags (only regarding libpjsip, not asterisk). I uploaded a backtrace without the errors and tomorrow I can also deliver a backtrace where the executable is not newer than the core dump.
By: Tobias Gunkel (tgunkel) 2014-10-08 01:25:31.855-0500

I forgot to mention that this problem was reproduced on three different machines, so it's unlikely a hardware error.
By: Matt Jordan (mjordan) 2014-10-08 07:46:30.724-0500

Thank you for your bug report. In order to move your issue forward, we require a backtrace[1] from the core file produced after the crash. Also, be sure you have DONT_OPTIMIZE enabled in menuselect within the Compiler Flags section, then:

make install

After enabling, reproduce the crash, and then execute the backtrace[1] instructions. When complete, attach that file to this issue report.

[1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace

None of the backtraces on the issue here were generated in such a fashion that the provide the information we need. Please follow the instructions on the wiki and obtain a backtrace from a core file that matches the executable running on the machine.
By: Tobias Gunkel (tgunkel) 2014-10-09 01:16:33.673-0500

I've uploaded {{backtrace_3.txt}}.

{{DONT_OPTIMIZE}} was set (also with the older backtraces I provided):
{code}$ grep OPTIMIZE /usr/src/asterisk-12.6.0/menuselect.makeopts
MENUSELECT_CFLAGS=DONT_OPTIMIZE LOADABLE_MODULES BUILD_NATIVE OPTIONAL_API G711_NEW_ALGORITHM G711_REDUCED_BRANCHING TEST_CODING_TABLES TEST_TANDEM_TRANSCODING{code}
By: Tobias Gunkel (tgunkel) 2014-10-16 07:56:28.461-0500

Another backtrace, now with {{DONT_OPTIMIZE}} and {{BETTER_BACKTRACES}} enabled.
By: Matt Jordan (mjordan) 2014-10-16 09:01:01.968-0500

The crashes appear to be occurring in {{libssl}}:

{noformat}
#0  0x00007f3a48285077 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#0  0x00007f3a48285077 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       resultvar = 0
       pid = 32750
       selftid = 382
#1  0x00007f3a48286458 in __GI_abort () at abort.c:89
       save_stage = 2
       act = {__sigaction_handler = {sa_handler = 0x16, sa_sigaction = 0x16}, sa_mask = {__val = {139886548066032, 8512, 139888301211143, 5, 0, 5, 139888295243048, 1472, 8512, 1, 139888301237029, 128, 139888296104477, 139888299034512, 128, 139882739098720}}, sa_flags = -50751744, sa_restorer = 0x7f3a485f0be0 <_IO_helper_jumps>}
       sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f3a4764278f in OpenSSLDie () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
No symbol table info available.
#3  0x00007f3a479fb603 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
No symbol table info available.
#4  0x00007f3a479f58da in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
No symbol table info available.
#5  0x00007f3901140015 in dtls_perform_handshake (instance=instance@entry=0x7f39e00090e8, dtls=dtls@entry=0x7f39e0010348, rtcp=rtcp@entry=0) at res_rtp_asterisk.c:1593
       rtp = <optimized out>
#6  0x00007f39011400a5 in ast_rtp_on_ice_complete (ice=<optimized out>, status=0) at res_rtp_asterisk.c:1623
       instance = 0x7f39e00090e8
       rtp = 0x7f39e000d630
{noformat}

Which version of {{libssl}} are you using? If possible, can you upgrade to the latest version of {{libssl}}? It would also help if you could install a version of {{libssl}} with debug symbols.

That being said, there isn't anything jumping out here that looks like Asterisk doing something wrong. If we can look at the SSL library's symbols, we may be able to determine what exactly is going wrong - but there's a good chance that this is a bug in that library, and not Asterisk.
By: Tobias Gunkel (tgunkel) 2014-10-16 09:17:24.169-0500

Ah, thanks for the tip.

We have actually libssl 1.0.1i-2 installed, which comes with Debian jessie.
But indeed there seems to be a conflict, because we also have an old version 0.9.8o-4squeeze14 installed, which was not properly uninstalled during the system upgrade.

We'll try to rebuild asterisk after cleaning up this mess :)

Best regards,
Tobias
By: Tobias Gunkel (tgunkel) 2014-10-20 02:11:40.091-0500

New {{backtrace_5.txt}} with libssl debugging symbols enabled. Libssl version used: 1.0.1i-2.
By: Rusty Newton (rnewton) 2014-11-03 16:14:47.171-0600

The crash happens in OpenSSL, Matt Jordan looked over the trace and couldn't find that Asterisk was doing anything wrong in how Asterisk is calling into OpenSSL.. . You will want to file an issue on their tracker using the debug that you have shown here.

Once you file an issue on their tracker, you can post a link here for the sake of continuity.


By: Rusty Newton (rnewton) 2014-11-18 09:47:38.173-0600

Closing this out since we don't believe this is a bug in Asterisk.

[~tgunkel] please post a link to the bug you file with OpenSSL when you get a chance. That will be helpful to others who may find this issue later on.