[Home]

Summary:ASTERISK-24131: DTLS Crash Out of libssl
Reporter:JoshE (n8ideas)Labels:
Date Opened:2014-07-29 03:39:50Date Closed:2014-09-04 20:14:37
Priority:MajorRegression?
Status:Closed/CompleteComponents:Resources/res_rtp_asterisk
Versions:11.11.0 Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) dtls_crash.txt
( 1) full-dtls-crash-openssl.txt
Description:I am seeing a fairly consistent crash on 11.11 servers with relatively heavy WebRTC use via DTLS.  The top of the stack always appears in libssl, but in slightly different places. dtls1_send_server_hello or SSL_clear.

Haven't had a chance to dig into this yet, but curious if anyone else is seeing this.
Comments:By: JoshE (n8ideas) 2014-07-29 03:40:19.405-0500

Attaching the most recent stack trace.

By: Matt Jordan (mjordan) 2014-07-30 09:53:52.311-0500

I took a look at the backtrace, and the crash is originating from a call to {{SSL_read}} in {{res_rtp_asterisk}}:

{code}
BIO_write(dtls->read_bio, buf, len);

len = SSL_read(dtls->ssl, buf, len);

if ((len < 0) && (SSL_get_error(dtls->ssl, len) == SSL_ERROR_SSL)) {
{code}

Looking at your backtrace, the pointers/variables look fine prior to the call to {{SSL_read}}.

For us to have any additional information, you'll need to install a version of OpenSSL with debug symbols and get a backtrace that shows the symbols within OpenSSL. Even with that information, this appears to be a bug in OpenSSL, not Asterisk.

As it is, I'm not sure what else we can do here.

By: Matt Jordan (mjordan) 2014-07-30 09:54:37.557-0500

Also: Please add what version of OpenSSL you are using to this issue. Others who come across may be able to avoid a similar situation with that information.

By: JoshE (n8ideas) 2014-07-30 09:59:18.008-0500

I am running OpenSSL 1.0.1e-fips 11 Feb 2013.  This is the default version in patched CentOS 6 and 7 installations.

Also, as a further note, this bug seems to reproduce when you have multiple WebRTC clients sitting in an incoming queue with Ring All set on it.  Something with the concurrent incoming calls to multiple WebRTC clients seems to cause the issue relatively reliably.

By: SirLouen (sirlouen) 2014-08-04 04:32:43.478-0500

# OpenSSL 1.0.1 14 Mar 2012
Here, same happening

By: JoshE (n8ideas) 2014-08-06 14:20:12.265-0500

I manually updated this to 1.0.1h and recompiled with symbols.  Full backtrace is attached.  Also going to work on the openssl angle on this one.

By: Rusty Newton (rnewton) 2014-08-21 14:24:31.853-0500

[~n8ideas] Are you going to file the bug on http://rt.openssl.org/ , if so, can you link the issue here?

By: JoshE (n8ideas) 2014-08-21 14:44:12.464-0500

Issue is open with OpenSSL here: http://rt.openssl.org/Ticket/Display.html?id=3487

By: Rusty Newton (rnewton) 2014-09-04 20:14:25.798-0500

Thanks JoshE. Closing this out since it doesn't appear we can do anything further. If you collect new debug that shows an issue in Asterisk then please contact a bug marshal in #asterisk-bugs on the IRC network irc.freenode.net to have them reopen the issue.