[Home]

Summary:ASTERISK-24333: Crash in DTLS
Reporter:Badalian Vyacheslav (slavon)Labels:
Date Opened:2014-09-16 15:45:00Date Closed:2014-10-01 15:21:45
Priority:CriticalRegression?
Status:Closed/CompleteComponents:Channels/chan_sip/WebSocket
Versions:11.12.0 Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) ASTERISK-24333-11.diff
( 1) valgrind.txt
( 2) valgrind2.txt
( 3) valgrind3.txt
( 4) valgrind4.txt
Description:Asterisk. DTLS. Only WSS. 300+ Clients

CentOS 6. All last updates

m1-asterisk01*CLI> *** glibc detected *** /usr/sbin/asterisk: double free or corruption (!prev): 0x00007f95f40854a0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e76)[0x7f968725fe76]
/lib64/libc.so.6(+0x789b3)[0x7f96872629b3]
/usr/lib64/libcrypto.so.10(CRYPTO_realloc_clean+0xf3)[0x7f968662fde3]
/usr/lib64/libcrypto.so.10(BUF_MEM_grow_clean+0x86)[0x7f968669f996]
/usr/lib64/libcrypto.so.10(+0xda3f3)[0x7f96866a13f3]
/usr/lib64/libcrypto.so.10(BIO_write+0x77)[0x7f96866a05f7]
/usr/lib64/libcrypto.so.10(+0xdc621)[0x7f96866a3621]
/usr/lib64/libssl.so.10(dtls1_do_write+0x18d)[0x7f96869e58ad]
/usr/lib64/libssl.so.10(dtls1_accept+0xaa7)[0x7f96869dfc77]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xf6d5)[0x7f964048c6d5]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xf7a6)[0x7f964048c7a6]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x199d1)[0x7f96404969d1]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x48792)[0x7f96404c5792]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xad46)[0x7f9640487d46]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x39cbf)[0x7f96404b6cbf]
/lib64/libpthread.so.0(+0x79d1)[0x7f96861ad9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f96872d286d]




Comments:By: Matt Jordan (mjordan) 2014-09-17 16:44:21.244-0500

Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks!


By: Badalian Vyacheslav (slavon) 2014-09-22 23:52:59.467-0500

Done!
By: Badalian Vyacheslav (slavon) 2014-09-24 05:29:06.259-0500

any updates?
By: Matt Jordan (mjordan) 2014-09-25 11:25:59.462-0500

So, there's a whole mess of uninitialized values getting used by {{libssl}}. There's not much we can do about that.

However, valgrind did complain about an invalid read (off by one) when reading from the master key buffer allocated in {{res_srtp}}:

{noformat}
==23717== Thread 88:
==23717== Invalid read of size 1
==23717==    at 0xA794F00: v128_copy_octet_string (in /usr/lib64/libsrtp.so.0.0.0)
==23717==    by 0xA792898: aes_icm_context_init (in /usr/lib64/libsrtp.so.0.0.0)
==23717==    by 0xA78CFFC: srtp_kdf_init (in /usr/lib64/libsrtp.so.0.0.0)
==23717==    by 0xA78D089: srtp_stream_init_keys (in /usr/lib64/libsrtp.so.0.0.0)
==23717==    by 0xA78D59E: srtp_stream_init (in /usr/lib64/libsrtp.so.0.0.0)
==23717==    by 0xA78ED8C: srtp_add_stream (in /usr/lib64/libsrtp.so.0.0.0)
==23717==    by 0xA78EE93: srtp_create (in /usr/lib64/libsrtp.so.0.0.0)
==23717==    by 0xA5853A8: ast_srtp_create (res_srtp.c:450)
==23717==    by 0x558421: ast_rtp_instance_add_srtp_policy (rtp_engine.c:2072)
==23717==    by 0xF566BA7: dtls_srtp_setup (res_rtp_asterisk.c:1661)
==23717==    by 0xF566FEC: __rtp_recvfrom (res_rtp_asterisk.c:1739)
==23717==    by 0xF5672A9: rtp_recvfrom (res_rtp_asterisk.c:1789)
==23717==  Address 0x29b20f2e is 0 bytes after a block of size 30 alloc'd
==23717==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
==23717==    by 0x596BC8: _ast_calloc (utils.h:514)
==23717==    by 0xA584B04: ast_srtp_policy_set_master_key (res_srtp.c:293)
==23717==    by 0xF566ABD: dtls_srtp_setup (res_rtp_asterisk.c:1649)
==23717==    by 0xF566FEC: __rtp_recvfrom (res_rtp_asterisk.c:1739)
==23717==    by 0xF5672A9: rtp_recvfrom (res_rtp_asterisk.c:1789)
==23717==    by 0xF5706B2: ast_rtp_read (res_rtp_asterisk.c:3862)
==23717==    by 0x551286: ast_rtp_instance_read (rtp_engine.c:314)
==23717==    by 0xC43C641: sip_rtp_read (chan_sip.c:8194)
==23717==    by 0xC43CDF0: sip_read (chan_sip.c:8291)
==23717==    by 0x47C91C: __ast_read (channel.c:4054)
==23717==    by 0x47E6C5: ast_read (channel.c:4408)
{noformat}

That's a little odd, since the length of the {{unsigned char}} buffer passed to that function shouldn't be NULL terminated or have any other requirements that necessitate an additional byte of space. This may actually be a bug in {{libsrtp}} - see [http://sourceforge.net/p/srtp/bugs/7/]. It certainly appears like {{aes_icm_context_init}} is reading past the key. There's a few other bug reports from browser vendors regarding this same function; they may be related.

I've attached something that _may_ fix this - it simply allocates one additional byte of space in {{res_srtp}}. That should be harmless for SDES-SRTP or DTLS, as the actual key length would be unaffected.

As this does appear to be a bug in {{libsrtp}}, if this doesn't work around the issue, you may have to push the issue to that project.
By: Rusty Newton (rnewton) 2014-09-25 13:00:37.072-0500

[~slavon] let us know when you've tested the potential work-around. I'll set the issue to waiting on feedback. Thanks!
By: Badalian Vyacheslav (slavon) 2014-09-29 23:24:36.179-0500

Hello! Does not help....But valgrind log look more clear...

Crash 1:
{code}
0x0000003be3c89902 in memcpy () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003be3c89902 in memcpy () from /lib64/libc.so.6
#1  0x0000003bea43bff7 in do_dtls1_write () from /usr/lib64/libssl.so.10
#2  0x0000003bea43e80a in dtls1_do_write () from /usr/lib64/libssl.so.10
#3  0x0000003bea438c77 in dtls1_accept () from /usr/lib64/libssl.so.10
#4  0x0000003bea43d4ad in dtls1_read_bytes () from /usr/lib64/libssl.so.10
#5  0x0000003bea427d50 in ?? () from /usr/lib64/libssl.so.10
#6  0x00007ffd42b2cef4 in __rtp_recvfrom (instance=0x7ffcc4003a58, buf=0x7ffd28bd9c90, size=8192, flags=0, sa=0x7ffd28bdbc90, rtcp=1) at res_rtp_asterisk.c:1723
#7  0x00007ffd42b2d260 in rtcp_recvfrom (instance=0x7ffcc4003a58, buf=0x7ffd28bd9c90, size=8192, flags=0, sa=0x7ffd28bdbc90) at res_rtp_asterisk.c:1784
#8  0x00007ffd42b34333 in ast_rtcp_read (instance=0x7ffcc4003a58) at res_rtp_asterisk.c:3466
#9  0x00007ffd42b36645 in ast_rtp_read (instance=0x7ffcc4003a58, rtcp=1) at res_rtp_asterisk.c:3851
#10 0x0000000000551287 in ast_rtp_instance_read (instance=0x7ffcc4003a58, rtcp=1) at rtp_engine.c:314
#11 0x00007ffd79c18663 in sip_rtp_read (ast=0x7ffcc4016b78, p=0x7ffcc4034688, faxdetect=0x7ffd28bdc604) at chan_sip.c:8197
#12 0x00007ffd79c18df1 in sip_read (ast=0x7ffcc4016b78) at chan_sip.c:8291
#13 0x000000000047c91d in __ast_read (chan=0x7ffcc4016b78, dropaudio=0) at channel.c:4054
#14 0x000000000047e6c6 in ast_read (chan=0x7ffcc4016b78) at channel.c:4408
#15 0x00007ffd2e441a1a in wait_for_answer (in=0x7ffcc4016b78, out_chans=0x7ffd28bde950, to=0x7ffd28bde94c, peerflags=0x7ffd28bdeeb0, opt_args=0x7ffd28bde190, pa=0x7ffd28bde270, num_in=0x7ffd28bde930, result=0x7ffd28bde26c,
   dtmf_progress=0x0, ignore_cc=1, forced_clid=0x7ffd28bde040, stored_clid=0x7ffd28bddff0) at app_dial.c:1562
#16 0x00007ffd2e446f70 in dial_exec_full (chan=0x7ffcc4016b78, data=0x7ffd28be1110 "SIP/avaya/989090054050,300,Tt", peerflags=0x7ffd28bdeeb0, continue_exec=0x0) at app_dial.c:2683
#17 0x00007ffd2e449789 in dial_exec (chan=0x7ffcc4016b78, data=0x7ffd28be1110 "SIP/avaya/989090054050,300,Tt") at app_dial.c:3130
#18 0x000000000052b0e1 in pbx_exec (c=0x7ffcc4016b78, app=0x26a07f0, data=0x7ffd28be1110 "SIP/avaya/989090054050,300,Tt") at pbx.c:1622
#19 0x0000000000535afe in pbx_extension_helper (c=0x7ffcc4016b78, con=0x0, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=22, label=0x0, callerid=0x7ffc8001e960 "84996051913", action=E_SPAWN,
   found=0x7ffd28be378c, combined_find_spawn=1) at pbx.c:4915
#20 0x0000000000538f99 in ast_spawn_extension (c=0x7ffcc4016b78, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=22, callerid=0x7ffc8001e960 "84996051913", found=0x7ffd28be378c, combined_find_spawn=1)
   at pbx.c:6037
#21 0x00007ffd2d4180b0 in _macro_exec (chan=0x7ffcc4016b78, data=0x7ffd28be6490 "dialout-trunk,2,989090054050,,off", exclusive=0) at app_macro.c:412
#22 0x00007ffd2d419342 in macro_exec (chan=0x7ffcc4016b78, data=0x7ffd28be6490 "dialout-trunk,2,989090054050,,off") at app_macro.c:585
#23 0x000000000052b0e1 in pbx_exec (c=0x7ffcc4016b78, app=0x269b180, data=0x7ffd28be6490 "dialout-trunk,2,989090054050,,off") at pbx.c:1622
#24 0x0000000000535afe in pbx_extension_helper (c=0x7ffcc4016b78, con=0x0, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=5, label=0x0, callerid=0x7ffc8001e960 "84996051913", action=E_SPAWN,
   found=0x7ffd28be8b70, combined_find_spawn=1) at pbx.c:4915
#25 0x0000000000538f99 in ast_spawn_extension (c=0x7ffcc4016b78, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=5, callerid=0x7ffc8001e960 "84996051913", found=0x7ffd28be8b70, combined_find_spawn=1)
   at pbx.c:6037
#26 0x000000000053a736 in __ast_pbx_run (c=0x7ffcc4016b78, args=0x0) at pbx.c:6512
#27 0x000000000053c213 in pbx_thread (data=0x7ffcc4016b78) at pbx.c:6842
#28 0x0000000000598580 in dummy_start (data=0x7ffcc402c2a0) at utils.c:1169
#29 0x0000003be44079d1 in start_thread () from /lib64/libpthread.so.0
#30 0x0000003be3ce886d in clone () from /lib64/libc.so.6
{code}

Crash 2:
run in 2 console. Its ONE crash bug in gdb and system baktrace print:

{code}
======= Backtrace: =========
/lib64/libc.so.6[0x3be3c75e76]
/lib64/libc.so.6[0x3be3c789b3]
/usr/lib64/libcrypto.so.10(CRYPTO_realloc_clean+0xf3)[0x3be6868de3]
/usr/lib64/libcrypto.so.10(BUF_MEM_grow_clean+0x86)[0x3be68d8996]
/usr/lib64/libcrypto.so.10[0x3be68da3f3]
/usr/lib64/libcrypto.so.10(BIO_write+0x77)[0x3be68d95f7]
/usr/lib64/libcrypto.so.10[0x3be68dc621]
/usr/lib64/libssl.so.10(dtls1_do_write+0x18d)[0x3bea43e8ad]
/usr/lib64/libssl.so.10(dtls1_accept+0xaa7)[0x3bea438c77]
/usr/lib64/libssl.so.10(dtls1_read_bytes+0x6ed)[0x3bea43d4ad]
/usr/lib64/libssl.so.10[0x3bea427d50]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xbef4)[0x7f82cf76cef4]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xc260)[0x7f82cf76d260]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x13333)[0x7f82cf774333]
/usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x15645)[0x7f82cf776645]
/usr/sbin/asterisk(ast_rtp_instance_read+0x2b)[0x551287]
/usr/lib/asterisk/modules/chan_sip.so(+0x2d663)[0x7f82d68cb663]
/usr/lib/asterisk/modules/chan_sip.so(+0x2ddf1)[0x7f82d68cbdf1]
/usr/sbin/asterisk[0x47c91d]
/usr/sbin/asterisk(ast_read+0x1d)[0x47e6c6]
/usr/lib/asterisk/modules/app_dial.so(+0x8a1a)[0x7f82bf15ea1a]
/usr/lib/asterisk/modules/app_dial.so(+0xdf70)[0x7f82bf163f70]
/usr/lib/asterisk/modules/app_dial.so(+0x10789)[0x7f82bf166789]
/usr/sbin/asterisk(pbx_exec+0x214)[0x52b0e1]
/usr/sbin/asterisk[0x535afe]
/usr/sbin/asterisk(ast_spawn_extension+0x65)[0x538f99]
/usr/lib/asterisk/modules/app_macro.so(+0x30b0)[0x7f82be1350b0]
/usr/lib/asterisk/modules/app_macro.so(+0x4342)[0x7f82be136342]
/usr/sbin/asterisk(pbx_exec+0x214)[0x52b0e1]
/usr/sbin/asterisk[0x535afe]
/usr/sbin/asterisk(ast_spawn_extension+0x65)[0x538f99]
/usr/sbin/asterisk[0x53a736]
/usr/sbin/asterisk[0x53c213]
/usr/sbin/asterisk[0x598580]
/lib64/libpthread.so.0[0x3be44079d1]
/lib64/libc.so.6(clone+0x6d)[0x3be3ce886d]

{code}

{code}
0x0000003be3c32635 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003be3c32635 in raise () from /lib64/libc.so.6
#1  0x0000003be3c33e15 in abort () from /lib64/libc.so.6
#2  0x0000003be3c70547 in __libc_message () from /lib64/libc.so.6
#3  0x0000003be3c75e76 in malloc_printerr () from /lib64/libc.so.6
#4  0x0000003be3c789b3 in _int_free () from /lib64/libc.so.6
#5  0x0000003be6868de3 in CRYPTO_realloc_clean () from /usr/lib64/libcrypto.so.10
#6  0x0000003be68d8996 in BUF_MEM_grow_clean () from /usr/lib64/libcrypto.so.10
#7  0x0000003be68da3f3 in ?? () from /usr/lib64/libcrypto.so.10
#8  0x0000003be68d95f7 in BIO_write () from /usr/lib64/libcrypto.so.10
#9  0x0000003be68dc621 in ?? () from /usr/lib64/libcrypto.so.10
#10 0x0000003bea43e8ad in dtls1_do_write () from /usr/lib64/libssl.so.10
#11 0x0000003bea438c77 in dtls1_accept () from /usr/lib64/libssl.so.10
#12 0x0000003bea43d4ad in dtls1_read_bytes () from /usr/lib64/libssl.so.10
#13 0x0000003bea427d50 in ?? () from /usr/lib64/libssl.so.10
#14 0x00007f82cf76cef4 in __rtp_recvfrom (instance=0x7f82e8031e98, buf=0x7f82b88fac90, size=8192, flags=0, sa=0x7f82b88fcc90, rtcp=1) at res_rtp_asterisk.c:1723
#15 0x00007f82cf76d260 in rtcp_recvfrom (instance=0x7f82e8031e98, buf=0x7f82b88fac90, size=8192, flags=0, sa=0x7f82b88fcc90) at res_rtp_asterisk.c:1784
#16 0x00007f82cf774333 in ast_rtcp_read (instance=0x7f82e8031e98) at res_rtp_asterisk.c:3466
#17 0x00007f82cf776645 in ast_rtp_read (instance=0x7f82e8031e98, rtcp=1) at res_rtp_asterisk.c:3851
#18 0x0000000000551287 in ast_rtp_instance_read (instance=0x7f82e8031e98, rtcp=1) at rtp_engine.c:314
#19 0x00007f82d68cb663 in sip_rtp_read (ast=0x7f82e8049668, p=0x7f82e8026488, faxdetect=0x7f82b88fd604) at chan_sip.c:8197
#20 0x00007f82d68cbdf1 in sip_read (ast=0x7f82e8049668) at chan_sip.c:8291
#21 0x000000000047c91d in __ast_read (chan=0x7f82e8049668, dropaudio=0) at channel.c:4054
#22 0x000000000047e6c6 in ast_read (chan=0x7f82e8049668) at channel.c:4408
#23 0x00007f82bf15ea1a in wait_for_answer (in=0x7f82e8049668, out_chans=0x7f82b88ff950, to=0x7f82b88ff94c, peerflags=0x7f82b88ffeb0, opt_args=0x7f82b88ff190, pa=0x7f82b88ff270, num_in=0x7f82b88ff930, result=0x7f82b88ff26c,
   dtmf_progress=0x0, ignore_cc=1, forced_clid=0x7f82b88ff040, stored_clid=0x7f82b88feff0) at app_dial.c:1562
#24 0x00007f82bf163f70 in dial_exec_full (chan=0x7f82e8049668, data=0x7f82b8902110 "SIP/avaya/989123032784,300,Tt", peerflags=0x7f82b88ffeb0, continue_exec=0x0) at app_dial.c:2683
#25 0x00007f82bf166789 in dial_exec (chan=0x7f82e8049668, data=0x7f82b8902110 "SIP/avaya/989123032784,300,Tt") at app_dial.c:3130
#26 0x000000000052b0e1 in pbx_exec (c=0x7f82e8049668, app=0x1e146d0, data=0x7f82b8902110 "SIP/avaya/989123032784,300,Tt") at pbx.c:1622
#27 0x0000000000535afe in pbx_extension_helper (c=0x7f82e8049668, con=0x0, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=22, label=0x0, callerid=0x7f826c001ec0 "84996051913", action=E_SPAWN,
   found=0x7f82b890478c, combined_find_spawn=1) at pbx.c:4915
#28 0x0000000000538f99 in ast_spawn_extension (c=0x7f82e8049668, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=22, callerid=0x7f826c001ec0 "84996051913", found=0x7f82b890478c, combined_find_spawn=1)
   at pbx.c:6037
#29 0x00007f82be1350b0 in _macro_exec (chan=0x7f82e8049668, data=0x7f82b8907490 "dialout-trunk,2,989123032784,,off", exclusive=0) at app_macro.c:412
#30 0x00007f82be136342 in macro_exec (chan=0x7f82e8049668, data=0x7f82b8907490 "dialout-trunk,2,989123032784,,off") at app_macro.c:585
#31 0x000000000052b0e1 in pbx_exec (c=0x7f82e8049668, app=0x1e0e5e0, data=0x7f82b8907490 "dialout-trunk,2,989123032784,,off") at pbx.c:1622
#32 0x0000000000535afe in pbx_extension_helper (c=0x7f82e8049668, con=0x0, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=5, label=0x0, callerid=0x7f826c001ec0 "84996051913", action=E_SPAWN,
   found=0x7f82b8909b70, combined_find_spawn=1) at pbx.c:4915
#33 0x0000000000538f99 in ast_spawn_extension (c=0x7f82e8049668, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=5, callerid=0x7f826c001ec0 "84996051913", found=0x7f82b8909b70, combined_find_spawn=1)
   at pbx.c:6037
#34 0x000000000053a736 in __ast_pbx_run (c=0x7f82e8049668, args=0x0) at pbx.c:6512
#35 0x000000000053c213 in pbx_thread (data=0x7f82e8049668) at pbx.c:6842
#36 0x0000000000598580 in dummy_start (data=0x7f82e803e9f0) at utils.c:1169
#37 0x0000003be44079d1 in start_thread () from /lib64/libpthread.so.0
#38 0x0000003be3ce886d in clone () from /lib64/libc.so.6
{code}
By: Badalian Vyacheslav (slavon) 2014-09-29 23:41:49.418-0500

Also 2 runs in Valgrind
By: Rusty Newton (rnewton) 2014-10-01 15:21:28.038-0500

Thanks for the additional information. [~kmoore] reviewed the debug, finding that the issue still appears to be a bug in libsrtp and not in Asterisk.

I'm going to close this issue out since there isn't anything we can do at this point. If you file a new issue with the libsrtp project then please link the issue in a comment here.
By: Badalian Vyacheslav (slavon) 2014-10-01 15:25:15.691-0500

Sorry. We have 1-3 Crash in WS mode every day.
Glibc say for random errors like:

asterisk[3328]: segfault at d9c36b47930 ip 0000003be3c47e3c sp 00007ffd16531410 error 4 in libc-2.12.so[3be3c00000+18a000]
asterisk[47139] general protection ip:3be3c47e3c sp:7f360d06d410 error:0 in libc-2.12.so[3be3c00000+18a000]
Also have errors for double free.
WS mode also use SRTP?
By: Badalian Vyacheslav (slavon) 2014-10-01 15:57:11.370-0500

Also last update of libSRTP is 2007-05-04. Maybe need to change library? SRTP and other secure now in priority,

Is update to asterisk 12.x will fix problem? You have another sip library in 12.x...