Summary: | ASTERISK-24333: Crash in DTLS | ||
Reporter: | Badalian Vyacheslav (slavon) | Labels: | |
Date Opened: | 2014-09-16 15:45:00 | Date Closed: | 2014-10-01 15:21:45 |
Priority: | Critical | Regression? | |
Status: | Closed/Complete | Components: | Channels/chan_sip/WebSocket |
Versions: | 11.12.0 | Frequency of Occurrence | |
Related Issues: | |||
Environment: | Attachments: | ( 0) ASTERISK-24333-11.diff ( 1) valgrind.txt ( 2) valgrind2.txt ( 3) valgrind3.txt ( 4) valgrind4.txt | |
Description: | Asterisk. DTLS. Only WSS. 300+ Clients
CentOS 6. All last updates m1-asterisk01*CLI> *** glibc detected *** /usr/sbin/asterisk: double free or corruption (!prev): 0x00007f95f40854a0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x75e76)[0x7f968725fe76] /lib64/libc.so.6(+0x789b3)[0x7f96872629b3] /usr/lib64/libcrypto.so.10(CRYPTO_realloc_clean+0xf3)[0x7f968662fde3] /usr/lib64/libcrypto.so.10(BUF_MEM_grow_clean+0x86)[0x7f968669f996] /usr/lib64/libcrypto.so.10(+0xda3f3)[0x7f96866a13f3] /usr/lib64/libcrypto.so.10(BIO_write+0x77)[0x7f96866a05f7] /usr/lib64/libcrypto.so.10(+0xdc621)[0x7f96866a3621] /usr/lib64/libssl.so.10(dtls1_do_write+0x18d)[0x7f96869e58ad] /usr/lib64/libssl.so.10(dtls1_accept+0xaa7)[0x7f96869dfc77] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xf6d5)[0x7f964048c6d5] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xf7a6)[0x7f964048c7a6] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x199d1)[0x7f96404969d1] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x48792)[0x7f96404c5792] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xad46)[0x7f9640487d46] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x39cbf)[0x7f96404b6cbf] /lib64/libpthread.so.0(+0x79d1)[0x7f96861ad9d1] /lib64/libc.so.6(clone+0x6d)[0x7f96872d286d] | ||
Comments: | By: Matt Jordan (mjordan) 2014-09-17 16:44:21.244-0500 Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks! By: Badalian Vyacheslav (slavon) 2014-09-22 23:52:59.467-0500 Done! By: Badalian Vyacheslav (slavon) 2014-09-24 05:29:06.259-0500 any updates? By: Matt Jordan (mjordan) 2014-09-25 11:25:59.462-0500 So, there's a whole mess of uninitialized values getting used by {{libssl}}. There's not much we can do about that. However, valgrind did complain about an invalid read (off by one) when reading from the master key buffer allocated in {{res_srtp}}: {noformat} ==23717== Thread 88: ==23717== Invalid read of size 1 ==23717== at 0xA794F00: v128_copy_octet_string (in /usr/lib64/libsrtp.so.0.0.0) ==23717== by 0xA792898: aes_icm_context_init (in /usr/lib64/libsrtp.so.0.0.0) ==23717== by 0xA78CFFC: srtp_kdf_init (in /usr/lib64/libsrtp.so.0.0.0) ==23717== by 0xA78D089: srtp_stream_init_keys (in /usr/lib64/libsrtp.so.0.0.0) ==23717== by 0xA78D59E: srtp_stream_init (in /usr/lib64/libsrtp.so.0.0.0) ==23717== by 0xA78ED8C: srtp_add_stream (in /usr/lib64/libsrtp.so.0.0.0) ==23717== by 0xA78EE93: srtp_create (in /usr/lib64/libsrtp.so.0.0.0) ==23717== by 0xA5853A8: ast_srtp_create (res_srtp.c:450) ==23717== by 0x558421: ast_rtp_instance_add_srtp_policy (rtp_engine.c:2072) ==23717== by 0xF566BA7: dtls_srtp_setup (res_rtp_asterisk.c:1661) ==23717== by 0xF566FEC: __rtp_recvfrom (res_rtp_asterisk.c:1739) ==23717== by 0xF5672A9: rtp_recvfrom (res_rtp_asterisk.c:1789) ==23717== Address 0x29b20f2e is 0 bytes after a block of size 30 alloc'd ==23717== at 0x4A0577B: calloc (vg_replace_malloc.c:593) ==23717== by 0x596BC8: _ast_calloc (utils.h:514) ==23717== by 0xA584B04: ast_srtp_policy_set_master_key (res_srtp.c:293) ==23717== by 0xF566ABD: dtls_srtp_setup (res_rtp_asterisk.c:1649) ==23717== by 0xF566FEC: __rtp_recvfrom (res_rtp_asterisk.c:1739) ==23717== by 0xF5672A9: rtp_recvfrom (res_rtp_asterisk.c:1789) ==23717== by 0xF5706B2: ast_rtp_read (res_rtp_asterisk.c:3862) ==23717== by 0x551286: ast_rtp_instance_read (rtp_engine.c:314) ==23717== by 0xC43C641: sip_rtp_read (chan_sip.c:8194) ==23717== by 0xC43CDF0: sip_read (chan_sip.c:8291) ==23717== by 0x47C91C: __ast_read (channel.c:4054) ==23717== by 0x47E6C5: ast_read (channel.c:4408) {noformat} That's a little odd, since the length of the {{unsigned char}} buffer passed to that function shouldn't be NULL terminated or have any other requirements that necessitate an additional byte of space. This may actually be a bug in {{libsrtp}} - see [http://sourceforge.net/p/srtp/bugs/7/]. It certainly appears like {{aes_icm_context_init}} is reading past the key. There's a few other bug reports from browser vendors regarding this same function; they may be related. I've attached something that _may_ fix this - it simply allocates one additional byte of space in {{res_srtp}}. That should be harmless for SDES-SRTP or DTLS, as the actual key length would be unaffected. As this does appear to be a bug in {{libsrtp}}, if this doesn't work around the issue, you may have to push the issue to that project. By: Rusty Newton (rnewton) 2014-09-25 13:00:37.072-0500 [~slavon] let us know when you've tested the potential work-around. I'll set the issue to waiting on feedback. Thanks! By: Badalian Vyacheslav (slavon) 2014-09-29 23:24:36.179-0500 Hello! Does not help....But valgrind log look more clear... Crash 1: {code} 0x0000003be3c89902 in memcpy () from /lib64/libc.so.6 (gdb) bt #0 0x0000003be3c89902 in memcpy () from /lib64/libc.so.6 #1 0x0000003bea43bff7 in do_dtls1_write () from /usr/lib64/libssl.so.10 #2 0x0000003bea43e80a in dtls1_do_write () from /usr/lib64/libssl.so.10 #3 0x0000003bea438c77 in dtls1_accept () from /usr/lib64/libssl.so.10 #4 0x0000003bea43d4ad in dtls1_read_bytes () from /usr/lib64/libssl.so.10 #5 0x0000003bea427d50 in ?? () from /usr/lib64/libssl.so.10 #6 0x00007ffd42b2cef4 in __rtp_recvfrom (instance=0x7ffcc4003a58, buf=0x7ffd28bd9c90, size=8192, flags=0, sa=0x7ffd28bdbc90, rtcp=1) at res_rtp_asterisk.c:1723 #7 0x00007ffd42b2d260 in rtcp_recvfrom (instance=0x7ffcc4003a58, buf=0x7ffd28bd9c90, size=8192, flags=0, sa=0x7ffd28bdbc90) at res_rtp_asterisk.c:1784 #8 0x00007ffd42b34333 in ast_rtcp_read (instance=0x7ffcc4003a58) at res_rtp_asterisk.c:3466 #9 0x00007ffd42b36645 in ast_rtp_read (instance=0x7ffcc4003a58, rtcp=1) at res_rtp_asterisk.c:3851 #10 0x0000000000551287 in ast_rtp_instance_read (instance=0x7ffcc4003a58, rtcp=1) at rtp_engine.c:314 #11 0x00007ffd79c18663 in sip_rtp_read (ast=0x7ffcc4016b78, p=0x7ffcc4034688, faxdetect=0x7ffd28bdc604) at chan_sip.c:8197 #12 0x00007ffd79c18df1 in sip_read (ast=0x7ffcc4016b78) at chan_sip.c:8291 #13 0x000000000047c91d in __ast_read (chan=0x7ffcc4016b78, dropaudio=0) at channel.c:4054 #14 0x000000000047e6c6 in ast_read (chan=0x7ffcc4016b78) at channel.c:4408 #15 0x00007ffd2e441a1a in wait_for_answer (in=0x7ffcc4016b78, out_chans=0x7ffd28bde950, to=0x7ffd28bde94c, peerflags=0x7ffd28bdeeb0, opt_args=0x7ffd28bde190, pa=0x7ffd28bde270, num_in=0x7ffd28bde930, result=0x7ffd28bde26c, dtmf_progress=0x0, ignore_cc=1, forced_clid=0x7ffd28bde040, stored_clid=0x7ffd28bddff0) at app_dial.c:1562 #16 0x00007ffd2e446f70 in dial_exec_full (chan=0x7ffcc4016b78, data=0x7ffd28be1110 "SIP/avaya/989090054050,300,Tt", peerflags=0x7ffd28bdeeb0, continue_exec=0x0) at app_dial.c:2683 #17 0x00007ffd2e449789 in dial_exec (chan=0x7ffcc4016b78, data=0x7ffd28be1110 "SIP/avaya/989090054050,300,Tt") at app_dial.c:3130 #18 0x000000000052b0e1 in pbx_exec (c=0x7ffcc4016b78, app=0x26a07f0, data=0x7ffd28be1110 "SIP/avaya/989090054050,300,Tt") at pbx.c:1622 #19 0x0000000000535afe in pbx_extension_helper (c=0x7ffcc4016b78, con=0x0, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=22, label=0x0, callerid=0x7ffc8001e960 "84996051913", action=E_SPAWN, found=0x7ffd28be378c, combined_find_spawn=1) at pbx.c:4915 #20 0x0000000000538f99 in ast_spawn_extension (c=0x7ffcc4016b78, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=22, callerid=0x7ffc8001e960 "84996051913", found=0x7ffd28be378c, combined_find_spawn=1) at pbx.c:6037 #21 0x00007ffd2d4180b0 in _macro_exec (chan=0x7ffcc4016b78, data=0x7ffd28be6490 "dialout-trunk,2,989090054050,,off", exclusive=0) at app_macro.c:412 #22 0x00007ffd2d419342 in macro_exec (chan=0x7ffcc4016b78, data=0x7ffd28be6490 "dialout-trunk,2,989090054050,,off") at app_macro.c:585 #23 0x000000000052b0e1 in pbx_exec (c=0x7ffcc4016b78, app=0x269b180, data=0x7ffd28be6490 "dialout-trunk,2,989090054050,,off") at pbx.c:1622 #24 0x0000000000535afe in pbx_extension_helper (c=0x7ffcc4016b78, con=0x0, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=5, label=0x0, callerid=0x7ffc8001e960 "84996051913", action=E_SPAWN, found=0x7ffd28be8b70, combined_find_spawn=1) at pbx.c:4915 #25 0x0000000000538f99 in ast_spawn_extension (c=0x7ffcc4016b78, context=0x7ffcc40179c8 "macro-dialout-trunk", exten=0x7ffcc4017a18 "s", priority=5, callerid=0x7ffc8001e960 "84996051913", found=0x7ffd28be8b70, combined_find_spawn=1) at pbx.c:6037 #26 0x000000000053a736 in __ast_pbx_run (c=0x7ffcc4016b78, args=0x0) at pbx.c:6512 #27 0x000000000053c213 in pbx_thread (data=0x7ffcc4016b78) at pbx.c:6842 #28 0x0000000000598580 in dummy_start (data=0x7ffcc402c2a0) at utils.c:1169 #29 0x0000003be44079d1 in start_thread () from /lib64/libpthread.so.0 #30 0x0000003be3ce886d in clone () from /lib64/libc.so.6 {code} Crash 2: run in 2 console. Its ONE crash bug in gdb and system baktrace print: {code} ======= Backtrace: ========= /lib64/libc.so.6[0x3be3c75e76] /lib64/libc.so.6[0x3be3c789b3] /usr/lib64/libcrypto.so.10(CRYPTO_realloc_clean+0xf3)[0x3be6868de3] /usr/lib64/libcrypto.so.10(BUF_MEM_grow_clean+0x86)[0x3be68d8996] /usr/lib64/libcrypto.so.10[0x3be68da3f3] /usr/lib64/libcrypto.so.10(BIO_write+0x77)[0x3be68d95f7] /usr/lib64/libcrypto.so.10[0x3be68dc621] /usr/lib64/libssl.so.10(dtls1_do_write+0x18d)[0x3bea43e8ad] /usr/lib64/libssl.so.10(dtls1_accept+0xaa7)[0x3bea438c77] /usr/lib64/libssl.so.10(dtls1_read_bytes+0x6ed)[0x3bea43d4ad] /usr/lib64/libssl.so.10[0x3bea427d50] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xbef4)[0x7f82cf76cef4] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0xc260)[0x7f82cf76d260] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x13333)[0x7f82cf774333] /usr/lib/asterisk/modules/res_rtp_asterisk.so(+0x15645)[0x7f82cf776645] /usr/sbin/asterisk(ast_rtp_instance_read+0x2b)[0x551287] /usr/lib/asterisk/modules/chan_sip.so(+0x2d663)[0x7f82d68cb663] /usr/lib/asterisk/modules/chan_sip.so(+0x2ddf1)[0x7f82d68cbdf1] /usr/sbin/asterisk[0x47c91d] /usr/sbin/asterisk(ast_read+0x1d)[0x47e6c6] /usr/lib/asterisk/modules/app_dial.so(+0x8a1a)[0x7f82bf15ea1a] /usr/lib/asterisk/modules/app_dial.so(+0xdf70)[0x7f82bf163f70] /usr/lib/asterisk/modules/app_dial.so(+0x10789)[0x7f82bf166789] /usr/sbin/asterisk(pbx_exec+0x214)[0x52b0e1] /usr/sbin/asterisk[0x535afe] /usr/sbin/asterisk(ast_spawn_extension+0x65)[0x538f99] /usr/lib/asterisk/modules/app_macro.so(+0x30b0)[0x7f82be1350b0] /usr/lib/asterisk/modules/app_macro.so(+0x4342)[0x7f82be136342] /usr/sbin/asterisk(pbx_exec+0x214)[0x52b0e1] /usr/sbin/asterisk[0x535afe] /usr/sbin/asterisk(ast_spawn_extension+0x65)[0x538f99] /usr/sbin/asterisk[0x53a736] /usr/sbin/asterisk[0x53c213] /usr/sbin/asterisk[0x598580] /lib64/libpthread.so.0[0x3be44079d1] /lib64/libc.so.6(clone+0x6d)[0x3be3ce886d] {code} {code} 0x0000003be3c32635 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003be3c32635 in raise () from /lib64/libc.so.6 #1 0x0000003be3c33e15 in abort () from /lib64/libc.so.6 #2 0x0000003be3c70547 in __libc_message () from /lib64/libc.so.6 #3 0x0000003be3c75e76 in malloc_printerr () from /lib64/libc.so.6 #4 0x0000003be3c789b3 in _int_free () from /lib64/libc.so.6 #5 0x0000003be6868de3 in CRYPTO_realloc_clean () from /usr/lib64/libcrypto.so.10 #6 0x0000003be68d8996 in BUF_MEM_grow_clean () from /usr/lib64/libcrypto.so.10 #7 0x0000003be68da3f3 in ?? () from /usr/lib64/libcrypto.so.10 #8 0x0000003be68d95f7 in BIO_write () from /usr/lib64/libcrypto.so.10 #9 0x0000003be68dc621 in ?? () from /usr/lib64/libcrypto.so.10 #10 0x0000003bea43e8ad in dtls1_do_write () from /usr/lib64/libssl.so.10 #11 0x0000003bea438c77 in dtls1_accept () from /usr/lib64/libssl.so.10 #12 0x0000003bea43d4ad in dtls1_read_bytes () from /usr/lib64/libssl.so.10 #13 0x0000003bea427d50 in ?? () from /usr/lib64/libssl.so.10 #14 0x00007f82cf76cef4 in __rtp_recvfrom (instance=0x7f82e8031e98, buf=0x7f82b88fac90, size=8192, flags=0, sa=0x7f82b88fcc90, rtcp=1) at res_rtp_asterisk.c:1723 #15 0x00007f82cf76d260 in rtcp_recvfrom (instance=0x7f82e8031e98, buf=0x7f82b88fac90, size=8192, flags=0, sa=0x7f82b88fcc90) at res_rtp_asterisk.c:1784 #16 0x00007f82cf774333 in ast_rtcp_read (instance=0x7f82e8031e98) at res_rtp_asterisk.c:3466 #17 0x00007f82cf776645 in ast_rtp_read (instance=0x7f82e8031e98, rtcp=1) at res_rtp_asterisk.c:3851 #18 0x0000000000551287 in ast_rtp_instance_read (instance=0x7f82e8031e98, rtcp=1) at rtp_engine.c:314 #19 0x00007f82d68cb663 in sip_rtp_read (ast=0x7f82e8049668, p=0x7f82e8026488, faxdetect=0x7f82b88fd604) at chan_sip.c:8197 #20 0x00007f82d68cbdf1 in sip_read (ast=0x7f82e8049668) at chan_sip.c:8291 #21 0x000000000047c91d in __ast_read (chan=0x7f82e8049668, dropaudio=0) at channel.c:4054 #22 0x000000000047e6c6 in ast_read (chan=0x7f82e8049668) at channel.c:4408 #23 0x00007f82bf15ea1a in wait_for_answer (in=0x7f82e8049668, out_chans=0x7f82b88ff950, to=0x7f82b88ff94c, peerflags=0x7f82b88ffeb0, opt_args=0x7f82b88ff190, pa=0x7f82b88ff270, num_in=0x7f82b88ff930, result=0x7f82b88ff26c, dtmf_progress=0x0, ignore_cc=1, forced_clid=0x7f82b88ff040, stored_clid=0x7f82b88feff0) at app_dial.c:1562 #24 0x00007f82bf163f70 in dial_exec_full (chan=0x7f82e8049668, data=0x7f82b8902110 "SIP/avaya/989123032784,300,Tt", peerflags=0x7f82b88ffeb0, continue_exec=0x0) at app_dial.c:2683 #25 0x00007f82bf166789 in dial_exec (chan=0x7f82e8049668, data=0x7f82b8902110 "SIP/avaya/989123032784,300,Tt") at app_dial.c:3130 #26 0x000000000052b0e1 in pbx_exec (c=0x7f82e8049668, app=0x1e146d0, data=0x7f82b8902110 "SIP/avaya/989123032784,300,Tt") at pbx.c:1622 #27 0x0000000000535afe in pbx_extension_helper (c=0x7f82e8049668, con=0x0, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=22, label=0x0, callerid=0x7f826c001ec0 "84996051913", action=E_SPAWN, found=0x7f82b890478c, combined_find_spawn=1) at pbx.c:4915 #28 0x0000000000538f99 in ast_spawn_extension (c=0x7f82e8049668, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=22, callerid=0x7f826c001ec0 "84996051913", found=0x7f82b890478c, combined_find_spawn=1) at pbx.c:6037 #29 0x00007f82be1350b0 in _macro_exec (chan=0x7f82e8049668, data=0x7f82b8907490 "dialout-trunk,2,989123032784,,off", exclusive=0) at app_macro.c:412 #30 0x00007f82be136342 in macro_exec (chan=0x7f82e8049668, data=0x7f82b8907490 "dialout-trunk,2,989123032784,,off") at app_macro.c:585 #31 0x000000000052b0e1 in pbx_exec (c=0x7f82e8049668, app=0x1e0e5e0, data=0x7f82b8907490 "dialout-trunk,2,989123032784,,off") at pbx.c:1622 #32 0x0000000000535afe in pbx_extension_helper (c=0x7f82e8049668, con=0x0, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=5, label=0x0, callerid=0x7f826c001ec0 "84996051913", action=E_SPAWN, found=0x7f82b8909b70, combined_find_spawn=1) at pbx.c:4915 #33 0x0000000000538f99 in ast_spawn_extension (c=0x7f82e8049668, context=0x7f82e804a4b8 "macro-dialout-trunk", exten=0x7f82e804a508 "s", priority=5, callerid=0x7f826c001ec0 "84996051913", found=0x7f82b8909b70, combined_find_spawn=1) at pbx.c:6037 #34 0x000000000053a736 in __ast_pbx_run (c=0x7f82e8049668, args=0x0) at pbx.c:6512 #35 0x000000000053c213 in pbx_thread (data=0x7f82e8049668) at pbx.c:6842 #36 0x0000000000598580 in dummy_start (data=0x7f82e803e9f0) at utils.c:1169 #37 0x0000003be44079d1 in start_thread () from /lib64/libpthread.so.0 #38 0x0000003be3ce886d in clone () from /lib64/libc.so.6 {code} By: Badalian Vyacheslav (slavon) 2014-09-29 23:41:49.418-0500 Also 2 runs in Valgrind By: Rusty Newton (rnewton) 2014-10-01 15:21:28.038-0500 Thanks for the additional information. [~kmoore] reviewed the debug, finding that the issue still appears to be a bug in libsrtp and not in Asterisk. I'm going to close this issue out since there isn't anything we can do at this point. If you file a new issue with the libsrtp project then please link the issue in a comment here. By: Badalian Vyacheslav (slavon) 2014-10-01 15:25:15.691-0500 Sorry. We have 1-3 Crash in WS mode every day. Glibc say for random errors like: asterisk[3328]: segfault at d9c36b47930 ip 0000003be3c47e3c sp 00007ffd16531410 error 4 in libc-2.12.so[3be3c00000+18a000] asterisk[47139] general protection ip:3be3c47e3c sp:7f360d06d410 error:0 in libc-2.12.so[3be3c00000+18a000] Also have errors for double free. WS mode also use SRTP? By: Badalian Vyacheslav (slavon) 2014-10-01 15:57:11.370-0500 Also last update of libSRTP is 2007-05-04. Maybe need to change library? SRTP and other secure now in priority, Is update to asterisk 12.x will fix problem? You have another sip library in 12.x... |