Summary:	ASTERISK-26690: res_pjsip: segfault in ssl_write from pjsip_endpt_process_rx_data
Reporter:	Frederik Vermeulen (frederik)	Labels:
Date Opened:	2017-01-04 01:27:49.000-0600	Date Closed:	2017-01-25 11:56:06.000-0600
Priority:	Major	Regression?	No
Status:	Closed/Complete	Components:	Channels/chan_pjsip pjproject/pjsip
Versions:	13.13.1	Frequency of Occurrence	Frequent
Related Issues:
Environment:		Attachments:	( 0) backtrace.txt
Description:	Asterisk crashes after a few hours of running, see trace below. This is asterisk-13.13.1 with pjproject-2.5.5. This seems to be in pjsip_endpt_process_rx_data. My first idea would be that an openssl socket is being accessed from multiple asterisk threads? {noformat} Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff3d64700 (LWP 6347)] __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:116 116     ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory. (gdb) bt #0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:116 #1  0x00007ffff6e03239 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2  0x00007ffff6e021dc in BIO_write () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3  0x00007ffff712cf12 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 #4  0x00007ffff712d5d4 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 #5  0x00007ffff0744268 in ssl_write () from /usr/local/lib/libpj.so.2 #6  0x00007ffff0744699 in pj_ssl_sock_send () from /usr/local/lib/libpj.so.2 #7  0x00007ffff20f12bf in tls_send_msg () from /usr/local/lib/libpjsip.so.2 #8  0x00007ffff20e5825 in pjsip_transport_send () from /usr/local/lib/libpjsip.so.2 #9  0x00007ffff20fadc2 in tsx_send_msg () from /usr/local/lib/libpjsip.so.2 #10 0x00007ffff20fbe3e in tsx_on_state_proceeding_uas () from /usr/local/lib/libpjsip.so.2 #11 0x00007ffff20fbbf0 in tsx_on_state_trying () from /usr/local/lib/libpjsip.so.2 #12 0x00007ffff20fa2ed in pjsip_tsx_send_msg () from /usr/local/lib/libpjsip.so.2 #13 0x00007ffff2ce0f09 in ast_sip_send_stateful_response (rdata=rdata@entry=0x7fffcc018ce8, tdata=0x7fffcc01f8b8,    sip_endpoint=sip_endpoint@entry=0x1260158) at res_pjsip.c:4156 #14 0x00007fff9b35fa81 in register_aor_core (rdata=0x7fffcc018ce8, endpoint=0x1260158, aor=0x124a578,    aor_name=0x124a8f0 "deleted", contacts=<optimized out>) at res_pjsip_registrar.c:514 {noformat}
Comments:	By: Asterisk Team (asteriskteam) 2017-01-04 01:27:50.774-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Rusty Newton (rnewton) 2017-01-05 09:08:18.168-0600 Thank you for the crash report. However, we need more information to investigate the crash. Please provide: 1. A backtrace generated from a core dump using the instructions provided on the Asterisk wiki [1]. 2. Specific steps taken that lead to the crash. 3. All configuration information necesary to reproduce the crash. Thanks! [1]: https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace By: Rusty Newton (rnewton) 2017-01-05 09:09:26.153-0600 Thanks for the report. 1. Please don't post lengthy debug in the description or comment fields. Follow the guidelines. 2. Please follow the linked instructions for getting a backtrace, making sure to recompile with the appropriate options. Attach it to the issue. By: Rusty Newton (rnewton) 2017-01-05 09:11:02.038-0600 Oh and you probably want debug symbols installed for your ssl and crypto libraries for the new trace. By: Rusty Newton (rnewton) 2017-01-05 09:22:34.434-0600 [~jcolp] identified this issue https://trac.pjsip.org/repos/ticket/1969 in pjsip, which is fixed in the latest Asterisk bundled pjproject. It may be the same issue. A full trace (following the instructions ) should help us identify. By: Asterisk Team (asteriskteam) 2017-01-19 12:00:00.999-0600 Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1]. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines By: Frederik Vermeulen (frederik) 2017-01-20 06:43:53.367-0600 New backtrace attached. By: Joshua C. Colp (jcolp) 2017-01-20 07:41:22.956-0600 Is this new backtrace from bundled on Asterisk 13.13.1? By: Frederik Vermeulen (frederik) 2017-01-20 08:51:32.751-0600 I downloaded and compiled pjproject-2.5.5 separately. Should I remove pjproject and recompile Asterisk with " ./configure --with-pjproject-bundled"? By: Joshua C. Colp (jcolp) 2017-01-20 09:00:15.303-0600 The fix mentioned is not yet in a PJSIP release. Our bundled has a backport of it which is applied. You do not need to remove your installed PJSIP, you can simply enable the bundled option and it will be used instead. By: Frederik Vermeulen (frederik) 2017-01-25 02:36:31.884-0600 After recompiling Asterisk with pjsip bundled, the crash has not happened anymore. By: Rusty Newton (rnewton) 2017-01-25 11:56:06.705-0600 Thanks for the follow up.