Summary:	ASTERISK-29305: ASTERISK-29203 / AST-2021-002 -- Another scenario is causing a crash
Reporter:	Gregory Massel (gmza)	Labels:	patch
Date Opened:	2021-02-20 04:01:31.000-0600	Date Closed:	2021-03-04 10:03:55.000-0600
Priority:	Blocker	Regression?	No
Status:	Closed/Complete	Components:	Resources/res_pjsip_t38
Versions:	16.16.1	Frequency of Occurrence
Related Issues:
Environment:	Ubuntu 18.04.5 LTS, Asterisk 16.16.1, kernel 5.0.0-29-generic, Intel Xeon E3-1285 v6	Attachments:	( 0) AST-29305-2-cpe-to-proxy.pcap ( 1) AST-29305-2-trunk-leg.pcap ( 2) AST-29305-cpe-to-proxy.pcap ( 3) AST-29305-trunk-leg.pcap ( 4) AST-29305-with-patch.pcap ( 5) ASTERISK-29305.diff ( 6) core-brief.txt ( 7) core-full.txt ( 8) core-info.txt ( 9) core-locks.txt (10) core-thread1.txt (11) log.txt
Description:	Asterisk is still crashing under certain instances when there is a state change within T.38. Segmentation Fault (SIGSEGV) in t38_reinvite_response_cb. Backtrace will be uploaded and tarball made available.
Comments:	By: Asterisk Team (asteriskteam) 2021-02-20 04:01:41.667-0600 This issue has been automatically restricted and set to a blocker due to being a security type issue. If this is not a security vulnerability issue it will be moved to the appropriate issue type when triaged. Please DO NOT put a code review up for this change at this time. Attach any applicable patches to this issue. By: Asterisk Team (asteriskteam) 2021-02-20 04:01:42.154-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/]. By: Gregory Massel (gmza) 2021-02-20 04:03:02.146-0600 Backtrace attached By: Gregory Massel (gmza) 2021-02-20 04:05:22.028-0600 Full tarball can be downloaded from: https://www.switchtel.co.za/ASTERISK-29305/core.tar.gz 85MB in size, hence I have not attached. Please advise once downloaded as I don't want to leave this link accessible for too long. By: Joshua C. Colp (jcolp) 2021-02-20 04:07:59.300-0600 I've downloaded the tarball, but do you have a SIP trace of the specific interaction? By: Joshua C. Colp (jcolp) 2021-02-20 04:14:12.014-0600 As well, an Asterisk core debug log would also be useful to show the internal state changes and what is going on so we can be sure. By: Gregory Massel (gmza) 2021-02-20 04:28:03.702-0600 Attached are PCAPs of the call that triggered the crash upon T.38 re-INVITE. The file AST-29305-cpe-to-proxy is of the Grandstream DP752 VoIP Phone (105.226.69.128) calling towards our OpenSIPS proxy (196.216.192.5). I don't have a PCAP of the leg from the proxy (196.216.192.5) towards the Asterisk box (196.216.192.3). The second file is the leg from Asterisk box 196.216.192.3 (the one that receives the call via the OpenSIPS proxy and crashes) towards Asterisk box 196.216.192.4 (which did NOT crash and is also running 16.16.1). So the call flow is: VoIP Phone (105.226.69.128) -> Proxy (196.216.192.5) -> Asterisk1 (196.216.192.3) -> Asterisk2 (196.216.192.4) -> Trunk The T.38 re-INVITE comes from the direction of the Trunk via Asterisk2 (called party). The VoIP Phone, being a regular phone, does NOT support T.38. Asterisk2 remains up but Asterisk1 crashes. Both are version 16.16.1. The issue is substantively identical to Asterisk-29203 and both Asterisk boxes were upgraded to 16.16.1 only a few hours ago. This issue cropped up within a few hours of upgrade. Prior to that, Asterisk2 was running 16.11.1 and Asterisk1 a patched version of 16.12.0. The upgrade was done specifically because it was thought that Asterisk-29203 was fully resolved, however, it would appear that this is a closely related scenario. I did test the patch for Asterisk-29203 and it resolved the scenario that we were testing, however, this is a different B-number that I hadn't encountered before and the call signalling must be subtly different. By: Gregory Massel (gmza) 2021-02-20 04:32:47.903-0600 I didn't have verbose debugging on the production server owing to the volumes it generates. I have just tried to reproduce this by calling the same B number, however, it doesn't crash when I make the call. By: Gregory Massel (gmza) 2021-02-20 04:49:12.908-0600 I have tried to replicate now this using two different T.38 capable devices (Grandstream HT802, Linksys SPA2102) as well as a non-T.38 device (Grandstream GXP2170) and I cannot replicate the issue. I'm going to try and locate a Grandstream DP752 (identical CPE to the call that caused this crash), will match up the firmware version as well, and then try and replicate with that. By: Gregory Massel (gmza) 2021-02-20 05:28:05.177-0600 I managed to replicate it with the exact same CPE (Grandstream DP752). Attached is the detailed debug log. By: Gregory Massel (gmza) 2021-02-20 05:43:22.854-0600 Herewith another pair of PCAPs from the replicated issue. In this one, 196.216.192.22 is the Asterisk box that crashes, 169.0.168.84 the Grandstream DP752 phone. Per the previous example, 196.216.192.4 remains the second Asterisk box (that doesn't crash) and 196.216.192.5 remains the OpenSIPS proxy. All Asterisk boxes throughout are version 16.16.1. By: Joshua C. Colp (jcolp) 2021-02-22 03:44:30.464-0600 Does this attached patch resolve the issue? By: Gregory Massel (gmza) 2021-02-22 09:57:09.865-0600 I confirm that the patch works. I made a test call, crashed the server, then immediately restarted patched Asterisk and re-tested, and it cleared the call with ISUP cause 58 ("58 – bearer capability not presently available"). For good measure I re-tested a further three times in a row and all were cleared by Asterisk with cause 58. Thank you! By: Gregory Massel (gmza) 2021-02-22 09:57:47.272-0600 PCAP of call flow after applying the patch where Asterisk correctly clears the call. By: Friendly Automation (friendly-automation) 2021-03-04 10:03:56.923-0600 Change 15581 merged by Friendly Automation: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15581|https://gerrit.asterisk.org/c/asterisk/+/15581] By: Friendly Automation (friendly-automation) 2021-03-04 10:17:53.616-0600 Change 15560 merged by George Joseph: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15560|https://gerrit.asterisk.org/c/asterisk/+/15560] By: Friendly Automation (friendly-automation) 2021-03-04 10:18:15.298-0600 Change 15582 merged by George Joseph: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15582|https://gerrit.asterisk.org/c/asterisk/+/15582] By: Friendly Automation (friendly-automation) 2021-03-04 10:18:29.710-0600 Change 15583 merged by George Joseph: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15583|https://gerrit.asterisk.org/c/asterisk/+/15583] By: Friendly Automation (friendly-automation) 2021-03-04 10:18:42.341-0600 Change 15584 merged by George Joseph: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15584|https://gerrit.asterisk.org/c/asterisk/+/15584] By: Friendly Automation (friendly-automation) 2021-03-04 10:28:14.458-0600 Change 15565 merged by George Joseph: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15565|https://gerrit.asterisk.org/c/asterisk/+/15565] By: Friendly Automation (friendly-automation) 2021-03-04 10:28:53.603-0600 Change 15566 merged by George Joseph: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15566|https://gerrit.asterisk.org/c/asterisk/+/15566] By: Friendly Automation (friendly-automation) 2021-03-04 10:29:28.782-0600 Change 15567 merged by George Joseph: AST-2021-006 - res_pjsip_t38.c: Check for session_media on reinvite. [https://gerrit.asterisk.org/c/asterisk/+/15567|https://gerrit.asterisk.org/c/asterisk/+/15567]