Summary:	ASTERISK-28517: Asterisk segfault in t38_interpret_parameters at res_pjsip_t38.c:457
Reporter:	John Cahill (jcahill)	Labels:	fax patch pjsip
Date Opened:	2019-08-30 09:01:28	Date Closed:	2019-09-05 11:16:57
Priority:	Major	Regression?	No
Status:	Closed/Complete	Components:	Resources/res_pjsip_t38
Versions:	16.2.1 16.5.0	Frequency of Occurrence
Related Issues:	is related to ASTERISK-28495 res_pjsip_t38: 200 OK with SDP answer with declined stream causes crash
Environment:	Debian 10.0, 4.9.0-3-amd64 2GB RAM, 4 VCPUs, Xen 4.4 Dell PowerEdge R430	Attachments:	( 0) 69ed619.diff ( 1) backtrace.txt
Description:	Asterisk segfault in t38_interpret_parameters at res_pjsip_t38.c:457 during t.38 pass-through fax transmission. Happens every time. Same crash on 16.5 and 16.2.1 Backtrace in reference notes
Comments:	By: Asterisk Team (asteriskteam) 2019-08-30 09:01:30.256-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: John Cahill (jcahill) 2019-08-30 09:05:23.402-0500 I can supply a SIP trace for the above but do not wish it to be publicly accessible. Many Thanks for your time. By: Joshua C. Colp (jcolp) 2019-08-30 10:32:21.344-0500 This is now locked down so you can attach things. In the future if you believe there's a security issue ensure you use the "Security" type as it will automatically lock down and get flagged. By: John Cahill (jcahill) 2019-09-02 03:19:19.795-0500 pcap of fax to 01865251758 attached By: Joshua C. Colp (jcolp) 2019-09-02 05:37:37.706-0500 I believe this is already a known security vulnerability which should be resolved in an upcoming security release occurring shortly. By: John Cahill (jcahill) 2019-09-02 16:29:05.023-0500 Hi, Thanks. Any idea on the timescale for a fix? Are we talking about days or weeks? I will probably need to roll back several systems to chan_sip. Are there other open tickets that you can reference where I can read more about the issue? Many Thanks. Kind Regards, John By: Joshua C. Colp (jcolp) 2019-09-02 16:37:05.051-0500 Likely this week. They are security issues, so you do not have access and access can not be given on a per-issue basis. By: Joshua C. Colp (jcolp) 2019-09-02 16:40:35.805-0500 Attached is the current fix. By: John Cahill (jcahill) 2019-09-03 02:16:57.049-0500 May thanks for the patch. If I do a git pull from gerrit for branch 16.5 may I appy this patch? Can you give instructions on how to apply to whatever branch is appropriate? I will apply, test and report back. Thanks. By: John Cahill (jcahill) 2019-09-03 04:18:17.406-0500 Hi, I applied the patch and reinstalled: cp ./69ed619.diff /usr/src/asterisk-16.5 cd /usr/src/asterisk-16.5 git add ./69ed619.diff git apply ./69ed619.diff make clean make make install This patch fixes the segfault crash which is great but T.38 negotiation still fails. Asterisk immidiately sends a non t.38 re-invite on B leg after it receives a t.38 re-invite from the B leg. A (lcr-02) and B (coltukpriv) endpoints are below. T.38 fax works fine with this carrier (Colt) using chan_sip. I will attach the full log and a SIP trace. Is there any other information I should provide?Thanks. Regards, John [lcr-02] type = endpoint context = voipin dtmf_mode = rfc4733 disallow = all allow = alaw allow = ulaw allow = g729 rtp_symmetric = yes force_rport = yes rewrite_contact = yes rtp_timeout = 120 moh_suggest = default timers = yes direct_media = no send_rpid = yes trust_id_inbound = no inband_progress = yes sdp_session = V4U SBC v2.0 aors = lcr-02 t38_udptl = yes t38_udptl_ec = redundancy t38_udptl_maxdatagram = 400 t38_udptl_nat=yes 100rel=no [coltukpriv] type = endpoint context = coltin dtmf_mode = rfc4733 disallow = all allow = alaw allow = ulaw allow = g729 rtp_symmetric = yes force_rport = yes rewrite_contact = yes rtp_timeout = 120 moh_suggest = default timers = yes direct_media = no send_pai = yes trust_id_outbound=yes trust_id_inbound=yes inband_progress = yes sdp_session = V4U SBC v2.0 aors = coltukpriv t38_udptl = yes t38_udptl_ec = redundancy t38_udptl_maxdatagram = 400 t38_udptl_nat=yes 100rel=no By: John Cahill (jcahill) 2019-09-03 04:19:43.246-0500 /var/log/asterisk/full By: John Cahill (jcahill) 2019-09-03 04:21:31.156-0500 pcap for failed t.38 pass-through after patch was applied. By: Joshua C. Colp (jcolp) 2019-09-03 04:59:46.287-0500 Once the patch is released you can open a new issue and it will go into the normal queue of things. There is no time frame on when it would get looked into. By: John Cahill (jcahill) 2019-09-03 07:19:31.936-0500 Thanks. Can you suggest any tips to debug further? I'm struggling to understand the behavior of the immediate non t.38 re-invite sent from asterisk when it receives the t.38 re-invite from the B leg. Why would asterisk do that? By: Joshua C. Colp (jcolp) 2019-09-03 08:44:02.155-0500 The reason why stuff is falling apart is likely the fact that we are receiving a 491 response to our re-invite attempt. That would be a starting point. By: John Cahill (jcahill) 2019-09-03 09:18:47.068-0500 Thanks. The carrier on  the B leg has sent asterisk a t.38 re-invite. Asterisk doesn't acknowledge this and instead immediately sends back an INVITE. I think the 491 pending response is proper behavior on the part of the carrier SBC. Any ideas as to why asterisk would do that? The carrier works fine as regards t.38 fax with chan_sip. By: Joshua C. Colp (jcolp) 2019-09-03 09:28:57.563-0500 They both appear to be sending the re-invite at the same time so I don't dispute a 491 is correct, but it's not a scenario that the state machine in PJSIP has been written for/tested against so it is entirely possible that it is causing things to go wrong. Fixing it requires tracing down the specific scenario, understanding where the various state machines are going wrong, tweaking things, and attempting to fix. By: Kevin Harwell (kharwell) 2019-09-03 09:40:24.669-0500 {quote} I can supply a SIP trace for the above but do not wish it to be publicly accessible. {quote} [~jcahill], I'm not sure if you meant you didn't want to make it publicly accessible due to the vulnerability, or if you have sensitive information you didn't want seen in the trace. If you meant the latter just be aware that this issue will be made public once the security release goes out, so you'll want to remove any sensitive attachments and/or information. By: John Cahill (jcahill) 2019-09-03 16:48:36.472-0500 Hi Kevin, I meant the latter. My asterisk machines only talk to carrier SBCs. They are are locked down by IP. Can you email details of ASTERISK-28495. I do not have the necessary permissions to view. When will the issue become public? Thanks. Regards, John By: Kevin Harwell (kharwell) 2019-09-04 09:54:35.450-0500 As far as the details on the other issue it just crashes in a similar way to the de-referencing of a NULL pointer, but under a different scenario (different area of the code). The attached patch fixed all areas this could potentially happen, thus fixing the problem in a few different spots. Currently the next security release is scheduled for this Thursday (Sept 6). Note, that's the earliest it'll go out. It could be delayed. At that time all issues associated with the security release will be made public. By: Kevin Harwell (kharwell) 2019-09-05 11:08:24.342-0500 I went ahead and removed the attached pcaps and log as they might contain sensitive information since this issue is now becoming public.