Summary:	ASTERISK-27382: crash after an invalid rtcp packet from GT48 FXS gateway
Reporter:	Tzafrir Cohen (tzafrir)	Labels:	patch
Date Opened:	2017-10-30 11:18:48	Date Closed:	2017-12-13 07:52:41.000-0600
Priority:	Major	Regression?	Yes
Status:	Closed/Complete	Components:
Versions:	13.13.1 13.17.2	Frequency of Occurrence	Constant
Related Issues:	duplicates ASTERISK-27429 res_rtp_asterisk: Multiple reports in an RTCP packet will write past where it should
Environment:	Centos 7	Attachments:	( 0) 27382-traces.tgz ( 1) AST-2017-012.txt ( 2) AST-2017-012-13.diff ( 3) core.7369-thread1.txt ( 4) crash_2.cap ( 5) crash.cap
Description:	I get a fully reproducible crash in calls from a GreaTel GT48R FXS gateway. Whenever a call is long enough and has audio, it crashes. I used current git branch 13 (2e5e7e0b143c2fdbd852d98a5d356a3fe1c3bc43 , 2017-10-27). SIP configuration: added the following to sip.conf: {code} [template](!) host = dynamic type = friend transport = udp secret=xxxxxx [558011](template) ; ditto for the other ports {code} The following will not crash: {code} channel originate SIP/558011 application Wait 50 channel originate SIP/558011 application Playback beep {code} The following does crash: {code} channel originate SIP/558011 application Playback demo-instruct {code} The shortest sound on the test system that seems to crash is sorry_didnt_get . Earlier versions of asterisk 13 did not crash from this. After bisecting, I found that branch 13 started crashing from this after applying commit b96f18560b529b614d0773a060bc03ef73498c61 astobj2: (Declare private variable data_size for AO2_DEBUG only.). Reverting that commit on branch 13 makes the crash go away, but I'm not yet sure how. Network traces seem to show that right at the end of the call (before the crash) the gateway sends an RTCP packet which seems to have (according to wireshark) odd values.
Comments:	By: Asterisk Team (asteriskteam) 2017-10-30 11:18:49.216-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Tzafrir Cohen (tzafrir) 2017-10-30 11:23:46.232-0500 Attaching crash.cap: tcpdump of a call that crashes asterisk. By: George Joseph (gjoseph) 2017-10-30 11:57:11.523-0500 Can you get backtraces? https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace --- Oh, you just did. :) By: George Joseph (gjoseph) 2017-10-30 12:04:11.353-0500 So 13.17.2 crashes but if you revert commit b96f18560b529b614d0773a060bc03ef73498c61, which was from a year ago, it works??? How about 13.16? By: Tzafrir Cohen (tzafrir) 2017-10-30 12:32:22.966-0500 As I mentioned, the version I tested was a pretty recent one: git commit 2e5e7e0b143c2fdbd852d98a5d356a3fe1c3bc43 , from a few days ago. It crashes. If on top of it I revert b96f18560b529b614d0773a060bc03ef73498c61, it does not crash. By: George Joseph (gjoseph) 2017-10-30 13:07:53.293-0500 Yeah what I was trying to understand was if it's been failing since last October or whether something recent might have triggered the issue even though reverting the commit from last year seems to fix it. From the pcap, it looks like the first rtcp packet the gateway sends causes the crash.  Anyway, I'm looking. By: George Joseph (gjoseph) 2017-10-30 13:24:23.942-0500 Can you repeat the crash with {noformat} rtcp set debug ip 192.168.6.10 {noformat} and post the log result? By: Tzafrir Cohen (tzafrir) 2017-10-30 13:47:12.423-0500 {code} RTCP from 192.168.6.10:10199 PT: 200(Sender Report) Reception reports: 1 SSRC of sender: 657616110 NTP timestamp: 3595367282.799888 RTP timestamp: 19992 SPC: 122 SOC: 19520 RTCP from 192.168.6.10:10199 PT: 201(Receiver Report) Reception reports: 1 SSRC of sender: 657616110 {code} In some cases I only saw the first one (Sender Report) logged before the crash. By: Tzafrir Cohen (tzafrir) 2017-10-30 13:53:59.296-0500 Attached crash_2.cap: a packet capture of a crashing call where two such RTCP packets are shown. By: George Joseph (gjoseph) 2017-10-30 13:58:41.800-0500 There's nothing strange there but something's stomping on memory.  Can you try with a recompile with MALLOC_DEBUG turned on? By: Tzafrir Cohen (tzafrir) 2017-10-30 14:56:04.609-0500 Enabled it. It now fails to crash. Enabling dev mode alone does not prevent crashing. By: George Joseph (gjoseph) 2017-11-02 12:37:44.353-0500 Hmmm.  I've tried extracting the rtp/rtcp stream from your pcap and playing it back into asterisk and can't reproduce the crash but something's stomping on memory.  Can you reproduce, ast_coredumper and attach all of the .txt files.  We need to see what else is going on at the time of the crash. By: Tzafrir Cohen (tzafrir) 2017-11-29 07:18:52.941-0600 Included traces from four core dumps. The second one took a while longer to happen. By: Joshua C. Colp (jcolp) 2017-11-29 08:28:16.316-0600 I'll be looking at another RTCP issue so I'll look into this as well soon. By: Joshua C. Colp (jcolp) 2017-11-30 11:01:40.992-0600 I've confirmed that this is a security issue that has also been reported by another individual. The fix will also resolve this. By: Joshua C. Colp (jcolp) 2017-11-30 18:51:42.856-0600 I have attached the draft security advisory and patch which is currently up for review that resolves the problem. I do not currently have a time frame for when a release will occur that incorporates it but will update this issue when I do. By: Tzafrir Cohen (tzafrir) 2017-12-03 12:37:45.227-0600 Patch fixes the issue on 13 here. I can no longer reproduce the issue in latest master, even though I remember I could (and I tried a few past versions), and thus I did not test the patch on master (but the patch basically applies there). By: Joshua C. Colp (jcolp) 2017-12-12 09:25:51.846-0600 The release time for the security release is looking like tomorrow. By: Friendly Automation (friendly-automation) 2017-12-13 07:52:42.625-0600 Change 7549 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7549|https://gerrit.asterisk.org/7549] By: Friendly Automation (friendly-automation) 2017-12-13 07:52:45.980-0600 Change 7550 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7550|https://gerrit.asterisk.org/7550] By: Friendly Automation (friendly-automation) 2017-12-13 07:52:48.518-0600 Change 7554 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7554|https://gerrit.asterisk.org/7554] By: Friendly Automation (friendly-automation) 2017-12-13 07:52:50.945-0600 Change 7553 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7553|https://gerrit.asterisk.org/7553] By: Friendly Automation (friendly-automation) 2017-12-13 07:53:16.032-0600 Change 7552 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7552|https://gerrit.asterisk.org/7552] By: Friendly Automation (friendly-automation) 2017-12-13 07:53:17.177-0600 Change 7551 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7551|https://gerrit.asterisk.org/7551] By: Friendly Automation (friendly-automation) 2017-12-13 08:30:22.904-0600 Change 7556 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7556|https://gerrit.asterisk.org/7556] By: Friendly Automation (friendly-automation) 2017-12-13 08:30:25.320-0600 Change 7557 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7557|https://gerrit.asterisk.org/7557] By: Friendly Automation (friendly-automation) 2017-12-13 08:30:31.111-0600 Change 7555 merged by George Joseph: AST-2017-012: Place single RTCP report block at beginning of report. [https://gerrit.asterisk.org/7555|https://gerrit.asterisk.org/7555]