[Home]

Summary:ASTERISK-28499: translate: Crash when frame does not have a "src" field set
Reporter:Gregory Massel (gmza)Labels:
Date Opened:2019-08-07 10:43:42Date Closed:2019-09-05 07:55:16
Priority:MajorRegression?Yes
Status:Closed/CompleteComponents:Core/Channels
Versions:13.28.0 16.5.0 Frequency of
Occurrence
Frequent
Related
Issues:
Environment:Ubuntu 18.04.3 LTS, kernel 4.18.0-16-generic, x86_64 Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz Asterisk 13.28.0Attachments:( 0) core-brief.txt
( 1) core-brief-2.txt
( 2) core-full.txt
( 3) core-locks.txt
( 4) core-locks-2.txt
( 5) core-thread1.txt
( 6) core-thread1-2.txt
( 7) wireshark.png
Description:Asterisk segfaults randomly.

Appears to have been stable since downgrading to 13.27.1 (however, this may just be luck).

I will attach backtraces.
Comments:By: Asterisk Team (asteriskteam) 2019-08-07 10:43:43.793-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

By: Gregory Massel (gmza) 2019-08-07 10:45:45.122-0500

Backtace of one crash

By: Gregory Massel (gmza) 2019-08-07 10:46:50.091-0500

Backtrace of a subsequent crash

By: Gregory Massel (gmza) 2019-08-07 13:50:06.950-0500

I have been able to isolate the specific calls that trigger this crash.
At some point in each of the calls (usually the very moment the remote party stops playing on-hold music and actually connects to a phone), I receive a truncated RTP packet from the remote party.
As that truncated RTP packet comes in, Asterisk SegFaults.
Unfortunately for confidentiality reasons I cannot upload an entire PCAP of the calls, however, I'm uploading a Wireshark screen-shot.

By: Gregory Massel (gmza) 2019-08-07 13:50:36.628-0500

Wireshark showing truncated RTP packets

By: Kevin Harwell (kharwell) 2019-08-07 14:15:13.870-0500

[~gmza] Thanks for the report. I've temporarily restricted access to this issue due to the nature of the crash and the potential cause (could be security related). We'll reserve a full determination until later once we have more information.

By: Gregory Massel (gmza) 2019-08-07 14:34:43.409-0500

Futher to note - the issue will only occur when Asterisk is processing media.

By way of example, calls from the same FROM number and to the same TO number cause a SegFault when one leg was A-law and the other G.729a as Asterisk was forced to transcode, however, where both legs were A-law (allowing Asterisk to simply forward RTP blindly), no SegFault occurs.

I guess this is probably obvious, but I've noted it anyway because it may help to understand where in the code to look if it's known that it relates to transcoding of truncated RTP packets.

If you would like PCAPs of some the calls, please let me know how I can submit these confidentially (i.e. such that they will only be visibile to the Asterisk team and never to the public).

By: Gregory Massel (gmza) 2019-08-07 14:45:20.720-0500

I've managed to pull an almost identical call after downgrade from 13.28.0 to 13.27.1 including truncated RTP packets and transcoding from G.729a to A-law and Asterisk 13.27.1 did *NOT* segfault.

That would indicate that this is most likely a regression and should, hopefully, limit the number of changes that need to be looked at.

By: Joshua C. Colp (jcolp) 2019-08-19 13:09:27.303-0500

Do you still have the core dump for this? If so can you open it in gdb and do the following:

{noformat}
thread 1
frame 1
print *f
{noformat}

And attach the contents?

As well what is the dialplan for this? Are you using a jitterbuffer in Asterisk at all?

By: Joshua C. Colp (jcolp) 2019-08-23 09:14:50.915-0500

You can also email the pcap to me, jcolp@sangoma.com

By: Joshua C. Colp (jcolp) 2019-08-26 06:17:04.518-0500

[~gmza] I have not been able to reproduce this problem with a truncated RTP packet or through experimentation with various things. The fix will be simple, but it would be really good to know the specific case that causes the problem.

By: Friendly Automation (friendly-automation) 2019-09-05 07:55:17.761-0500

Change 12844 merged by George Joseph:
AST-2019-005 - translate: Don't assume all frames will have a src.

[https://gerrit.asterisk.org/c/asterisk/+/12844|https://gerrit.asterisk.org/c/asterisk/+/12844]

By: Friendly Automation (friendly-automation) 2019-09-05 07:55:20.618-0500

Change 12845 merged by George Joseph:
AST-2019-005 - translate: Don't assume all frames will have a src.

[https://gerrit.asterisk.org/c/asterisk/+/12845|https://gerrit.asterisk.org/c/asterisk/+/12845]

By: Friendly Automation (friendly-automation) 2019-09-05 07:55:23.907-0500

Change 12843 merged by George Joseph:
AST-2019-005 - translate: Don't assume all frames will have a src.

[https://gerrit.asterisk.org/c/asterisk/+/12843|https://gerrit.asterisk.org/c/asterisk/+/12843]