[Home]

Summary:ASTERISK-29392: chan_iax2: Asterisk crashes when queueing video with format
Reporter:Michael Welk (michael1234)Labels:patch
Date Opened:2021-04-13 14:43:41Date Closed:2021-07-22 15:06:55
Priority:MajorRegression?
Status:Closed/CompleteComponents:Channels/chan_iax2
Versions:16.2.1 18.2.2 18.3.0 Frequency of
Occurrence		Constant
Related
Issues:
Environment:FreePBX 15.0.17.24 - Intel Xeon plattform 64bitAttachments:( 0) Asterisk.dump
( 1) ASTERISK-29392.diff
( 2) Asterisk-Session_2.zip
( 3) backtrace.txt
( 4) backtrace-old.txt
( 5) core.freepbx.sangoma.local-2021-04-15T08-45-01+0000-brief-old.txt
( 6) core.freepbx.sangoma.local-2021-04-15T08-45-01+0000-full-old.txt
( 7) core.freepbx.sangoma.local-2021-04-15T08-45-01+0000-info-old.txt
( 8) core.freepbx.sangoma.local-2021-04-15T08-45-01+0000-locks-old.txt
( 9) core.freepbx.sangoma.local-2021-04-15T08-45-01+0000-thread1-old.txt
(10) core.freepbx.sangoma.local-2021-04-15T10-15-37+0000-brief.txt
(11) core.freepbx.sangoma.local-2021-04-15T10-15-37+0000-full.txt
(12) core.freepbx.sangoma.local-2021-04-15T10-15-37+0000-info.txt
(13) core.freepbx.sangoma.local-2021-04-15T10-15-37+0000-locks.txt
(14) core.freepbx.sangoma.local-2021-04-15T10-15-37+0000-thread1.txt
Description:Hello,

i have connected 2 FreePBX Systems via Dundi/IAX2 trunk. Normal calls are working fine, but when i activate Video on the phone A (Cisco, Yaelink...) and call another phone B (phone A >Server A >IAX2> Server B> phone B) Server B crashes. When i stay with phone A+B on the same Server or disable Video on the phone - everything works perfect. This only happens via IAX2 trunks, sometimes immediatly, sometimes after a while during the call.



The configuration of the trunk is very basic

dundi_custom.conf:
{noformat}
; DK0MAV
[22:e1:bc:08:83:5c]
host=44.149.94.136
include=priv
model=symmetric
order=primary
permit=priv
qualify=yes
secret=xxxxxxxxx
{noformat}
iax_custom.conf:
{noformat}
[iaxuser]
type=friend
dbsecret=dundi/secret
context=incomingdundi
{noformat}

I already disabled videosupport on the IAX-Trunk but still the same issue.

Trace from the Asterisk-console of the crashing server:
{noformat}
Connected to Asterisk 16.2.1 currently running on freepbx (pid = 26623)
– Accepting AUTHENTICATED call from 44.225.66.196:4569:
– > requested format = g722,
– > requested prefs = disabled,
– > actual format = g722,
– > host prefs = disabled,
– > priority = disabled
– Executing [315200612183@incomingdundi:1] Playback("IAX2/iaxuser-5566", "custom/voip-dk0mav") in new stack
– <IAX2/iaxuser-5566> Playing 'custom/voip-dk0mav.g722' (language 'en')
[2021-04-12 21:49:48] WARNING[26669]: chan_iax2.c:11782 socket_process_helper: Received mini frame before first full video frame
freepbx*CLI> <<<<<<<<<<<<<<<<crash>>>>>>>>>>>>>>>
Disconnected from Asterisk server
Asterisk cleanly ending (0).
Executing last minute cleanups
{noformat}


I have tried to downgrade to an older Asterisk version but no success.

Just for clarification:

I do not want video working via IAX-Trunks, that seems not possible for what reason ever. I just want that Asterisk do not crash when it receives a call from a phone that has video enabled via the trunk.

The same configuration on earlier Asterisk versions (11, 13) is working without any problems.

It seems Asterisk have a problem when frames are lost on the trunk and they arrive in a kind of wrong order. This leads to a crash.

Maybe interesting but very old issue: https://www.coresecurity.com/core-labs/advisories/asterisk-pbx-truncated-video-frame-vulnerability

Please help me.

Best Regards

Michael
Comments:By: Asterisk Team (asteriskteam) 2021-04-13 14:43:42.621-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/].
By: Joshua C. Colp (jcolp) 2021-04-13 14:48:57.182-0500

Thank you for the crash report. However, we need more information to investigate the crash. Please provide:

1. A backtrace generated from a core dump using the instructions provided on the Asterisk wiki [1].
2. Specific steps taken that lead to the crash.
3. All configuration information necesary to reproduce the crash.

Thanks!

[1]: https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace


By: Joshua C. Colp (jcolp) 2021-04-13 14:49:31.627-0500

Additionally Asterisk 16.2.1 is very old. Please update to the latest version and try again to ensure this is still an issue.
By: Joshua C. Colp (jcolp) 2021-04-13 14:51:05.748-0500

I would also suggest seeing if setting explicit allowed codecs (allow=!all,g722) prevents the issue from occurring.
By: Michael Welk (michael1234) 2021-04-13 17:57:57.200-0500

allow=!all,g722--------done
Upgraded to Asterisk 18-----done

Nothing helps..........same problem.

By: Michael Welk (michael1234) 2021-04-13 18:26:44.503-0500

The core-dump is 120MB...upload here is limited to 50MB...

How do i send this file?
By: Joshua C. Colp (jcolp) 2021-04-13 18:29:17.021-0500

The core dump file itself is not needed at this point. If you use the ast_coredumper script that is mentioned in the linked wiki page, it will produce text files that contain the gdb output and they can be attached to this issue.

You will also need to state what version of Asterisk it is from.
By: Michael Welk (michael1234) 2021-04-13 18:35:33.896-0500

[root@freepbx ~]# sudo /var/lib/asterisk/scripts/ast_coredumper core
which: no gdb in (/sbin:/bin:/usr/sbin:/usr/bin)
No suitable gdb was found in /sbin:/bin:/usr/sbin:/usr/bin
[root@freepbx ~]#

...it does nor work.

Asterisk is version 18.2.2
By: Joshua C. Colp (jcolp) 2021-04-13 18:38:12.714-0500

The gdb package has to be installed, as that is what opens the core dump file and produces the text files.
By: Michael Welk (michael1234) 2021-04-13 18:49:26.304-0500

Coredump attached files
By: Michael Welk (michael1234) 2021-04-14 05:03:21.401-0500

Just for information: The IAX-Trunk between the 2 PBX is via a 25 Miles (yes 25!) WLAN link. So loss of packets sometimes occur.
By: Kevin Harwell (kharwell) 2021-04-14 10:46:20.808-0500

Unfortunately the attached backtrace is missing symbols, so we are unable to extrapolate exactly where it is crashing. Please ensure Asterisk is built to contain debug symbols. From [getting a backtrace|https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-PreparingAsteriskToProduceCoreFilesOnCrash]:
{quote}
This can be done by selecting the 'DONT_OPTIMIZE' option in the Compiler Flags submenu in the 'make menuselect' tree before building Asterisk.
{quote}
After Asterisk is rebuilt with symbols (and re-installed), and once another crash occurs rerun _ast_coredumper_ again and attach the new output here.

Thanks!
By: Kevin Harwell (kharwell) 2021-04-14 10:52:12.388-0500

Actually noticed you are using FreePBX. Please see the [Providing Great Debug|https://wiki.freepbx.org/display/SUP/Providing+Great+Debug] wiki page. Specifically, the [Backtraces(Segfaults/CoreDumps/AsteriskCrashing)|https://wiki.freepbx.org/display/SUP/Providing+Great+Debug#ProvidingGreatDebug-Backtraces(Segfaults/CoreDumps/AsteriskCrashing)] section about how to install debug symbols for Asterisk with FreePBX.
By: Michael Welk (michael1234) 2021-04-14 12:09:04.154-0500

New files uploaded, i did everything possible described there:

/var/lib/asterisk/scripts/ast_coredumper /tmp/core.freepbx.sangoma.local-2021-04-13T20:32:08+0000

gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c core.freepbx.sangoma.local-2021-04-13T20:32:08+0000 > /tmp/backtrace.txt

By: Michael Welk (michael1234) 2021-04-14 12:50:43.154-0500

I hope this helps. If not, please send me a binary of Asterisk with all the things you want to have included, then i just swap the binary and i do everything again.
By: Kevin Harwell (kharwell) 2021-04-14 16:11:59.742-0500

Unfortunately the new files still don't have symbols available ...and you followed the _yum_ commands listed on the wiki? If so, and you can verify the Asterisk debug packages are installed then you might want to reach out and [report it to FreePBX|http://issues.freepbx.org/].
By: Kevin Harwell (kharwell) 2021-04-14 16:13:18.048-0500

Until we can get a better backtrace can you also attach, or post, the exact iax configuration for the endpoints in question, and relevant dialplan portion(s).

As well it appears it is failing during a playback. What file is it trying to playback? A custom one? Asterisk installed one? And what format is the file being played back in?

Also could you enable debug level 5, and for good measure iax debugging. See the [collecting debug information|https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information] wiki page for more information on how to do that. Then next time it fails attach the log here to the issue. Maybe it'll show something leading up to the crash.

Thanks!
By: Kevin Harwell (kharwell) 2021-04-14 16:18:18.793-0500

{quote}
If not, please send me a binary of Asterisk with all the things you want to have included, then i just swap the binary and i do everything again.
{quote}
Unfortunately, it's not as easy as that. Asterisk is comprised of many compiled modules and depending on your OS and other environment factors a simple binary swap especially of a FreePBX system (I am not familiar with what all they use and have configured) could lead to unexpected results. Thus at this time the asterisk team at Sangoma does not compile and distribute binaries.
By: Michael Welk (michael1234) 2021-04-14 16:32:27.570-0500

The yum commands are for Version 7 and older, i have Version 15 installed.
Nope, this bug not appear during playback, mostly during a placed call. The bug has nothing to do with playback bcause when i disable video on the phone everything works perfect.
Please refer to my bug description, this happens always after the error message:

[2021-04-12 21:49:48] WARNING[26669]: chan_iax2.c:11782 socket_process_helper: Received mini frame before first full video frame

In these cases the call is placed and i can talk to phone B, after 10-60sec or so Asterisk crashes when this message arrives (asterisk -vvvr) at the console.

I will have a look at level 5 debugging and come back to you.
By: Kevin Harwell (kharwell) 2021-04-14 17:32:14.465-0500

I spoke with some folks who know and work with FreePBX, and they said you should be able to install the debug symbol packages for your version. It should just be:
{noformat}
$ yum install sangoma-devel
$ yum install pjproject-debuginfo asterisk16-debuginfo
{noformat}
And if you are using Asterisk 18 then replace the 16 with 18 in the above.

Also the reason I believe it's crashing during playback is because that's what the stack trace shows from the currently attached backtrace(s) (from [^core.freepbx.sangoma.local-2021-04-13T20-32-08+0000-brief.txt]):
{noformat}
Thread 1 (Thread 0x7f04cfe26700 (LWP 2006)):
#0  0x00000000004f6055 in ast_format_get_type ()
#1  0x00000000004a1115 in __ast_read ()
#2  0x00000000004a3224 in ast_read ()
#3  0x00000000004f3ba6 in waitstream_core ()
#4  0x00000000004f411f in ast_waitstream ()
#5  0x00007f04797dbde0 in playback_exec () from /usr/lib64/asterisk/modules/app_playback.so
#6  0x00000000005418f3 in pbx_exec ()
#7  0x000000000052d2d3 in pbx_extension_helper ()
#8  0x00000000005310df in ast_spawn_extension ()
#9  0x0000000000531d99 in __ast_pbx_run ()
#10 0x0000000000533570 in pbx_thread ()
#11 0x00000000005c6f79 in dummy_start ()
#12 0x00007f04d295fea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f04d19fe8dd in clone () from /lib64/libc.so.6
{noformat}
Here we see at #5 playback is called (_app_playback_). and then further down (or up depending on how you want to read it) the stack at #0 the actual crash happens in the call to _ast_format_get_type_. So somehow playback is involved.
By: Michael Welk (michael1234) 2021-04-15 04:04:07.522-0500

Hi,
i installed the packages as described and did a real phone call. This call lasts for about 2min, then Asterisk crashed. The other dump was from an recorded voice, maybe this was misleading  (i do not want to call users at 1am, hihi). Now playback is not involved anymore.
By: Michael Welk (michael1234) 2021-04-15 05:27:08.150-0500

I have upgraded the system and did a full restart (yum update).
Did all the things again, now the dump-files look much better.
New files just uploaded, it seems the debugging symbols are now in.
New files uploaded, deleted all the old ones.
By: Michael Welk (michael1234) 2021-04-15 06:14:21.594-0500

I inspected an older coredump with the new version (maybe it is not good because the coredump and the tools to inspect it are from different versions) but i found it is maybe usefull because i saw something like "can not access memory blabla). These files are marked with -old.txt
By: Michael Welk (michael1234) 2021-04-15 06:25:00.042-0500

Forgot to tell:
Asterisk 18.3.0 is now running
By: Joshua C. Colp (jcolp) 2021-04-15 06:43:35.035-0500

You can't examine old core dumps with a different version of Asterisk and packages, the result is incorrect.
By: Michael Welk (michael1234) 2021-04-15 06:49:32.484-0500

Thanks, that is what i thought. So please ignore the -old.txt
By: Michael Welk (michael1234) 2021-04-15 07:26:33.108-0500

I bet my ass that this has something to do with packetloss on the IAX2-Trunk and the order of the processed frames, arriving at the crashing PBX......
By: Michael Welk (michael1234) 2021-04-15 07:51:51.832-0500

18.3.0 seems to be a debugging version of Asterisk, uploaded by Sangoma...cool.

!@!@!@! info.txt !@!@!@!

Asterisk 18.3.0 built by mockbuild @ jenkins7 on a x86_64 running Linux on 2021-04-09 17:41:27 UTC

System started: 2021-04-15 10:05:29.643245
Last reload: 2021-04-15 10:05:29.643245

Build options = DONT_OPTIMIZE, COMPILE_DOUBLE, OPTIONAL_API

By: Joshua C. Colp (jcolp) 2021-04-15 08:14:35.347-0500

The chan_iax2 module appears to be providing a media frame without stating what format it is in. In the time before streams we wouldn't have examined the format in this flow, and wouldn't care.
By: Joshua C. Colp (jcolp) 2021-04-15 08:21:50.007-0500

Is it possible for you to get a packet capture of a call that causes this so we can see the specific IAX2 protocol flow?
By: Michael Welk (michael1234) 2021-04-15 09:12:32.321-0500

Uuuuuh, yes, that is possible. I have to organize a switch with a mirror-port...maybe the router (MikroTik) is also able to do that, i have to check.
On which side do you need the stream-capture, on the side of the crashing PBX (far end for me, but i think that would make sense when frames are lost) or on the other side?
By: Joshua C. Colp (jcolp) 2021-04-15 09:15:16.917-0500

On the side of the crashing PBX.
By: Michael Welk (michael1234) 2021-04-15 09:27:58.070-0500

OK, i have to organize things, that can`t be done on the fly. This takes some time..i come back to you guys asap.
Many thanks so far!
By: Michael Welk (michael1234) 2021-04-15 10:57:46.760-0500

It went faster than expected, the router is able to do packet sniffing, yeah! Attached is the communication on port 4569 during the fault of 2 seperat calls, both of them failed after a while. You can open it with Wireshark.
By: Michael Welk (michael1234) 2021-04-15 11:36:25.839-0500

44.149.94.136 is the server that is crashing.
By: Michael Welk (michael1234) 2021-04-15 14:15:28.470-0500

" In the time before streams we wouldn't have examined the format in this flow, and wouldn't care."

Is that the reason why i can`t get video working via IAX2?
By: Joshua C. Colp (jcolp) 2021-04-15 14:37:22.817-0500

I have no idea. The chan_iax2 module hasn't been touched for any of that, 'nor tested for video. I doubt it was used for video much in the first place even when support for it was added.
By: Michael Welk (michael1234) 2021-04-15 14:50:46.989-0500

Understood. So maybe it is a good idea for me to switch to a SIP-Trunk in the near future. Is it possible to get Dundi working via a SIP-Trunk? Because there is no dialplan existing in our network, no chance to say 1xxxx is location 1, 2xxxx is location 2 and so on.
Edit: I just see it is possible. I will give it a try in the future.
By: Michael Welk (michael1234) 2021-04-17 04:52:57.126-0500

Just for information:
I switched the trunk of the affected PBX from IAX to SIP.
Now everything is fine, but nevertheless, this should be solved. At least nothing crashes anymore.
By: Kevin Harwell (kharwell) 2021-05-10 18:17:49.108-0500

[~michael1234] It seems you might have converted your configuration to use chan_pjsip, but I've attached a patch ([^ASTERISK-29392.diff]) that should hopefully fix the problem with chan_iax2. If you are able to apply, test, and verify it fixes your original chan_iax2 crash it would be much appreciated. Thanks!
By: Michael Welk (michael1234) 2021-05-11 11:58:34.730-0500

Hi Kevin,

i use the FreePBX software package directly from Sangoma, no chance to apply a .diff to the source :-( ......
Unfortunatly there are a tot of packages missing to build and compile.
By: Friendly Automation (friendly-automation) 2021-07-22 15:06:56.664-0500

Change 16204 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16204|https://gerrit.asterisk.org/c/asterisk/+/16204]
By: Friendly Automation (friendly-automation) 2021-07-22 15:13:40.163-0500

Change 16206 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16206|https://gerrit.asterisk.org/c/asterisk/+/16206]
By: Friendly Automation (friendly-automation) 2021-07-22 15:17:44.132-0500

Change 16203 merged by George Joseph:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16203|https://gerrit.asterisk.org/c/asterisk/+/16203]
By: Friendly Automation (friendly-automation) 2021-07-22 15:17:56.726-0500

Change 16205 merged by George Joseph:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16205|https://gerrit.asterisk.org/c/asterisk/+/16205]
By: Friendly Automation (friendly-automation) 2021-07-22 16:16:41.884-0500

Change 16195 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16195|https://gerrit.asterisk.org/c/asterisk/+/16195]
By: Friendly Automation (friendly-automation) 2021-07-22 16:16:47.925-0500

Change 16196 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16196|https://gerrit.asterisk.org/c/asterisk/+/16196]
By: Friendly Automation (friendly-automation) 2021-07-22 16:16:52.175-0500

Change 16191 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16191|https://gerrit.asterisk.org/c/asterisk/+/16191]
By: Friendly Automation (friendly-automation) 2021-07-22 16:16:58.902-0500

Change 16192 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16192|https://gerrit.asterisk.org/c/asterisk/+/16192]
By: Friendly Automation (friendly-automation) 2021-07-22 16:17:04.616-0500

Change 16193 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16193|https://gerrit.asterisk.org/c/asterisk/+/16193]
By: Friendly Automation (friendly-automation) 2021-07-22 16:17:16.468-0500

Change 16194 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16194|https://gerrit.asterisk.org/c/asterisk/+/16194]
By: Friendly Automation (friendly-automation) 2021-07-23 08:23:16.787-0500

Change 16212 merged by Friendly Automation:
AST-2021-008 - chan_iax2: remote crash on unsupported media format

[https://gerrit.asterisk.org/c/asterisk/+/16212|https://gerrit.asterisk.org/c/asterisk/+/16212]