[Home]

Summary:ASTERISK-26880: Asterisk crashes when multiple speex users join confbridge with pp_vad and dtx enabled
Reporter:Kirsty Tyerman (ktyerman)Labels:
Date Opened:2017-03-16 17:43:09Date Closed:2017-03-21 17:25:43
Priority:MajorRegression?
Status:Closed/CompleteComponents:Core/Bridging
Versions:13.9.1 14.3.0 Frequency of
Occurrence		Constant
Related
Issues:
duplicatesASTERISK-26761 SIGSEGV when user joins confbridge with speex and dtx is active
is duplicated byASTERISK-25269 Speex VAD: SIGSEGV when softmixing stasis-bridges
Environment:Fedora 25Attachments:( 0) backtrace_13.10.txt
( 1) backtrace_14.3.txt
( 2) backtrace.txt
( 3) backtrace-13.14-git.txt
( 4) backtrace-13-git-9613391.txt
( 5) codecs.conf
( 6) config-13.14-git.zip
( 7) debug_log_26880.txt
( 8) extensions.conf
( 9) Screenshot_from_2017-03-16_16-43-50.png
(10) sip.conf
Description:I have compiled Asterisk 14.3.0 on a Fedora 25 Workstation.

I have configured speex in asterisk to enable preprocessing and preprocessing voice activity detection. I have two sip clients (linphones) using speex. When the two sip clients join the same Confbridge asterisk crashes with a segmentation fault with the following output (after running asterisk -vvvg -c):
{noformat}
warning: The VAD has been replaced by a hack pending a complete rewrite
warning: The VAD has been replaced by a hack pending a complete rewrite
FRACK!, Failed assertion user_data is NULL (0) at line 5727 in ast_set_write_format of channel.c
[Mar 16 18:30:32] ERROR[31673][C-00000001]: channel.c:5727 ast_set_write_format: FRACK!, Failed assertion user_data is NULL (0)
   -- <CBAnn/3000-00000000;1> Playing 'confbridge-join.gsm' (language 'en')
Got 22 backtrace records
#0: [0x62966c] asterisk(__ast_assert_failed+0x8d) [0x62966c]
#1: [0x45ef3f] asterisk() [0x45ef3f]
#2: [0x45f816] asterisk(__ao2_ref+0x89) [0x45f816]
#3: [0x533d2c] asterisk(__ast_format_cap_append+0xa7) [0x533d2c]
#4: [0x4c3e96] asterisk(ast_set_write_format+0x70) [0x4c3e96]
#5: [0x4c29ae] asterisk(ast_write+0x10aa) [0x4c29ae]
#6: [0x48a0f5] asterisk() [0x48a0f5]
#7: [0x48a7ee] asterisk() [0x48a7ee]
#8: [0x48af54] asterisk(bridge_channel_internal_join+0x558) [0x48af54]
#9: [0x47039c] asterisk(ast_bridge_join+0x2c1) [0x47039c]
#10: [0x7f53456fa9b5] /usr/lib/asterisk/modules/app_confbridge.so(+0xb9b5) [0x7f53456fa9b5]
#11: [0x5a2952] asterisk(pbx_exec+0x119) [0x5a2952]
#12: [0x58ed5b] asterisk() [0x58ed5b]
#13: [0x592947] asterisk(ast_spawn_extension+0x50) [0x592947]
#14: [0x593580] asterisk() [0x593580]
#15: [0x594ca3] asterisk() [0x594ca3]
#16: [0x62663e] asterisk() [0x62663e]
Segmentation fault (core dumped)
{noformat}

This error is also caused when dtx is enabled in codecs.conf.

When pp_vad and dtx is disabled in codecs.conf, asterisk will not crash.

Attached are the asterisk config files that were configured to produce the error and a screen grab of the asterisk console after crash.


*STEPS TO REPRODUCE*
1. dnf install asterisk-13.9.1
2. use configuration files supplied in attatchments
3. configure two sip phones using speex and register to the asterisk server using sip accounts in sip.conf (may have to change sip bindaddr)
4. dial each sip phone into confbridge 3000

*UPDATE*
Please see further attached items, including a backtrace and an asterisk log. I have compiled asterisk from the git repository (commit b05d2fda0c8b3473c3d6d7bd1cc0473e2728b744) with the debugging flags on to obtain the backtrace.

bridge_channel.c:2348, is the source of the problem. It is not handling AST_FRAM_VOICE in switch statement correctly. Asterisk is caused to crash due to format being NULL.

Applying patch.txt to main/bridge_channel.c does not cause asterisk to crash when multiple speex users join a confbridge.

Comments:By: Asterisk Team (asteriskteam) 2017-03-16 17:43:10.472-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Kirsty Tyerman (ktyerman) 2017-03-16 17:49:55.499-0500

Attached are the asterisk config files that were configured to produce the error.
By: Kirsty Tyerman (ktyerman) 2017-03-16 17:51:21.017-0500

Attached is a screenshot of asterisk console after crash.
By: Kirsty Tyerman (ktyerman) 2017-03-16 22:11:28.523-0500

This issue is also being seen on asterisk 13.9.1 on Fedora 25 (from the Fedora repository).
By: Rusty Newton (rnewton) 2017-03-17 14:53:45.797-0500

Thank you for the crash report. However, we need more information to investigate the crash. Please provide:

1. A backtrace generated from a core dump using the instructions provided on the Asterisk wiki [1].
2. Specific steps taken that lead to the crash.
3. All configuration information necesary to reproduce the crash.

Thanks!

[1]: https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace


By: Rusty Newton (rnewton) 2017-03-17 14:56:37.356-0500

Thanks for the report. Please grab an additional backtrace as described in the linked documentation using the correct compiler flags and gdb commands, then attach it to the issue as a .txt file.

Grab an Asterisk log at the same time: https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information


By: Rusty Newton (rnewton) 2017-03-20 09:11:53.095-0500

[~ktyerman] , I had to delete your patch as it wasn't submitted under license agreement.

https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process

Please verify you have signed a license agreement and then re-submit your patch and select "mark as contribution".

You can then submit it to Gerrit and push it through the review process.

Also, thanks for all the extra debug!
By: Jens Vogler (JensV) 2017-03-20 10:49:44.844-0500

Attaching my backtrace as per instructions on the wikis.

Exactly the same issue reproduced on Asterisk 13.10.0
By: Jens Vogler (JensV) 2017-03-20 11:17:58.652-0500

Reproduced the same issue on Asterisk 14.3.0, attaching my backtrace.
By: Sean Bright (seanbright) 2017-03-20 12:10:20.860-0500

I'm trying to reproduce with Asterisk 13 from Git and I am not currently able to do so. Can you try with Asterisk 13 (or 14) from Git and let us know if you can reproduce there?

By: Jens Vogler (JensV) 2017-03-20 13:57:56.995-0500

Attached backtrace on version 13.14 directly built from the git repository. Also added my relevant configuration.

Steps to reproduce with my configuration:
Open two SIP calls to the number 11 and asterisk will crash.
By: Sean Bright (seanbright) 2017-03-20 13:59:57.196-0500

[~JensV], to clarify, you built against the latest Asterisk 13 branch in Git or you built against the 13.14 tag in Git?
By: Jens Vogler (JensV) 2017-03-20 14:01:22.547-0500

[~seanbright] I built against the 13.14 branch.
By: Sean Bright (seanbright) 2017-03-20 14:03:19.368-0500

[~JensV], please try against the 13 branch specifically. This is the branch that will eventually become 13.15.
By: Sean Bright (seanbright) 2017-03-20 14:06:14.867-0500

[~JensV], and for what it's worth, I've already done that (built against the 13 branch) and was unable to reproduce. I wouldn't ask you to do busy work if I could reproduce myself.
By: Sean Bright (seanbright) 2017-03-20 14:13:00.478-0500

[~JensV], delay that, I was just able to reproduce. The key (for me) was to actually speak once the caller entered the bridge. Will take a look.
By: Jens Vogler (JensV) 2017-03-20 14:15:57.355-0500

Attached backtrace. Asterisk built from git against branch 13. Rev 9613391
By: Jens Vogler (JensV) 2017-03-20 14:16:32.473-0500

Well.. What's done is done
By: Sean Bright (seanbright) 2017-03-20 16:33:27.135-0500

[Proposed fix is here|https://gerrit.asterisk.org/#/c/5262] if you want to try it out.
By: Kirsty Tyerman (ktyerman) 2017-03-20 18:45:23.288-0500

Awesome, I have tested the patch against asterisk 13.9.1 and 14.3.0 and asterisk is no longer crashing. Thanks!
By: Friendly Automation (friendly-automation) 2017-03-21 17:25:43.717-0500

Change 5263 merged by Joshua Colp:
bridge_softmix: Ignore non-voice frames from translator

[https://gerrit.asterisk.org/5263|https://gerrit.asterisk.org/5263]
By: Friendly Automation (friendly-automation) 2017-03-21 18:23:38.675-0500

Change 5264 merged by Joshua Colp:
bridge_softmix: Ignore non-voice frames from translator

[https://gerrit.asterisk.org/5264|https://gerrit.asterisk.org/5264]
By: Friendly Automation (friendly-automation) 2017-03-21 18:32:55.372-0500

Change 5262 merged by zuul:
bridge_softmix: Ignore non-voice frames from translator

[https://gerrit.asterisk.org/5262|https://gerrit.asterisk.org/5262]