Summary:	ASTERISK-23262: Audio degredation with codec_dahdi and ChanSpy'ing
Reporter:	Sean Bright (seanbright)	Labels:
Date Opened:	2014-02-07 11:35:15.000-0600	Date Closed:	2017-05-31 14:56:43
Priority:	Major	Regression?
Status:	Closed/Complete	Components:	Codecs/codec_dahdi
Versions:	SVN	Frequency of Occurrence	Constant
Related Issues:
Environment:	Linux pbx-host 3.5.0-45-generic #68~precise1-Ubuntu SMP Wed Dec 4 16:18:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux Dell PowerEdge R620	Attachments:	( 0) 0001-wip-codec_dahdi-Add-configurable-synchronous-mode.patch ( 1) irc.txt ( 2) rattle.sln
Description:	The server has 2 TCE400P cards and I'm using the latest wctc4xxp driver from the DAHDI-Linux git repository (master branch). There are 3 channels involved in this scenario: * {{SIP/from-internal-user/1}} * {{SIP/from-external-user/1}} * {{SIP/from-spying-user/1}} The native format for all 3 of these channels is G729. {{SIP/from-internal-user/1}} enters dialplan and goes immediately into {{MusicOnHold}}.  The {{MusicOnHold}} class consists of only .g729 files, so no encoders/decoders are used at this point. {{SIP/from-external-user/1}} enters dialplan, and hit these three apps/functions in this order: * {{MixMonitor(<somefile>.sln,...)}} * {{VOLUME(RX)=2}} * {{Bridge(SIP/from-internal-user/1,...)}} Now, the third channel channel, {{SIP/from-spying-user/1}}, enters dialplan and goes into {{ChanSpy}}: * {{ChanSpy(SIP/from-internal-user/1,...)}} At this point, both {{SIP/from-internal-user/1}} and {{SIP/from-external-user/1}} begin to hear distortion/delay and are unable to understand each other.  There is also distortion/delay observed by {{SIP/from-spying-user/1}}. Spying on {{SIP/from-external-user/1}} does not cause the audio degredation.  Also, based on a suggestion by Josh Colp, removing the call to {{VOLUME(RX)=2}} appears to resolve the problem as well.
Comments:	By: Sean Bright (seanbright) 2014-02-07 11:39:39.564-0600 This is the log of some relevant discussion from #asterisk-dev on the 6th and 7th of February 2014. By: Shaun Ruffell (sruffell) 2014-02-07 11:56:40.928-0600 Just putting my own comment on this issue from the discussions:  The current hypothesis (thanks jcolp) is that when the VOLUME(RX) call is on the channel there are potentially 3 calls to translate the audio in the path. The need to block on the hardware is accumulating which is not playing well with the jitter buffers on the end of the audio paths in Asterisk. I wanted to look over the path to see if there is any way to eliminate blocking so much on the hardware in this case. By: Shaun Ruffell (sruffell) 2014-02-12 21:13:18.049-0600 I've attached a patch that I think is worth trying. I've done some basic tests using the file covert utility. This patch allows the synchronous mode of codec_dahdi to be disabled and for frames with silence to be returned when the hardware doesn't have frames to send yet. This results in some bad frames at the beginning of a call, but once the pump is primed, things should be fine. This should prevent codec_dahdi from blocking the translation chain. This patch can be applied from the root of an asterisk-11 source directory like: {noformat} curl "https://issues.asterisk.org/jira/secure/attachment/49280/0001-wip-codec_dahdi-Add-configurable-synchronous-mode.patch" | patch -p1 {noformat} After applying the patch, you can enable the non-synchronous mode of operation by running {{asterisk -rx 'transcoder set synchronous off'}} or by adding a {{synchronous=off}} line in the {{[dahdi]}} section in codecs.conf. When synchronous mode is disabled, file convert will produce a file that doesn't have the correct audio since it will feed all the audio frames into the transcoder faster than the hardware can transcode them. Although I know both modes will work since if I add a 20ms delay in res_convert.c between each frame written, the audio is correct. The thought is that disabling the synchronous mode of codec_dahdi will prevent interfering with any jitter buffers in the translation path. By: Sean Bright (seanbright) 2014-02-14 07:41:33.590-0600 I have this running this morning and will let you know what happens.  Seeing this currently on the console: {noformat} -- Silent frames generated internally: 23. -- Silent frames generated internally: 3. -- Silent frames generated internally: 41. -- Silent frames generated internally: 2. {noformat} I don't know if those numbers are good or bad. By: Sean Bright (seanbright) 2014-02-14 07:44:47.283-0600 I had to disable this patch.  The initial part of the recording has a loud "rattling" sound.  I will attach an example. By: Sean Bright (seanbright) 2014-02-14 07:47:35.012-0600 Attached 'rattle.sln' - what the first half second of each recording sounds like. By: Shaun Ruffell (sruffell) 2014-02-14 09:16:13.011-0600 23 and 41 silent frames are definitely not what I would have expected. Normally I would think if packets are coming into the system at 20 ms intervals, you should only get one or two silent frames until the pump was primed, then there would always be a one-to-one correlation between a ready frame in and a frame from the hardware. So this makes me wonder if there are packets that are coming in at 40 or 60ms, and then are being smoothed out into 2 or 3 frames of 20 ms. So ugh, I guess I'm really going to have to try and setup the scenario as you described it and take my own recordings through chanspy when there is a VOLUME() on the channel. Can you confirm what the packet interval is from the end points? By: Sean Bright (seanbright) 2014-02-14 09:23:53.034-0600 The {{SIP/from-internal-user/1}} packets are G729 from a transcoder card in another machine, so I assume 20ms, but I will confirm.  {{SIP/from-external-user/1}} is pass-through from upstream, so I will check there as well. By: Shaun Ruffell (sruffell) 2014-02-14 11:17:51.097-0600 Another quick question, rattle.sln was from the first half second, but after that things sounded OK? By: Sean Bright (seanbright) 2014-02-14 11:20:53.842-0600 Correct. By: Sean Bright (seanbright) 2014-02-14 11:31:04.451-0600 I've confirmed that the RTP coming from {{SIP/from-external-user/1}} is arriving at 20ms intervals.  The RTP from Asterisk going to {{SIP/from-external-user/1}} is not quite as consistent, it's between 18ms and 22ms. By: Shaun Ruffell (sruffell) 2014-03-03 15:11:32.773-0600 Nothing new to report. Just logging on the issue this is back in my court. I need to setup the exact scenario and trace the audio path. By: Shaun Ruffell (sruffell) 2014-05-21 10:23:21.749-0500 Seanbright, There are about [25 new patches|http://git.asterisk.org/gitweb/?p=dahdi/linux.git;a=history;f=drivers/dahdi/wctc4xxp/base.c;h=6e2698] on the master branch for the transcoder.  I don't believe they will completely resolve this issue, but I would be interested in hearing if it changes anything for you. By: Shaun Ruffell (sruffell) 2014-06-02 08:46:38.096-0500 FYI, The dahdi-linux master branch was [updated again last week|http://git.asterisk.org/gitweb/?p=dahdi/linux.git;a=shortlog;h=624f30bbf6a98e2b88]. Another user of the transcoder card noticed that audio latency was increasing with the previous patchset after a certain number of calls were placed in conference. There is *still* the question of figuring out how to take out unnecessary transcoding paths. I.e. Drop a user into meetme, and there are 2 encoders / 2 decoders used when it seems like it should only be 1/1.