Summary:	ASTERISK-26924: Codec OPUS bad quality
Reporter:	TSAREGORODTSEV Yury (tsarik)	Labels:
Date Opened:	2017-04-05 12:05:48	Date Closed:	2017-10-24 09:51:55
Priority:	Major	Regression?
Status:	Closed/Complete	Components:	Codecs/codec_opus
Versions:		Frequency of Occurrence
Related Issues:
Environment:	Ubuntu 16.04 i386/x86_64	Attachments:	( 0) kharwell-audio.tar.gz
Description:	Codec OPUS developed by Digium doesn't provide same good quality as open source version in case of packet losses, delays. After comparing recorded degraded media with original sample using PESQ results shows average predicted MOS of Digium Opus almost twice less then open source. How to reproduce issue: 1. Environment: ubuntu 16.04 2. Libopus latest stable release 1.1.4 (compiled from sources). 3. Asterisk 13.14.0 4. Channel driver: chan_sip 5. Configure 2 asterisk hosts, where 1st host origination side, 2nd host termination side. 6. Configure dialplan: on 1st host Dialplan with AppMonitor {noformat} [opus-test] exten => _X.,1,Monitor(wav,/records/degraded-ID-${RAND(1,1000)},bi) exten => _X.,n,Dial(SIP/asterisk_host2/${EXTEN}) {noformat} 7. Configure dialplan: on 2nd host Playback any wave file (8000, 16bit): {noformat} [opus-test] exten => _X.,1,Answer() exten => _X.,n,Playback(demo-instruct) {noformat} 8. Opus config on both: {noformat} [opus] type=opus packet_loss=40 max_playback_rate=48000 bitrate=24000 cbr=0 fec=1 {noformat} 9. Compile ITU-T PESQ utility from https://github.com/dennisguse/ITU-T_pesq 10. Simulate packet loss/delays on 2nd host with    tc qdisc add dev eth0 root netem loss 20% delay 100ms 20ms distribution normal 11. Compare recorded degraded media (only incoming channel of course) with original: ./itu-t-pesq2005 +8000 original.wav recorded.wav P.S. Please keep in mind without applying patch ASTERISK-25629 asterisk doesn't ignore lately arrived RTP no matter of enabled, forced jitterbuffer. Which is make quality worst.
Comments:	By: Asterisk Team (asteriskteam) 2017-04-05 12:05:49.164-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Kevin Harwell (kharwell) 2017-04-07 11:13:55.231-0500 I ran some tests with the same criteria using two host machines. Unfortunately though I am unable to use the PESQ tool at this time as it requires a license. So all my testing was done by ear. However, there were significant enough differences in audio quality when testing various scenarios that made it easy enough to know if something was working or not. I used the following settings: ubuntu 16.04 libopus 1.1.4 and opusfile 0.7 compiled from source Asterisk 13.14.0 using chan_sip Digium Opus codec: 1.1.0 [Open source Asterisk codec|http://github.com/traud/asterisk-opus/archive/asterisk-13.7.tar.gz] host1's _sip.conf_: {noformat} [general] sipdebug=yes nat=force_rport ;jbenable=yes ;jbforce=yes [endpoint_t](!) type=friend context=default nat=force_rport directmedia=no [host2](endpoint_t) host=<host2's IP address> fromuser=host1 allow=!all,opus {noformat} When applicable I would un/comment the "jbenable" and "jbforce" settings. host1's _extensions.conf_: {noformat} [default] exten => 997,1,Noop()      same => n,Monitor(wav,/tmp/degraded-ID-${RAND(1,1000)},bi)      same => n,Dial(SIP/host2/999)      same => n,Hangup() {noformat} host2's _sip.conf_: {noformat} [general] sipdebug=yes nat=force_rport [endpoint_t](!) type=friend context=default nat=force_rport directmedia=no [host1](endpoint_t) host=<host1's IP address> fromuser=host2 allow=!all,opus {noformat} host2's _extensions.conf_: {noformat} exten => 999,1,Answer()      same => n,Playback(demo-instruct)      same => n,Hangup() {noformat} host1's and host2's _codec.conf_ (note these settings only apply for the digium opus codec): {noformat} [opus] type=opus packet_loss=40 max_playback_rate=48000 bitrate=24000 cbr=0 fec=1 ;max_bandwidth=medium {noformat} Below is a table of the testing I did. For each test I would initiate a call from an endpoint/phone that called extension 997 on host1. The table is a map of which codec implementation the a host was running. All combinations sounded good without delay and loss applied. On host2 I issued the following command to simulate packet loss with delay: {noformat} tc qdisc add dev eth0 root netem loss 20% delay 100ms 20ms distribution normal {noformat} OK = Intelligible with some audio anomalies, but what one would expect from 20% packet loss Bad = Unintelligible, poor audio quality. jb = Jitter buffer ||open source opus codec||open source opus codec with plc patch||digum opus codec||digum opus codec with jb||Result|| |host2 & host1| | | |Bad| |host2|host1| | |Not great, but not bad| |host2| |host1| |Bad| |host2| | |host1|OK| |host1| |host2| |Bad| |host1 (with jb)| |host2| |OK| | |host1|host2| |Bad| | |host1 (with jb)|host2| |OK| | | |host1 & host2| |Bad| | | |host2|host1|OK| However, using the above _codec_.conf settings it is doubtful the digium codec was actually encoding FEC. This is because along with setting the "fec" and "packet_loss" options the "max_bandwidth" must also be set to medium or narrow. That being said I saw the best results when using a jitter buffer and I uncommented/set the max_bandwidth using the digium opus codec. While there were still noticeable audio anomalies (at 20% packet loss you are going to get some) the audio did sound better than without FEC. By: Kevin Harwell (kharwell) 2017-04-07 11:16:07.184-0500 [~tsarik], It appears we are getting slightly different results. It would be good to know why. I tried to use similar settings, but does anything stand out in my config that is different from your setup? A few other things to note I did. I compiled the open source codec using the Makefile found in the parent directory. Meaning I did not patch Asterisk with regards to the codec, but I did patch Asterisk with the "enable_native_plc.patch" when testing that scenario. For the Digium codec I selected it from within "menuselect". I'm currently thinking that the obvious culprit is the packet delay. Delayed packets are not good when translating. Enabling a jitter buffer seems to have alleviated this problem. When the jitter buffer was in use a noticeable improvement was heard. Also I did some testing without packet delay, but packet loss only and the audio sounded similar to when the jitter buffer was in use. By: TSAREGORODTSEV Yury (tsarik) 2017-04-08 06:58:43.636-0500 I'm surprised that you got different results, all configs almost identical with mine, please add debug for lost packet cases in source of "open source opus" to be sure that codec is really processing correctly FEC and PLC patch working as well, silly question but anyway, are you sure you don't have both modules loaded same time ? As I mentioned in my previous results quality it's almost perfect with open source edition. Do you want we both share audio records? Regarding PESQ, let me disagree, why you have issues with license? it's open source and published by ITU-T. http://www.itu.int/rec/T-REC-P.862-200511-I!Amd2/en but in git it has small patches that prevent compilation as sources quite old. By: Kevin Harwell (kharwell) 2017-04-10 13:11:47.423-0500 {quote} please add debug for lost packet cases in source of "open source opus" to be sure that codec is really processing correctly FEC and PLC patch working as well, silly question but anyway, are you sure you don't have both modules loaded same time ? {quote} heh, not a silly question as it's an easy thing to do! I did make sure though that only a single module was loaded at a time. Also, per your suggestion I instrumented the code a bit and the PLC patch appears to be working. {quote} Do you want we both share audio records? {quote} Yes, I think this would be prudent. I have attached some audio files in "kharwell-audio.tar.gz". There should be a file for each of the following for both codecs: baseline, loss & delay, load & delay with jitter buffer. {quote} Regarding PESQ, let me disagree, why you have issues with license? it's open source and published by ITU-T. http://www.itu.int/rec/T-REC-P.862-200511-I!Amd2/en but in git it has small patches that prevent compilation as sources quite old. {quote} I had referenced these pages with regards to licensing:  http://www.opticom.de/licensing/pesq.php  http://www.pesq.org/ By: Kevin Harwell (kharwell) 2017-04-10 13:25:25.120-0500 An update that could be the difference: In my testing I dialed extension 997@host1 from a phone using ulaw. I was then listening to the audio coming out of the phone. For a number of the cases it did not sound good. However, if I listened to the associated recorded .wav file the audio sounded much better for some of the cases. Because of this I updated my original chart in the comment above to reflect that. For each case where the jitter buffer was in use the audio sounded "OK". Meaning it was not the best audio, but it was intelligible. There was a difference too with the open source opus codec with the PLC patch. Again, the audio was noticeably degraded, but was still intelligible. It seems to me that the problem is the packet delay. Which makes sense since the Asterisk translate core does not handle delayed packets. The jitter buffer improved the audio of course since it is meant to handle that scenario. Likewise the open source plc patch improved the audio since it too attempts to handle that scenario as well. By: Kevin Harwell (kharwell) 2017-10-24 09:51:55.148-0500 Suspended due to lack of activity. This issue can be reopened or looked into further if/when more feedback is given.