| Summary: | ASTERISK-26478: codec_opus: Opus transcoding one-way audio issue | ||||||
| Reporter: | Luke Escude (lukeescude) | Labels: | |||||
| Date Opened: | 2016-10-17 12:08:33 | Date Closed: | 2017-01-12 19:20:27.000-0600 | ||||
| Priority: | Minor | Regression? | |||||
| Status: | Closed/Complete | Components: | Codecs/codec_opus | ||||
| Versions: | 14.0.2 | Frequency of Occurrence | |||||
| Related Issues: |
| ||||||
| Environment: | Centos x64 | Attachments: | ( 0) 2out1in.pcap ( 1) Console-1.rtf ( 2) flowroute-280984.pcap25.pcap ( 3) pcap0_3.pcap ( 4) PCAP2.pcap | ||||
| Description: | Are there any known issues with transcoding with the Opus codec from Digium?
About 1 in every 4 handsets is having one-way audio problems on Opus, during outbound calls (handset -> opus -> Asterisk 14 -> uLaw -> Trunk) It could be the handsets themselves as well, since Grandstream only recently started supporting opus. | ||||||
| Comments: | By: Asterisk Team (asteriskteam) 2016-10-17 12:08:34.082-0500 We appreciate the difficulties you are facing, however information request type issues would be better served in a different forum. The Asterisk community provides support over IRC, mailing lists, and forums as described at http://asterisk.org/community. The Asterisk issue tracker is used specifically to track issues concerning bugs and documentation errors. If this issue is actually a bug please use the Bug issue type instead. Please see the Asterisk Issue Guidelines [1] for instruction on the intended use of the Asterisk issue tracker. Thanks! [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines By: Asterisk Team (asteriskteam) 2016-10-17 12:08:34.835-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Luke Escude (lukeescude) 2016-10-17 12:09:05.310-0500 Packet Capture from a handset with one-way audio issues. By: Luke Escude (lukeescude) 2016-10-17 12:21:39.263-0500 Hopefully this comment will re-open the ticket. By: Asterisk Team (asteriskteam) 2016-10-17 12:21:39.388-0500 This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable. By: Joshua C. Colp (jcolp) 2016-10-17 14:19:50.534-0500 We're going to need to a console log as well but from the perspective of the pcap it shows opus as being negotiated, but the device in question sending ulaw instead (while we send opus). By: Luke Escude (lukeescude) 2016-10-17 14:31:17.363-0500 Attached 2 files: PCAP2 is from a handset with one-way audio on the Opus leg. Console-1 is the log capture with some channel readouts on the same phone call. By: Luke Escude (lukeescude) 2016-10-19 18:32:42.207-0500 I no longer think this is related to Opus. We ditched Opus entirely for uLaw on our testbed servers, and still got occasional one-way audio, despite no transcoding being done. No logs, no packet captures, nothing will show us why. So we're simply reverting to Ast 13 and waiting for the Opus codec to release for 13. By: Kevin Harwell (kharwell) 2016-11-02 16:40:08.331-0500 [~lukeescude] Did reverting to 13 solve your problem? Have you tried it with the opus codec and 13? By: Luke Escude (lukeescude) 2016-11-02 16:52:50.562-0500 13, 14, doesn't matter, we're having all sorts of strange random one-way audio issues. These issues didn't occur for the last year, and no matter what asterisk version I use, they still happen. It's got to be something within our infrastructure, so we may just want to close this ticket until I can at least get uLaw to work without one-way audio problems. By: Kevin Harwell (kharwell) 2016-11-02 17:41:01.134-0500 Looking at the attached pcaps there does seem to be some strangeness. Before suspending the issue I want to dig a bit more into it just in case. If possible, could you share the snippet of the dialplan where the call comes in or describe what's happening exactly? For instance it looks like you are using early media in the scenario? In the pcaps I don't see the outgoing call leg. I'm assuming in the dialplan you do answer first then? If you are using early media what happens if early media is not involved (if possible to test that scenario on your setup)? By: Luke Escude (lukeescude) 2016-11-02 17:48:37.246-0500 Very interesting ideas... I'll pull dial plan right now. Until then, here are a couple more packet captures: http://provisioning.primevox.net/flowroute-256788-cap2.pcap Toward the end, extension 102 tries to call 817-233-1397. Time: 5617.926352 http://provisioning.primevox.net/flowroute-256788-cap1.pcap "Scott <102>" to "2143954681" start time: 4515.353396. You'll see that Hardenbrook "2143954681" called him back after, and the audio was fine. I'll pull dialplan now. This issue seems to happen more on outbound calls instead of inbound calls. By: Luke Escude (lukeescude) 2016-11-02 17:49:13.637-0500 Also, what is early media? By: Luke Escude (lukeescude) 2016-11-02 17:52:10.613-0500 Trunk config: username=12345 fromuser=12345 secret=password1 host=sip.flowroute.com type=friend context=from-trunk insecure=port,invite trustrpid=yes sendrpid=yes directmedia=no qualify=yes keepalive=45 nat=force_rport,comedia dtmfmode=rfc2833 disallow=all allow=ulaw t38pt_udptl=yes,redundancy,maxdatagram=400 defaultexpiry=120 registertimeout=1 registerattempts=0 Peer Config: [101] secret=password2 dial=SIP/101 mailbox=101@voicemail callerid=101 <101> deny=0.0.0.0/0.0.0.0 permit=0.0.0.0/0.0.0.0 disallow=all allow=ulaw dtmfmode=rfc2833 canreinvite=no context=from-internal host=dynamic trustrpid=yes sendrpid=pai type=friend nat=force_rport,comedia port=5060 qualify=10000 qualifyfreq=15 transport=udp,tcp,tls avpf=no force_avp=no icesupport=no encryption=no namedcallgroup= namedpickupgroup= callcounter=yes faxdetect=no cc_monitor_policy=generic By: Luke Escude (lukeescude) 2016-11-02 17:52:58.099-0500 Inbound call dial plan: [from-trunk] exten => 14693517083,1,Answer same => n,Set(CALLERID(num)=${CALLERID(num):1}) same => n,Set(CALLERID(name)=${CALLERID(name)}) same => n,Set(CDR(cdr_application)=Inbound Call) same => n,Set(CDR(cdr_direction)=1) same => n,Set(CDR(cdr_inbound_did)=14693517083) same => n,Set(CDR(cdr_inbound_from_phonenumber)=${CALLERID(num)}) same => n,Set(CDR(cdr_inbound_from_callerid)=${CALLERID(name)}) same => n,Set(CDR(cdr_inbound_to_extension)=150) same => n,GotoIf($["${CURL("127.0.0.1/blacklist_check.php?user=0&num=${CALLERID(num)}")}" != "1"]?noblock) same => n,Busy same => n,Hangup same => n(noblock),noop same => n,Set(__inboundrec=1) same => n,Set(filepath=/var/www/html/call_recordings/${STRFTIME(${EPOCH},UTC,%Y)}/${STRFTIME(${EPOCH},UTC,%m)}/${STRFTIME(${EPOCH},UTC,%d)}/${STRFTIME(${EPOCH},UTC,%Y)}-${STRFTIME(${EPOCH},UTC,%m)}-${STRFTIME(${EPOCH},UTC,%d)}_${STRFTIME(${EPOCH},UTC,%H)}:${STRFTIME(${EPOCH},UTC,%M)}:${STRFTIME(${EPOCH},UTC,%S)}_${UNIQUEID}.wav) same => n,Set(CDR(cdr_recording_path)=${filepath}) same => n,Set(AUDIOHOOK_INHERIT(MixMonitor)=yes) same => n,MixMonitor(${filepath}) same => n,Goto(from-internal,150,1) exten => fax4693517083,1,Busy() By: Luke Escude (lukeescude) 2016-11-02 17:53:32.583-0500 Dial into an extension: exten => 101,hint,SIP/101 same => 1,Answer same => n,GotoIf($["${CURL("127.0.0.1/blacklist_check.php?user=5804f58837462&num=${CALLERID(num)}")}" != "1"]?noblock) same => n,Busy same => n,Hangup same => n(noblock),noop same => n,GoSub(recordcheck,s,1(${CALLERID(num)},101)) same => n,Set(CHANNEL(musicclass)=0) same => n,GotoIf($["${CDR(cdr_direction)}" != ""]?keepDir) same => n,Set(CDR(cdr_direction)=3) same => n(keepDir),Set(CDR(cdr_internal_from_extension)=${CALLERID(num)}) same => n,Set(CDR(cdr_internal_to_extension)=101) same => n,Set(CDR(cdr_application)=Dial) same => n,Dial(SIP/101,20,W) same => n,GotoIf($[${HANGUPCAUSE}=17]?skipA) same => n,Set(__unavailable=u) same => n,Goto(skipB) same => n(skipA),Set(__unavailable=b) same => n(skipB),noop(HANGUPCAUSE=${HANGUPCAUSE}) same => n,GotoIf($["${DIALSTATUS}" = "NOANSWER"]?101-NOANSWER,1) same => n,GotoIf($["${DIALSTATUS}" = "BUSY"]?101-BUSY,1) same => n,GotoIf($["${DIALSTATUS}" = "CHANUNAVAIL"]?101-CHANUNAVAIL,1) same => n,Goto(101-CHANUNAVAIL,1) exten => 101-CALLWAITING,1,Goto(*0101,1) exten => 101-NOANSWER,1,Goto(*0101,1) exten => 101-BUSY,1,Goto(*0101,1) exten => 101-CHANUNAVAIL,1,Goto(*0101,1) exten => _+XXXXXX./101,1,Goto(dial-out-101,${EXTEN:1},1) exten => _XXXXXX./101,1,Goto(dial-out-101,${EXTEN},1) exten => 911/101,1,Goto(dial-out-101,${EXTEN},1) exten => 9911/101,1,Goto(dial-out-101,${EXTEN},1) exten => **101,1,Pickup(101) exten => *97/101,hint,Custom:vm101 same => 1,Set(CDR(cdr_application)=Voicemail) same => n,GotoIf($["${CDR(cdr_direction)}" != ""]?keepDir) same => n,Set(CDR(cdr_direction)=3) same => n(keepDir),Set(CDR(cdr_internal_from_extension)=${CALLERID(num)}) same => n,Set(CDR(cdr_internal_to_extension)=*97/101) same => n,StopMixMonitor same => n,VoiceMailMain(101@voicemail,s) By: Kevin Harwell (kharwell) 2016-11-02 18:12:21.705-0500 {quote} Also, what is early media? {quote} Basically it allows you to play audio on a call before it is answered. When it is configured you'll see a "SIP/2.0 183 Session Progress" in the SIP traffic instead of a 180. More info here if you are interested: https://wiki.asterisk.org/wiki/display/AST/Early+Media+and+the+Progress+Application By: Kevin Harwell (kharwell) 2016-11-03 12:44:26.503-0500 Unfortunately nothing stands directly out in your dialplan or config to me. Looking at the pcaps you linked however I did notice in flowroute-256788-cap2.pcap the media between the mentioned call seems to be off. For instance call sequence starting at No. 21084: {noformat} Invite - source ip = 76.186.126.55 dest ip = 206.126.62.152 media ip = 10.20.74.143 183 Session Progress and 200 OK - source ip = 206.126.62.152 dest ip = 76.186.126.55 media = 206.126.62.152 {noformat} If I understand correctly media should flow between 10.20.74.143 and 206.126.62.152. After the 183 Session Progress (No. 21306) media appears to flow between those two addresses for a few frames. Then it appears to swap to flowing between 76.186.126.55 and 206.126.62.152 after the phone sends media to asterisk for some reason even though it negotiated a different media address. Something to try if you haven't already to potentially narrow it down to the phone: Does the problem occur with other phones? Try swapping the phone out for the same endpoint and see if the problem still happens. Also you may investigate why the media address differs from the phone in the initial invite (but that all depends on your setup). By: Kevin Harwell (kharwell) 2016-11-03 12:45:55.853-0500 Also since you think this may currently be on your end I'm going to put this issue in "waiting for feedback" and if/when you find something new or relevant then the status can change and it can be investigated more then. By: Luke Escude (lukeescude) 2016-11-03 13:29:43.985-0500 Unfortunately it's happening with both yealinks and grandstreams. Two different customers. I moved one customer to 13.12.1 with opus and they were having one way audio issues still, despite my successful test calls. They are back on uLaw for now so we'll see how long it lasts. By: Kevin Harwell (kharwell) 2016-11-09 15:15:09.980-0600 Are you still having one way audio issues? If so is it only happening when opus is in the mix or also happening with ulaw too? By: Luke Escude (lukeescude) 2016-11-09 15:28:12.867-0600 Kevin - it seems as though we've resolved the one-way audio issues in general, using uLaw. The issues were with several new customers at the same time, and we hadn't worked with their firewalls. However, we did move a beta customer to 13.12.1 and enabled opus, and he continued to have one-way audio with Opus. Unfortunately he declined further beta testing, so now it's up to me to try and recreate the issues. We don't have Pcaps from those calls. So there are still issues with Opus transcoding. By: Kevin Harwell (kharwell) 2016-11-09 16:42:30.933-0600 [~lukeescude] Thanks for the update. I'm glad to hear for the most part you have resolved the one way audio issues. Sounds like there may still be a problem with opus though. If/When you get a chance to try things out again one thing to check with regards to opus is to make sure the playback rates and bit rates/bandwidths are negotiated correctly and one phone is not sending audio at a higher rate than the receiving phone can handle. So for instance I was doing some testing with a browser->phone setup (both configured to use opus). The browser was sending audio at a higher bit rate (48000) however the phone was set to a wide bandwidth (with a bit rate of 24000) so there was no audio in one direction since the phone could not handle the higher rate. If that is the case, some of these settings can be adjusted or "capped" using configuration options. More info about opus configuration options can be found at the following: https://wiki.asterisk.org/wiki/display/AST/Codec+Opus By: Luke Escude (lukeescude) 2016-11-09 16:45:10.874-0600 Interesting - I could see that as being an issue. Is it possible to force a certain bitrate via dial plan? like allow=opus48 or whatever. Our entire system is Opus-48 only, I don't think any of our hardware supports different bitrates besides 48. By: Kevin Harwell (kharwell) 2016-11-09 17:55:13.218-0600 You can't do it via the dialplan, but you can configure it through conf files. For instance in codecs.conf something like the following needs exist: {noformat} [myopus] type=opus max_playback_rate=8000 bitrate=8000 max_bandwidth=narrow {noformat} And then in your sip.conf or pjsip.conf you'll need to specify "myopus" in the allow line for a given endpoint (note example snippet from a pjsip.conf file): {noformat} [109] type=endpoint aors=109 allow=!all,myopus {noformat} When the sdp gets negotiated between asterisk and that endpoint then the endpoint should hopefully limit sending data to that of the negotiated rates. By: Luke Escude (lukeescude) 2016-11-09 20:42:38.368-0600 Alright so now I'm reliably getting Opus to constantly have one-way audio - however only on outbound calls. An outbound call, like Handset -> Opus-8000 -> Asterisk -> uLaw-8000 -> Flowroute. The cell phone (flowroute side) can hear the Opus phone, but the Opus phone hears nothing. Inbound calls don't seem to have the issue. Additionally, core show translation shows 2 Opus columns and 2 Opus rows. By: Kevin Harwell (kharwell) 2016-11-10 09:24:11.288-0600 Can you attach pcaps of the calls (one for inbound, one for outbound)? By: Luke Escude (lukeescude) 2016-11-10 10:15:36.420-0600 Will do, soon - it was interesting though that Asterisk tried to send audio to the phone's private IP address (the phone is behind NAT, asterisk is not). The private IP audio lasted about 15 frames, then it rediverted to the normal NAT port/public IP. I'll send PCAPs soon. By: Luke Escude (lukeescude) 2016-11-10 12:47:58.446-0600 2 outbound calls with one-way audio, 1 inbound with full two-way audio. (6 streams total) 2 streams are just extraneous and can be ignored. By: Luke Escude (lukeescude) 2016-11-10 12:49:42.116-0600 You can ignore the stream with 1001 and 875, those are a result of us taking down the firewall for testing. There's definitely some weirdness going on. You can tell the server hears both sides of the call, but the Opus handset simply isn't receiving any audio. This only happens on outbound calls. It should be using 48,000 Opus since that's all we support. If you would like access to the cloud PBX, I can give you a login for the web UI and give you SSH access to get into the dial plan and whatnot. By: Kevin Harwell (kharwell) 2016-11-10 17:26:43.506-0600 Hi Luke, I took a look at the attached pcap. Maybe a few of odd things to note, but not sure if any would be the problem: 1. I never see the outgoing leg of the call, just the incoming. For instance for the call sequence starting at No. 6378 I see the initial Invite (phone->asterisk) and then the invite with auth (phone->asterisk). After that I see a 183 progress and it appears audio begins to flow between the phone and asterisk. Later Asterisk sends the 200 OK and then a BYE. 2. Around that same call sequence around No. 7054 I see an RTCP Goodbye coming from the phone. Meaning the source is no longer active. It's hard to tell if that is associated specifically with the above call sequence though. If it is then for some reason the phone source goes away, so no audio. (note it happens just after Asterisk sends the 200 OK). 3. In both the 183 and 200 OK sent from Asterisk the SDP contains an empty fmtp line for opus. This is a known issue (ASTERISK-26520) that perhaps the phone can't handle properly. That issue should be worked fairly soon so you may want to keep an eye on it for when the fix goes in to see if that helps. Unfortunately, at this time I don't have much else to recommend until there is more to go on. Except I'd suggest trying to narrow the scope a bit and start with a simple scenario if possible. For instance one that does not involve nat. That way things can start being ruled out. Also if you have another endpoint (hard/soft phone, browser) that can do opus that might tell you something as well. Hope that helps some. By: Luke Escude (lukeescude) 2016-11-10 18:40:59.934-0600 Kevin, First off, I HIGHLY appreciate the work and time you've put into this, and I know my customers do too. I'm able to see both legs of the call... Let me clarify the IP addresses for you. 207.91.156.217 --> My PBX server (testbed2.primevox.net) 97.77.119.234 --> My handset with Opus enabled, behind NAT (ubiquiti edgerouter with no SIP ALG, works flawlessly with uLaw). 216.115.69.114 --> The Flowroute SIP proxy (kamailio) - no media handled here 74.120.93.199 --> One of Flowroute's many media handling servers. Media travels here. uLaw forced. By: Luke Escude (lukeescude) 2016-11-11 11:54:20.628-0600 Also what are RTP checksums, and should they be enabled? By: Kevin Harwell (kharwell) 2016-11-11 17:15:30.159-0600 {quote} Also what are RTP checksums, and should they be enabled? {quote} I believe it used for checking if there are possible errors in a packet. From what I can tell it appears this is handled at the socket layer (i.e. the kernel checks to make sure the incoming packet checksum is equal to the calculated value). I'd gather that you would usually have this enabled. However, since it can be optionally disabled there are probably some use cases where it would make sense to do so. That of course all depends on the type of problem one is having (firewall/NATs messing up the checksum potentially, not sure though). By: Luke Escude (lukeescude) 2016-11-11 17:19:04.602-0600 Here's another packet capture with one-way audio - uLaw completely all the way through. Two inbound calls from "DAVID MENARD" around the middle of the capture. By: Luke Escude (lukeescude) 2017-01-12 19:20:27.856-0600 This is the same bug as ASTERISK-26666. | ||||||