| Summary: | ASTERISK-27239: PJSIP losing RTP connection (no more audio) | ||||
| Reporter: | Frederic Steinfels (fredo) | Labels: | |||
| Date Opened: | 2017-09-01 05:34:12 | Date Closed: | 2018-03-07 12:58:09.000-0600 | ||
| Priority: | Minor | Regression? | No | ||
| Status: | Closed/Complete | Components: | Channels/chan_pjsip | ||
| Versions: | 13.17.1 14.6.1 15.0.0-beta1 | Frequency of Occurrence | Constant | ||
| Related Issues: |
| ||||
| Environment: | independent | Attachments: | ( 0) log.txt | ||
| Description: | 1. The specific steps or actions you took that caused you to encounter the problem.
Placing an outbound call 2. The behavior you expected and the location of documentation that led you to that expectation. I expect there to be audio when placing a call. 3. The behavior you actually encountered. There is no audio in case of a codec change. Details below. Using PJSIP together with the biggest telecom provider in Switzerland, Swisscom, in conjunction with there "Smart Business Connect Trunk" solution, there is a scenario resulting in losing the audio connection. First, the call is initiated and RTP packets are exchanged (code alaw). After a few seconds, the call is diverted for some reason and Swisscom is changing to g722. At this point, PJSIP or asterisk will not reinitiate the RTP connection as it is supposed to. If a call is picked up normally, swisscom knows the correct codec from the beginning so there is no codec change after ringing. That works perfectly. However if the call is diverted to an answering machine or another party that requires a different codec, the problem occurs. When I am talking about diversion, I mean diversion + call pickup using a different codec. I have complained at Swisscom, they are telling me its my (asterisks) fault. Their response was (translated to english): In the worst case, BTN is starting 17:13: RTP. As soon as the ringing is sent, the MS is turned on and is playing the ringing. As soon as the BTN is taking the call, the P4_eMRS_NNI2 is starting the RTP towards the ATN and is reinitiating it. However, the ATN is only starting the RTP but not reinitiating it. ATN propably means Party A or calling party, ie me. BTN probably means Party B or called party. If there is a diversion without a change of codec, the audio is not lost. If the call is initiated with g722 from the beginning, the audio is available. I can not force Swisscom to only use g722 to avoid the problem. If I do so, the call can not be initiated. If you want to try this, please PM me to so I can give you a phone number that behaves this way. Here are some logs: {noformat} -- Executing [s@outboundfst:14] Dial("SCCP/Frederic-000006DE", "PJSIP/xxxx@swisscomTestTrunk,180,gIT") in new stack -- Called PJSIP/xxxx@swisscomTestTrunk -- Connected line update to SCCP/Frederic-000006DE prevented. -- PJSIP/swisscomTestTrunk-000003ea is making progress passing it to SCCP/Frederic-000006DE == Using SCCP RTP TOS bits 184 == Using SCCP RTP CoS mark 6 > 0x7f2150013300 -- Probation passed - setting RTP source address to 192.168.99.166:27936 {noformat} RTP Packets are flowing (192.168.99.19 is swisscom): {noformat} Sent RTP packet to 192.168.99.19:24496 (type 09, seq 006312, ts 1155988568, len 000160) Got RTP packet from 192.168.99.19:24496 (type 09, seq 028521, ts 3235640619, len 000160) {noformat} at the time of the diversion, this happens: {noformat} -- PJSIP/swisscomTestTrunk-000003ea requested media update control 26, passing it to SCCP/Frederic-000006DE -- PJSIP/swisscomTestTrunk-000003ea answered SCCP/Frederic-000006DE -- SEP001F9EAC4817: (sccp_pbx_answer) Outgoing call SCCP/Frederic-000006DE has been answered by remote party -- Channel PJSIP/swisscomTestTrunk-000003ea joined 'simple_bridge' basic-bridge <64e1db67-f435-4393-a360-05e59ef59887> -- Channel SCCP/Frederic-000006DE joined 'simple_bridge' basic-bridge <64e1db67-f435-4393-a360-05e59ef59887> {noformat} From this point, there are no further "Got RTP packet from" messages from the corresponding IP. It does not matter, if the call is made from a PJSIP or SCCP device (tried both). The corresponding SIP headers in case of the diversion and codec change: {noformat} -- PJSIP/swisscomTestTrunk-0000000d requested media update control 26, passing it to SCCP/Frederic-0000001E <--- Received SIP response (1022 bytes) from UDP:192.168.99.19:5060 ---> SIP/2.0 200 OK Via: SIP/2.0/UDP 192.168.99.10:5060;rport;branch=z9hG4bKPjf91cf3c1-f62b-44bb-b336-c7556481f67f From: "xxx.ch" <sip:+41435440808@192.168.99.10>;tag=db836e4b-0a44-4a95-ad4e-93a8d885393c To: <sip:xxxx@192.168.99.19>;tag=D590A070-1660 Date: Tue, 08 Aug 2017 10:04:31 GMT Call-ID: 329cc7ac-938a-40a5-87d4-17733d86220a CSeq: 12944 INVITE Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER Allow-Events: telephone-event Contact: <sip:xxxx@192.168.99.19:5060> Supported: replaces Supported: sdp-anat Server: Cisco-SIPGateway/IOS-15.5.2.16.T Supported: timer Content-Type: application/sdp Content-Disposition: session;handling=required Content-Length: 271 v=0 o=CiscoSystemsSIP-GW-UserAgent 1121 8511 IN IP4 192.168.99.19 s=SIP Call c=IN IP4 192.168.99.19 t=0 0 m=audio 21872 RTP/AVP 9 101 c=IN IP4 192.168.99.19 a=rtpmap:9 G722/8000 a=fmtp:9 bitrate=64 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ptime:20 <--- Transmitting SIP request (428 bytes) to UDP:192.168.99.19:5060 ---> ACK sip:xxxx@192.168.99.19:5060 SIP/2.0 Via: SIP/2.0/UDP 192.168.99.10:5060;rport;branch=z9hG4bKPj3335f178-728b-47a0-b3ae-a187da37fb92 From: "xxx.ch" <sip:+41435440808@192.168.99.10>;tag=db836e4b-0a44-4a95-ad4e-93a8d885393c To: <sip:xxxx@192.168.99.19>;tag=D590A070-1660 Call-ID: 329cc7ac-938a-40a5-87d4-17733d86220a CSeq: 12944 ACK Max-Forwards: 70 User-Agent: Asterisk PBX Content-Length: 0 {noformat} The difference is: g722: a=rtpmap:9 G722/8000 a=fmtp:9 bitrate=64 not g722: a=rtpmap:8 PCMA/8000 As I said before, if the codec remains g711, no problem. If changed to g722, the call will fail. | ||||
| Comments: | By: Asterisk Team (asteriskteam) 2017-09-01 05:34:14.323-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Joshua C. Colp (jcolp) 2017-09-01 05:42:31.994-0500 You'll need to provide a packet trace for this (using Wireshark or tcpdump) and also additional information - such as, did this just start occurring when you upgraded? As well as complete console output. By: Frederic Steinfels (fredo) 2017-09-01 05:46:51.585-0500 This problem occured when swisscom allowed g722 between different VOIP parties and their mobile network about half a year ago. I have tried asterisk 13, 14 and 15. No difference. Please point me to some wireshark or tcpdump parameters so that I can catch everything you need. And if we are talking about this: there is a second bug in asterisk/pjsip that I could avoid by supplying the 'T' flag to the dial string preventing asterisk to use the native bridge which also has no audio AT ALL. However using 'T' to force the simple bridge at least has audio. If this bug here can be resolved, I will place a second bug report for the bridge issue but I just wanted to explain why I am using the 'T' option. By: Joshua C. Colp (jcolp) 2017-09-01 05:53:03.663-0500 Thank you for taking the time to report this bug and helping to make Asterisk better. Unfortunately, we cannot work on this bug because your description did not include enough information. Please read over the Asterisk Issue Guidelines [1] which discusses the information necessary for your issue to be resolved and the format that information needs to be in. We would be grateful if you would then provide a more complete description of the problem. At a minimum, we need: 1. The specific steps or actions you took that caused you to encounter the problem. 2. The behavior you expected and the location of documentation that led you to that expectation. 3. The behavior you actually encountered. To demonstrate the issue in detail, please include Asterisk log files generated per the instructions on the wiki [2]. If applicable, please ensure that protocol-level trace debugging is enabled, e.g., 'sip set debug on' if the issue involves chan_sip, and configuration information such as dialplan and channel configuration. Thanks! [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines [2] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information By: Rusty Newton (rnewton) 2017-09-01 17:02:34.236-0500 Thanks for the report! In order to maximize efficiently, there are project guidelines for how to report issues. Please read through the Asterisk Issue Guidelines [1]. After reading the guidelines, please clean up this issue so that bug marshals can more easily help you. In particular: 1. Don't post extensive debug or logs inside the Description or Comment fields. 2. Use the Description field for a description of the issue, referencing *attached* debug with links or notes. 3. Use the Comment fields for discussion regarding the issue. 4. If you need to put a few lines of debug or logs into any field, surround the text with *noformat* tags to help us read it easily. 5. Attach files with a '.txt' extension where possible so that they can be analyzed futher by bug marshals. Thanks! [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines By: Frederic Steinfels (fredo) 2017-09-02 02:30:43.705-0500 I have deleted my log post and reuploaded it as attachment. I have reformatted my text using the noformat tag which took me some time to figure out how. I hope this satisfies the guidelines. By: Rusty Newton (rnewton) 2017-09-05 18:34:36.307-0500 Thanks for formatting and providing the additional information. Can you revert to a GIT commit previous to bc51d4324a69a0b8ee4a3be208b91bb2081124ff and report back if the issue no longer occurs? That is, use anything before that commit. I'm thinking the root problem here is the same as on ASTERISK-27250 and they are reporting that that commit caused the regression. Possibly also the same as ASTERISK-27223. I may be off, but this will help narrow it down. By: Frederic Steinfels (fredo) 2017-09-06 08:07:42.288-0500 Thanks Rusty, I will look into this next week. However when reading the other bug reports, they are talking about receiving the wrong packets or garbled packets. In my case, rtp debug is telling me that I am not receving ANY packets at all from my cisco voip gateway. By: Rusty Newton (rnewton) 2017-09-06 09:55:17.367-0500 In some cases the others are reporting no audio as well, where Asterisk is ignoring the RTP stream from the far end after the far end changes what it is sending (and in the SDP of course). Very similar to your case. By: Frederic Steinfels (fredo) 2017-09-19 05:36:14.448-0500 Hi. I can not reveret using git. I will get an error message: git revert -m1 bc51d4324a69a0b8ee4a3be208b91bb2081124ff error: could not revert bc51d43... Merge "pjsip: Extend 'asymmetric_rtp_codec' option to include us changing." into 13 hint: after resolving the conflicts, mark the corrected paths hint: with 'git add <paths>' or 'git rm <paths>' hint: and commit the result with 'git commit' Please provide me with the proper parameters for git. By: Rusty Newton (rnewton) 2017-09-19 14:51:05.305-0500 This is the change that may cause the problem: https://gerrit.asterisk.org/#/q/Ib1647f6902a0843e8c435946f831c2159e8d1d51 From that search result page you can navigate to the review for the appropriate branch. Then you should spot the commit ID number on that individual review page. Once you have that commit ID, go to your local git directory for that branch and then look at the git log {noformat} git log --pretty=fuller {noformat} Find the commit with that commit ID, then find the commit ID just previous to that commit (look at the commit dates) in time. Once you have the commit ID of the commit previous to the potential regression, go back to your git directory for the branch and do {noformat} git checkout <commit ID> {noformat} Then you can build and install from there. For example, the commit ID for 13, shown on the review for 13 is "996a4791ff123e80d71d44cb0fd13bb201d197b1" So if I search the git log in 13 for that, then I find that the previous commit is "812f5b51cb56a36668decc6dfc83adeca185429e" In my local clone of branch 13 I would then run {noformat} git checkout 812f5b51cb56a36668decc6dfc83adeca185429e {noformat} Then I would build and install from there. Let me know if that makes sense. By: Asterisk Team (asteriskteam) 2017-10-04 12:00:01.374-0500 Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1]. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines By: Frederic Steinfels (fredo) 2017-10-07 01:41:13.263-0500 Hi. I have followed your instructions to de detail, have verified that the source code is the old version (according to the patch) and can confirm that this problem is NOT EXISTANT under these circumstances. It seems this git commint broke asterisk. Please advise on how to proceed. By: Asterisk Team (asteriskteam) 2017-10-07 01:41:13.671-0500 This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable. By: Joshua C. Colp (jcolp) 2017-10-20 11:40:19.606-0500 This issue should be resolved in the current release candidates. A fix was put in to resolve an issue where we would not switch our sending codec as expected. Please try and see. By: Frederic Steinfels (fredo) 2017-10-23 16:02:57.444-0500 I have downloaded the latest asterisk 15 branch (Asterisk GIT-15-415c8e9c8f) from github and tried this. Unfortunately I still/again have no audio in both directions. So the bug is not fixed. On the recorded call I can hear two rings and then silence from the called party side. My side is always audible. The other party told me that he could not hear me. By: Kevin Harwell (kharwell) 2017-10-24 18:12:24.090-0500 As stated earlier for an issue like this a packet capture would be helpful. As far as parameters for the packet capture you want to make sure it's listening on the same network as Asterisk. If you have a lot of other traffic on that network you'll want to filter on SIP and RTP packets for the given relevant addresses/ports. A full [Asterisk debug log|https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information] (at least debug >= 5 and "pjsip set logger on") of the full scenario (start of call to end) is also required. By: Frederic Steinfels (fredo) 2017-10-25 13:07:15.953-0500 You already have a full debug log. However I can do more logging for you with the current version. On the SIP/RTP please provide an example (command line tool, settings) how to capture the packets in the way you want them captured. Of course I will be able to fill in the IP adresses and port numbers. And is there a way to upload the results in a private manner? I do not want to post SIP credentials in the open. When attaching files here, I only have "Public" or "All users". By: Joshua C. Colp (jcolp) 2017-10-25 13:19:58.299-0500 There is no mechanism except for security issues generally that allow such details to be posted privately. We ask that reporters sanitize their postings as they see fit. As well credentials are not included in SIP logs in plain text or that can be used. By: Kevin Harwell (kharwell) 2017-10-25 17:44:28.596-0500 {quote} On the SIP/RTP please provide an example (command line tool, settings) how to capture the packets in the way you want them captured {quote} If you want to use tcpdump something like the following should get you most of the way: {noformat} tcpdump -i <interface> -w <output file name> {noformat} I don't use tcpdump enough to know what settings to use for filters and such. The man page however lists the various options you'll need if you go that route. I'm also not familiar with the wireshark command line interface, but it too has a man page that lists the options for reference if you'd rather use that. If you use the wireshark UI, select the interface you want to listen on, and then start listening. You can then filter packets after the fact if needed. Again using the UI. {quote} You already have a full debug log. However I can do more logging for you with the current version. {quote} Yes, please provide a full log of the scenario with debugging turned up and with SIP logging ("pjsip set logger on") that aligns (is captured at the same time) with the packet capture. By: Asterisk Team (asteriskteam) 2017-11-09 12:00:01.855-0600 Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1]. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines By: Frederic Steinfels (fredo) 2017-12-06 04:53:21.752-0600 Sorry, there is too much traffic on my network and I do not have the time to build a secondary system or to sanitze personal information that would result from such a capture. The information I gave so far should really be more than enough to fix this problem. By: Asterisk Team (asteriskteam) 2017-12-06 04:53:22.370-0600 This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable. By: Joshua C. Colp (jcolp) 2017-12-06 17:55:45.988-0600 I'm afraid it's not enough. I've looked over what you've provided again and nothing stands out. It all looks fine. I've also tested things and they are working. If you can't get the RTP debug then more information on the Asterisk side would be needed to understand precisely what it is doing. This would be the output of "core show channel" for the channel in question, as well as the core debug if possible. By: Frederic Steinfels (fredo) 2017-12-08 11:01:10.112-0600 You have the RTP debug log, you have the core set verbose xxx log. Furthermore you know that without that particular patch, audio is working and with that patch included (not reverted), there is no audio. How about sending me some patches, Ill try them out? By: Joshua C. Colp (jcolp) 2017-12-08 11:29:32.021-0600 There is noone working on this issue as of this time, it is still in a triage state. I don't have a time frame on when it will be looked into once the issue is accepted. If you don't want to provide more information then that's fine, but it'll make it harder for the person who does work on this issue to work it and also likely result in further back and forth. By: Joshua C. Colp (jcolp) 2017-12-08 11:30:49.285-0600 I've marked it as open. By: Frederic Steinfels (fredo) 2018-01-09 19:02:48.818-0600 The person who is going to fix this can still contact me to get further information from me. I'd be willing to do more traces but as nobody knows if this bug will ever get fixed, and when and if my report will be used for fixing this, I'd rather wait to be contacted. After all, we know exactly that https://gerrit.asterisk.org/#/q/Ib1647f6902a0843e8c435946f831c2159e8d1d51 is causing this behaviour and the version without that "fix" will work. It's just those 30 additional lines of code that will do something special and the verbose log and pjsip log is already available in this bug report. By: Frederic Steinfels (fredo) 2018-03-07 12:56:07.454-0600 I have made a lot of captures, tried various versions of Asterisk and 3CX and I think I have resolved this issue. For some reasons, swisscom is no longer doing a codec change in the scenario as described. So that error has gone and is no longer recreatable. However, I figured out, that there is a problem with the Cisco Session Border controller and the software supplied by swisscom. This was a both directions no audio/no rtp scenario. Swisscom has fixed this in a beta version for their SBC. They will probably update the SBC software for all their customers in April. In the meantime, if a swisscom SmartBusinessConnect Trunk customer is calling a Swisscom InOne customer having a diversion configured, there will always be no audio AFAIK. I might have been mislead by the initial response from swisscom blaming asterisk. So far, we dont know if there is still a bug in asterisk however we know there is/was definately one with Swisscoms SBC. Please consider closing the ticket. By: Joshua C. Colp (jcolp) 2018-03-07 12:58:09.888-0600 Closing per last comment. | ||||