Summary:	ASTERISK-30269: res_srtp: One way audio after unhold
Reporter:	Tyler Pearson (Tyler_Pearson)	Labels:
Date Opened:	2022-10-20 11:37:00	Date Closed:	2022-11-11 03:13:31.000-0600
Priority:	Minor	Regression?
Status:	Closed/Complete	Components:	Resources/res_pjsip_outbound_registration Resources/res_srtp
Versions:	18.15.0	Frequency of Occurrence	Constant
Related Issues:
Environment:	Debian GNU/Linux 10 (buster) in GCP.	Attachments:	( 0) Screenshot_linphone1.png ( 1) Screenshot_linphone2.png
Description:	We just upgraded our phone servers to the new Incredible PBX (Asterisk Version: 18.2.1). I can't find any other form online that has offered a suggestion for a fix to a hold issue we are having. When a call comes in, audio on both ends works until that call is placed on hold from our end for what seems to be a certain amount of time; When the call is retrieved from hold, there is no audio on our end, but sometimes the other end can still hear us. I can pretty reliably reproduce the issue by calling from my cell phone into our phone server, placing myself on hold, and then waiting to retrieve that call for about 15-20+ minutes (anytime the call is retrieved before that time, audio resumes okay on both ends). Our phone server at that time then continuously produces the following logs until the call is terminated by either side: 19243 [2022-10-04 16:44:21] VERBOSE[21514][C-0000001c] res_srtp.c: SRTCP unprotect failed on SSRC 1667512219 because of replay check failed (index too old) 19244 [2022-10-04 16:44:21] VERBOSE[21514][C-0000001c] res_srtp.c: SRTP unprotect failed on SSRC 1667512219 because of authentication failure 160 This message sometimes also appears: 5904 [2022-10-04 11:21:18] WARNING[27806] res_pjsip_outbound_registration.c: No response received from 'sip:**our-telephony-provider.com**:5060' on registration attempt to 'sip:70511898@**our-telephony-provider.com**:5060', retrying in '60' Our telephony provider looked a bit into this and said it may be a media session timeout while on hold. They also could not tell from a PCAP of the issue if it was our sever or the softphone we are using (Linphone). They pointed us to contact Asterisk support. I reached out to people on voip-info.org (here is the thread: https://www.voip-info.org/forum/threads/srtcp-unprotect-failed-unhold-issues.26668/)  and they pointed me here. Not sure if this is the right place for this, but would someone be able to help me troubleshoot this?
Comments:	By: Asterisk Team (asteriskteam) 2022-10-20 11:37:04.153-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/]. By: Joshua C. Colp (jcolp) 2022-10-20 11:40:11.558-0500 It appears the bug you have submitted is against a rather old version of a supported branch of Asterisk. There have been many issues fixed between the version you are using and the current version of your branch. Please test with the latest version in your Asterisk branch and report whether the issue persists. Please see the Asterisk Versions [1] wiki page for info on which versions of Asterisk are supported. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions By: Joshua C. Colp (jcolp) 2022-10-20 11:41:18.118-0500 Additionally, you also need to state which version of libsrtp you are using. It is what actually does the SRTP. What phones are in use? What firmware version are they? By: Tyler Pearson (Tyler_Pearson) 2022-10-20 11:58:59.279-0500 Looks like we are on libsrtp2-1:amd64 2.2.0-1. Linphone version is Desktop 4.4.1 - Qt5.12.12 Core 5.1.19-1-g6cdd0918e. I will work on getting asterisk updated By: Tyler Pearson (Tyler_Pearson) 2022-10-21 12:20:50.819-0500 We upgraded to 18.15.0 and now there is only one way audio with the same error message of "res_srtp.c: SRTCP unprotect failed on SSRC 1120105267 because of replay check failed (index too old)". I also tried putting the call back on hold and resuming the call, but that cuts all audio and additionally has the message: "res_srtp.c: SRTP unprotect failed on SSRC 1120105267 because of authentication failure 160". By: Benjamin Keith Ford (bford) 2022-10-26 09:24:13.358-0500 Which side is the one way audio on after upgrading? Also, can you provide your PJSIP configuration for the endpoint? If you are taking additional steps to set up the call, can you provide that as well? By: Tyler Pearson (Tyler_Pearson) 2022-10-26 11:39:30.572-0500 the side calling into the PBX can still hear everything. there are no additional steps we are doing to setup the call. here is the config for the extension i am testing: [509] type=endpoint aors=509 auth=509-auth tos_audio=ef tos_video=af41 cos_audio=5 cos_video=4 allow=g722,ulaw,opus,g729,g726,gsm,alaw context=from-internal callerid=Tyler Pearson <509> dtmf_mode=rfc4733 direct_media=yes mailboxes=509@default mwi_subscribe_replaces_unsolicited=yes aggregate_mwi=yes use_avpf=no rtcp_mux=no max_audio_streams=1 max_video_streams=1 bundle=no ice_support=no media_use_received_transport=no trust_id_inbound=yes user_eq_phone=yes send_connected_line=yes media_encryption=sdes timers=yes timers_min_se=500 media_encryption_optimistic=no refer_blind_progress=yes refer_blind_progress=yes rtp_timeout=30 rtp_timeout_hold=300 send_pai=yes rtp_symmetric=yes rewrite_contact=yes force_rport=yes language=en one_touch_recording=on record_on_feature=apprecord record_off_feature=apprecord By: Tyler Pearson (Tyler_Pearson) 2022-10-26 11:42:13.624-0500 here is my linphone config. srtp is enabled. the "encryption is manditory" setting works on and off. By: Tyler Pearson (Tyler_Pearson) 2022-10-26 11:46:57.440-0500 I've attached more of my linphone config. AVPF is enabled and there is no ICE or STUN server configured or enabled By: Joshua C. Colp (jcolp) 2022-10-27 03:59:42.645-0500 We require additional debug to continue with triage of your issue. Please follow the instructions on the wiki [1] for how to collect debugging information from Asterisk. For expediency, where possible, attach the debug with a '.txt' file extension so that the debug will be usable for further analysis. Thanks! [1] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information I think we also need to see the debug level information, to see what keys are internally being used. Just a note as well - from my experience so far every time this has come up it's actually been a bug in the key handling of the remote endpoint and not Asterisk. By: Joshua C. Colp (jcolp) 2022-10-27 04:01:11.782-0500 In fact there is an option specifically around this[1]. If you disable it, does the issue resolve itself? [1] https://github.com/asterisk/asterisk/blob/master/configs/samples/rtp.conf.sample#L48 By: Tyler Pearson (Tyler_Pearson) 2022-11-01 16:05:10.890-0500 I tried adding that line to my rtp_additional.conf and uncommented the 'include rtp_additional.conf' in the rtp.conf file and that didn't work. I still got the "Index too old..." error. I have some .pcap files with debug level at 5 with verbose level also at 5, is there some way to send them to you that is not public? By: Joshua C. Colp (jcolp) 2022-11-01 16:52:14.629-0500 Did you confirm that the option was disabled using "rtp show settings"? As well, it is HIGHLY preferred that you sanitize the logs according to how you wish and attach them. If you can't, then it IS possible but doing so GREATLY limits the audience who would have any interest in looking at this. By: Tyler Pearson (Tyler_Pearson) 2022-11-01 17:39:04.330-0500 > rtp show settings General Settings: ----------------  Port start:      10000  Port end:        20000  Checksums:       Yes  DTMF Timeout:    1200  Strict RTP:      Yes  Probation:       4 frames  Replay Protect:  Yes  ICE support:     Yes  STUN address:    0.0.0.0:0 doesn't look like my change in the config file did anything. "Replay Protect" should be 'no' correct? what is the best method of going about this? By: Joshua C. Colp (jcolp) 2022-11-02 03:28:08.682-0500 I don't work on FreePBX so can't really comment on the way to do it there in regards to its configuration. By: George Joseph (gjoseph) 2022-11-02 09:42:18.604-0500 Did you do a restart or a reload?  For that file, you have to actually restart asterisk. By: Michael Bradeen (mbradeen) 2022-11-02 10:04:46.712-0500 FreePBX will over-write the rtp_additional.conf on a fwconsole restart/reload so you have to put the setting in rtp_custom.conf.  Both rtp_custom.conf and rtp_additional.conf should be included in rtp.conf by default.   Once you've added a srtpreplayprotection=no to rtp_custom.conf a FreePBX or Asterisk core reload should apply it. By: George Joseph (gjoseph) 2022-11-09 06:01:43.128-0600 [~Tyler_Pearson] Were you able to try this? By: Tyler Pearson (Tyler_Pearson) 2022-11-10 15:57:49.917-0600 EUREKA! IT WORKS!! looks like Joshua and Micheal's suggestion (https://github.com/asterisk/asterisk/blob/master/configs/samples/rtp.conf.sample#L48) was the solution! I had do do a few things different to get it to work with our server: I had to edit /etc/asterisk/rtp_custom.conf and add the line "srtpreplayprotection=no" and then restarting asterisk with init.d. The only oddity with this fix is that after 15 minutes on the call, the error still shows in the logs and the audio takes about 2ish seconds to re-establish the INVITES and then the errors stop. Everything works like a charm after that. I believe it is just a bug with Linphone. I will see if i can reach out to them to report this. Thank you guys so much for taking the time to help me sort through this issue!