Summary:	ASTERISK-29013: res_pjsip: Asterisk doesn't stop sending invites (with auth) on 407 replies
Reporter:	Sebastian Damm (sdamm)	Labels:
Date Opened:	2020-07-28 04:29:16	Date Closed:	2020-11-05 10:32:10.000-0600
Priority:	Minor	Regression?
Status:	Closed/Complete	Components:	Resources/res_pjsip Resources/res_pjsip_authenticator_digest
Versions:	17.6.0	Frequency of Occurrence	Constant
Related Issues:	is duplicated by ASTERISK-29059 Asterisk sends endless INVITE requests even call is ended
Environment:	Debian Buster, Asterisk Package built from sources	Attachments:	( 0) asterisk407.tar.gz
Description:	We have the following setup. (From our pjsip.conf) [domain.de](generic_endpoint) auth=domain_internal_auth outbound_auth=domain_internal_auth from_domain=domain.de outbound_proxy=sip:sip.domain.net\;lr aors=domain.de_aor [domain.de_aor] type=aor contact=sip:domain.de outbound_proxy=sip:sip.domain.net\;lr [domain_internal_auth] type=auth auth_type=userpass username=happyuser password=reallysecret This endpoint is used to reach our registered customer devices, with a Kamailio proxy in between. Now when we send out a call through this endpoint, the proxy server asks for Auth. Asterisk responds to the challenge, and normally the call goes through. But we have a client device (an Asterisk server) behind the proxy server asking for authentication, too. (Of course, we don't know any password for this client device.) In that scenario, Asterisk17 does not stop sending INVITEs toward the proxy. When the first 407 is received, an Proxy-Authorizationheader for authenticating against the proxy server gets created, and when the second 407 is received, Asterisk sends out the next INVITE with two Proxy-Authorization headers. {{Proxy-Authorization: Digest username="happyuser", realm="domain.de", nonce="Xxl+ZF8ZfTg2/dTjNjcsTCYGI3Z+f85d", uri="sip:004926439482507@domain.de", response="cc3cdb70fa0451b51aa8cbf9ccfb6426"}} {{Proxy-Authorization: Digest username="happyuser", realm="asterisk", nonce="545e619d", uri="sip:004926439482507@domain.de", response="66400b176d5c9d2c3f0aad26d3683391", algorithm=MD5}} After 30 seconds, the caller cancels the call, Asterisk sends out a CANCEL request, which - again - gets rejected with a 407 by the end user device. Asterisk does not re-send the CANCEL message, but does not stop sending out the INVITE requests. And this goes on forever. We have only noticed this behavior, because we saw a massive amount of memory getting used by the Asterisk process. And we didn't send any new traffic to Asterisk and {{core show channels}} didn't show any calls anymore, the INVITEs to this device kept on going. This could result in a DOS, if you know the setup and can setup a scenario like this and send a lot of calls through this setup. Multiple calls result in Asterisk using all of the available memory twice as fast. In my opinion, Asterisk should stop sending out INVITEs after receiving a maximum of 3 407 responses. Our old Asterisk11 boxes behave that way, when handling calls to the same customer device.
Comments:	By: Asterisk Team (asteriskteam) 2020-07-28 04:29:18.566-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: Joshua C. Colp (jcolp) 2020-07-28 04:40:57.227-0500 Thanks for the report and debug. However we also need protocol specific debug captured at the time of the issue. Please include the following: * Asterisk log files generated using the instructions on the Asterisk wiki [1], with the appropriate protocol debug options enabled, e.g. 'pjsip set logger on' if the issue involves the chan_pjsip channel driver. * Configuration information for the relevant channel driver, e.g. pjsip.conf. * A wireshark compatible packet capture, captured at the same time as the Asterisk log output. [1] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information By: Sebastian Damm (sdamm) 2020-07-28 05:00:04.741-0500 We can provide you with the pjsip.conf, although I had already included the relevant parts in the original report. Regarding Logs, I'm afraid, I can't provide them, because this happens on a live gateway which had about 300 calls at the time when it happened. And I have a trace with at least some of the call's packages, but I'm afraid I can't share them publicly as there are customer numbers and IP addresses etc. in it. I probably could try to simulate the setup in a lab environment, but I don't know if and when I have time for it. By: Joshua C. Colp (jcolp) 2020-07-28 05:15:36.179-0500 In the future please try to ensure that SIP logs are also included for issues. This prevents the case where there is back and forth in the future if someone works on the issue and the information is insufficient. By: Joshua C. Colp (jcolp) 2020-07-28 05:32:10.314-0500 I'm unable to reproduce the scenario with the information provided. I set up a SIPp scenario which responds to each INVITE with a 407 challenge. Each challenge I've added an additional Proxy-Authenticate header. Asterisk is able to send 2 INVITE attempts before it stops with "Authentication credentials not accepted by server.". I've tested both Asterisk 16 and 17 and both behave the same. You're going to need to determine the specific conditions and ideally provide a test case that shows this problem. By: Sebastian Damm (sdamm) 2020-07-28 06:04:06.748-0500 I'm going to try to create a docker setup for reproducing. I'll get back to you. By: Sebastian Damm (sdamm) 2020-07-29 16:46:01.820-0500 I managed to reproduce the scenario. When the nonce of the 407 doesn't change, Asterisk indeed stops re-sending the INVITEs after two attempts with the message you also got. However, when the nonce changes with each 407, Asterisk keeps sending INVITEs responding to the new challenge. And, if you let it running long enough, Asterisk will consume more and more memory. I have created a docker setup with two sipp instances calling each other, the called side answering with 407 for 10 times before finally replying with 404. Asterisk sends out a new INVITE for each 407 received. By: Sebastian Damm (sdamm) 2020-07-29 16:48:42.339-0500 The docker setup for reproducing the fault After running {{docker-compose build}} and {{docker-compose up -d}} you can run {{docker-compose exec sipp /testcase/start.sh}}, and then check the Asterisk logs with {{docker-compose logs -f -t asterisk}}. You can safely cancel the exec-command after a few seconds. By: Sebastian Damm (sdamm) 2020-11-03 11:05:59.753-0600 Why is this not a security issue? We stumbled upon this only, because some of our Asterisk servers were consuming more and more memory over time. If you are an attacker and can setup a scenario, where you dial through the victim's asterisk and reject every INVITE with a 407 and different nonce, you can probably create a DOS and have Asterisk run out of memory pretty quickly. By: Joshua C. Colp (jcolp) 2020-11-03 11:07:51.048-0600 It was decided that it was not a security issue because doing just that requires an unreasonable amount of control over things. You need to have Asterisk dial a malicious target on an endpoint configured for outbound authentication. By: Sebastian Damm (sdamm) 2020-11-04 02:03:38.189-0600 Actually, it doesn't need that much knowledge and control. In our case (we are a VoIP service provider), we discovered it because one of our customers had an Asterisk server registered to our service that had a configuration error, wanting incoming calls to be authenticated. If you do that on purpose and generate incoming calls with that scenario, you might reach that DOS scenario pretty quickly. By: Joshua C. Colp (jcolp) 2020-11-04 03:21:12.122-0600 We'll take being a VoIP service provider into consideration next time for determining if something is a security issue, as that is actually a different perspective then has been taken in the past generally. Issues like this are always difficult to decide on. This is also my fault for ultimately deciding on such, so the blame for it lands on me - but I'll take it as a learning experience! Thank you for your feedback and being part of the Asterisk project! By: Joshua C. Colp (jcolp) 2020-11-04 15:38:11.311-0600 Based on your feedback we are going to release a security advisory, as well as a security release which includes the fix to get this out soon. By: Friendly Automation (friendly-automation) 2020-11-05 10:32:11.438-0600 Change 15135 merged by Friendly Automation: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15135|https://gerrit.asterisk.org/c/asterisk/+/15135] By: Friendly Automation (friendly-automation) 2020-11-05 10:43:08.226-0600 Change 15138 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15138|https://gerrit.asterisk.org/c/asterisk/+/15138] By: Friendly Automation (friendly-automation) 2020-11-05 10:43:40.855-0600 Change 15139 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15139|https://gerrit.asterisk.org/c/asterisk/+/15139] By: Friendly Automation (friendly-automation) 2020-11-05 10:44:14.568-0600 Change 15136 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15136|https://gerrit.asterisk.org/c/asterisk/+/15136] By: Friendly Automation (friendly-automation) 2020-11-05 10:44:45.148-0600 Change 15120 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15120|https://gerrit.asterisk.org/c/asterisk/+/15120] By: Friendly Automation (friendly-automation) 2020-11-05 14:19:04.981-0600 Change 15150 merged by George Joseph: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15150|https://gerrit.asterisk.org/c/asterisk/+/15150] By: Friendly Automation (friendly-automation) 2020-11-05 14:59:59.350-0600 Change 15152 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15152|https://gerrit.asterisk.org/c/asterisk/+/15152] By: Friendly Automation (friendly-automation) 2020-11-05 15:08:52.448-0600 Change 15168 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15168|https://gerrit.asterisk.org/c/asterisk/+/15168] By: Friendly Automation (friendly-automation) 2020-11-05 15:09:08.308-0600 Change 15169 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15169|https://gerrit.asterisk.org/c/asterisk/+/15169] By: Friendly Automation (friendly-automation) 2020-11-05 15:25:36.921-0600 Change 15170 merged by Kevin Harwell: AST-2020-002 - res_pjsip: Stop sending INVITEs after challenge limit. [https://gerrit.asterisk.org/c/asterisk/+/15170|https://gerrit.asterisk.org/c/asterisk/+/15170]