| Summary: | ASTERISK-26899: Unable to apply outbound proxy on request to qualify | ||||||
| Reporter: | Luke Escude (lukeescude) | Labels: | |||||
| Date Opened: | 2017-03-27 19:07:10 | Date Closed: | 2017-12-13 12:23:15.000-0600 | ||||
| Priority: | Major | Regression? | |||||
| Status: | Closed/Complete | Components: | |||||
| Versions: | 13.14.0 | Frequency of Occurrence | |||||
| Related Issues: |
| ||||||
| Environment: | x64 CentOS | Attachments: | ( 0) core1.txt ( 1) core2.txt | ||||
| Description: | We configure the endpoints to be qualified every 15 seconds.
All of our PBX systems are configured the EXACT same way, but ONE of them is doing this: {noformat} [Mar 28 00:03:12] ERROR[4509]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:5061@65.23.19.66:4737 [Mar 28 00:03:12] ERROR[4509]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:1129@96.230.64.162:45255 [Mar 28 00:03:12] ERROR[4509]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:1316@192.171.21.154:10110 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:1024@207.89.36.58:22092 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:1078@207.89.37.218:4492 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:5002@65.23.19.66:4703 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:5013@65.23.19.66:4732 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:1128@96.230.64.162:45240 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:4022@192.171.21.138:5347 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:2224@40.135.193.42:4947 [Mar 28 00:03:12] ERROR[4505]: res_pjsip/pjsip_options.c:436 qualify_contact: Unable to apply outbound proxy on request to qualify contact sip:4019@192.171.21.138:13900 {noformat} It's happening to about 80% of our endpoints on that server, and it's happening every 15 seconds. There are no outbound proxies or anything configured for these endpoints. Nothing changed either - the server has been operating just fine for months. | ||||||
| Comments: | By: Asterisk Team (asteriskteam) 2017-03-27 19:07:10.989-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Luke Escude (lukeescude) 2017-03-27 22:21:02.211-0500 Apparently this was solved by stopping Asterisk, deleting the AstDB file, and restarting Asterisk. Is there a way to automatically prevent this from happening? The astdb file was 15MB. Now it's like 3MB By: Rusty Newton (rnewton) 2017-04-02 19:11:45.197-0500 bq. Is there a way to automatically prevent this from happening? The astdb file was 15MB. Now it's like 3MB Since we don't know what is causing the issue... I don't know. This issue ASTERISK-25621 is the only one I found that looks related. It was closed as unresolved due to a lack of information. However there is some detail there that might help us. Are you able to turn up logging so that we can get a debug log the next time the issue occurs? https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information If we can get a log with warning,error,notice,verbose and debug, with both verbose and debug log levels turned up to 5 each, that would help a lot. In addition if you can check the AstDB to see if you find the same issue that was found in ASTERISK-25621. Sanitized output from asterisk -rx 'database show' may help too. By: Luke Escude (lukeescude) 2017-04-07 10:25:24.511-0500 Rusty, we have new information, we were able to catch this same crash before it happened - however this particular customer has a LOT of phones so we did not have debug/verbose enabled. My engineer will post up some Core Dumps, and I'll post up the astdb.sqlite3 files (1 from the original crash two weeks ago, and 1 from today) EDIT: It's possible this is being caused by the AMI, as all of the core dumps have mention of PJSIPShowContacts or something. By: Luke Escude (lukeescude) 2017-04-07 10:30:49.019-0500 ASTDB from 2 weeks ago: http://provisioning.primevox.net/astdb.sqlite3.BAK ASTDB from today: http://provisioning.primevox.net/astdb.sqlite3.BAK2 The crash 2 weeks ago was definitely related to outbound_proxy being set to "asterisk" in the AstDB, but today's crash seems more like the customer's internet connection being stupid. By: Jared Hull (fortytwo) 2017-04-07 11:00:42.443-0500 http://provisioning.primevox.net/core.vpbx27.primevox.net-2017-03-29T18:19:01+0000 http://provisioning.primevox.net/core.vpbx27.primevox.net-2017-04-07T14:33:17+0000 I think asterisk seg faults when creating AMI events, either because we have over 500 registered extensions or a specific set is unregistering continuously due to UDP keepalive issues at the customer site. By: Richard Mudgett (rmudgett) 2017-04-07 11:14:27.915-0500 Core files are useless to other machines because machines vary so much. Please create and attach backtraces to the issue per the issue guidelines. https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace By: Jared Hull (fortytwo) 2017-04-07 12:40:12.911-0500 Core dump backtraces attached to issue By: Joshua C. Colp (jcolp) 2017-04-11 04:49:18.554-0500 One of your crashes has already been fixed in the latest version, as it is ASTERISK-26623. By: Rusty Newton (rnewton) 2017-04-13 09:40:22.332-0500 [~lukeescude] , [~fortytwo] , Does the "unable to apply outbound proxy" spam issue occur *without* one of these crashes, or is it always accompanied by a crash? If we muddy this ticket with symptoms of multiple separate issues that will reduce the chance of us tackling one of these issues successfully. We need to make sure that if there are separate issues occurring here that we move them out into separate tickets. There was no mention of a crash in your original report, so I presume the crash(s) are separate issues? By: Asterisk Team (asteriskteam) 2017-04-27 12:00:01.470-0500 Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1]. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines | ||||||