Summary:	ASTERISK-29657: Asterisk queue stops or hangs
Reporter:	Noppakoon Khamklin (noppakoon)	Labels:
Date Opened:	2021-09-15 20:58:34	Date Closed:	2021-10-13 12:00:02
Priority:	Critical	Regression?
Status:	Closed/Complete	Components:	Applications/app_queue
Versions:	16.7.0	Frequency of Occurrence	Occasional
Related Issues:
Environment:		Attachments:	( 0) asterisk-backtrace.zip
Description:	Asterisk can accept new incoming calls with IVR.  Whenever the caller goes to any queues, the caller doesn't hear music on hold and the queue member's phone doesn't ring at all. From the console, the "queue show" command doesn't return the output and the "core show channels" shows many many channels. To temporarily solve this problem, the asterisk must be restarted then it comes back to normal.
Comments:	By: Asterisk Team (asteriskteam) 2021-09-15 20:58:37.880-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/]. By: Noppakoon Khamklin (noppakoon) 2021-09-15 21:00:05.257-0500 Here is the asterisk backtrace file. By: Noppakoon Khamklin (noppakoon) 2021-09-15 21:04:09.615-0500 I also see this log message WARNING[9635][C-00000316] channel.c: Exceptionally long voice queue length queuing to Local/7022509@from-queue-00000586;2. By: George Joseph (gjoseph) 2021-09-16 08:37:03.771-0500 It appears the bug you have submitted is against a rather old version of a supported branch of Asterisk. There have been many issues fixed between the version you are using and the current version of your branch. Please test with the latest version in your Asterisk branch and report whether the issue persists. Please see the Asterisk Versions [1] wiki page for info on which versions of Asterisk are supported. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions By: Noppakoon Khamklin (noppakoon) 2021-09-16 09:25:18.683-0500 If this is really a bug or problem, do you have any info about this problem?  We need some info to respond back and initiate change request to our customer before requesting to upgrade the software. In case the change request is not approved by our customer, are there any workarounds or temporary solutions to prevent this problem.  We created cronjob every morning to restart asterisk but it still happened last Monday. By: Joshua C. Colp (jcolp) 2021-09-16 09:31:18.227-0500 We don't know if this has been resolved or not, as we have not done deep research or investigation into it. As a project we don't spend time looking at issues against old versions like this. You are free to look at the issue tracker yourself and see if there is a matching resolved issue. From a project perspective if this can be demonstrated in a recent version then we will look at the provided information and if it appears to be an issue, then it would be accepted as an open issue. By: George Joseph (gjoseph) 2021-09-16 09:51:40.400-0500 Another issue is that the backtrace provided has no symbols so while we can see that there are threads locked in app_queue, we don't know where or on what. Please see https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace By: Noppakoon Khamklin (noppakoon) 2021-09-21 21:14:53.860-0500 Yesterday we upgraded the asterisk to 16.16.  Is this version the latest one in 16 branch?  The customer will monitor the problem and respond back to me. Regarding to the backtrace, we followed the instruction there to generate this backtrace with: # /var/lib/asterisk/scripts/ast_coredumper --running --no-default-search Is that correct? Else pls suggest. By: Joshua C. Colp (jcolp) 2021-09-22 04:34:52.625-0500 The latest version of 16 is 16.20.0 That is the correct use of ast_coredumper, however you still have to ensure that Asterisk is built correctly to have symbols. By: Noppakoon Khamklin (noppakoon) 2021-09-22 05:34:29.157-0500 The customer uses Issabel PBX with Asterisk inside.  When we did yum update, it is only asterisk 16.16 in the list. For backtrace, how do we know if Asterisk is built correctly to have symbols?  If it is not, how do we get the right backtrace for troubleshooting then. By: Joshua C. Colp (jcolp) 2021-09-22 05:38:51.269-0500 As we don't manage or produce Issabel PBX, we can't comment on its versioning or availability of things. As for knowing if it's built to have symbols you could do "file /usr/sbin/asterisk" and if it says it is stripped, then symbols aren't present. They may be available somehow through the distro but as we don't do that distro, we can't comment on that. The already provided link for getting a backtrace provides instructions on how to build Asterisk so the information is usable. It's in the section labeled "Preparing Asterisk To Produce Core Files On Crash". By: Noppakoon Khamklin (noppakoon) 2021-09-26 21:00:28 I checked the asterisk as said in the webpage. [root@issabel ~]# file /usr/sbin/asterisk /usr/sbin/asterisk: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=fdc3ad94f8f2b9de4b3e5f3e8105d777c4fcc62a, stripped [root@issabel ~]# ps -C asterisk u USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND asterisk 31217  0.2  2.8 1731680 57900 ?       Ssl  Sep24  11:08 /usr/sbin/asterisk -U asterisk -G asterisk -mqf -C /etc/asterisk/asterisk.conf [root@issabel ~]# So that means this asterisk can't provide the proper backtrace, right?  I think we can change the startup process with safe_asterisk (I hope there is impact to its operation).  With safe_asterisk startup, is it possible to get the proper backtrace?  What other alternatives can we do to get proper backtrace in Asterisk 16.16 so you can troubleshoot the problem? By: Joshua C. Colp (jcolp) 2021-09-27 04:18:08.146-0500 No, that won't change getting a proper backtrace. If Asterisk is stripped and not built with symbols, then Asterisk has to be changed. By: Noppakoon Khamklin (noppakoon) 2021-09-28 20:34:28.647-0500 If we replace that stripped asterisk file with non-stripped version, will this work to get the proper backtrace?  I hope it is just that single file only to be replaced.  Is it possible for  you to send me that non-stripped asterisk version?  I will do some tests in the testing environment and if it has no impact, we can deploy in the customer production server with safe_asterisk startup. By: Joshua C. Colp (jcolp) 2021-09-29 04:40:26.633-0500 You would need to replace all of Asterisk, including its modules. We also don't produce binary versions of Asterisk, and we have no idea how it was built for your distribution. Your question is really how to get a recent Asterisk that can be debugged with your distribution. We have no experience with your distribution and can't really answer that. A previous wiki page mentioned they may have debug packages, but that's about the extent of our knowledge. Reaching out to that project instead would be better in that regard. By: Asterisk Team (asteriskteam) 2021-10-13 12:00:01.910-0500 Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1]. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines