[Home]

Summary:ASTERISK-27532: ERROR[5383][C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648
Reporter:Mohit Taneja (mtaneja712)Labels:fax webrtc
Date Opened:2017-12-26 08:39:09.000-0600Date Closed:2020-01-14 11:13:52.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:. I did not set the category correctly.
Versions:13.17.0 Frequency of
Occurrence
Related
Issues:
Environment:Sangoma Freepbx official distro FreePBX 13.0.192.16 Asterisk 13.17.0Attachments:( 0) console.txt
( 1) Frack_version13_18.txt
( 2) log_2018-01-15_core_show_channels.txt
( 3) log_2018-01-15_top.txt
Description:Hi

We are recently installed Freepbx distro “FreePBX 13.0.192.16” running “Asterisk 13.17.0” on a x86_64 running Linux. From last 2 weeks the CPU suddenly goes to 100% utilization everyday between 9:50 AM EDT to 10:30 AM EDT.
In the full logs we saw below error during the high CPU utilization. Once the error stop appearing the CPU agin goes to normal
I have disabled CDR Reports and Call Recording Report modules but the problem is still happening
Comments:By: Asterisk Team (asteriskteam) 2017-12-26 08:39:10.479-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Joshua C. Colp (jcolp) 2017-12-27 18:49:14.914-0600

We're going to need to know more about the system itself. What is going on at the time? Is there other console output? How many calls are happening? As well please use the latest version of Asterisk, in this case 13.18.4.
By: Mohit Taneja (mtaneja712) 2017-12-28 02:47:48.138-0600

It's a virtual freepbx box with 12 GB memory , 4 vCPU and 50 GB HDD. We are using queuemetrics with the freepbx to provider logger and reporting functionalities to agents.
Normal call center activities happen during the time like login/logout/pause/unpasue , incoming calls, IVR selections, call queuing and agent selection and barging.
Core show channels verbose shows around 180 active channels and 100-110 active calls during the time .
What other console output you are looking for ?

As i mentioned we are using freepbx , how to upgrade asterisk to 13.18.4 in this scenerio?
Thanks,
Mohit Taneja
By: Mohit Taneja (mtaneja712) 2017-12-28 02:53:28.539-0600

Earlier we were using Asterisk 13.11.2 and it doesn't have this issue.
By: Joshua C. Colp (jcolp) 2017-12-28 04:52:17.199-0600

You'd have to look how to update FreePBX. We don't make it and I don't use it. Bug fixes in this area did occur so an upgrade would be extremely helpful before pursuing this further.
By: Mohit Taneja (mtaneja712) 2017-12-29 02:30:14.149-0600

As i mentioned earlier that on version 13.11 we didn't face this issue so what would you suggest , downgrading to version 13.11 or upgrading to version 13.18.4 ?
By: Joshua C. Colp (jcolp) 2017-12-29 05:07:22.291-0600

If you downgrade to 13.11 then there's nothing else I can do on this issue. If you update to the latest version and it still occurs then we can investigate further.
By: Asterisk Team (asteriskteam) 2018-01-12 12:00:02.267-0600

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines
By: Mohit Taneja (mtaneja712) 2018-01-15 07:17:16.557-0600

Hi,

We did upgrade to 13.18.4 and we are not seeing frack errors but we are still facing the issue . All of sudden between 9:30 to 10:30 AM ET the system CPU utilization goes to 100% . We are using queuemerics with asterisk to provide dynamic agent login and pause functionality. During high CPU utilization we saw multiple hung channels for queuemetrics context and high CPU IO wait. Attached the snippet of active channels and top command

See [^log_2018-01-15_top.txt]
See [^log_2018-01-15_core_show_channels.txt]

We are getting below intermittent warning in fulllogs
{noformat}
[2018-01-12 10:07:35] WARNING[20604] taskprocessor.c: The 'stasis-core-control' task processor queue reached 500 scheduled tasks again.
[2018-01-12 10:07:35] WARNING[20711][C-0001131a] taskprocessor.c: The 'subm:manager_topic-00000007' task processor queue reached 3000 scheduled tasks again
{noformat}

Please help us to diagnose this furthur.

Regards.
Mohit

Edit: Extracted the pasted logs and attached them as text files per issue guidelines.  Please *do not* paste large logs into issue comments it makes working with issues more difficult.
{quote}
Be courteous. Do not paste debug output in the description or a comment, instead please attach any debugging output as text files and reference them by file name.
{quote}


By: Asterisk Team (asteriskteam) 2018-01-15 07:17:17.316-0600

This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable.
By: Kevin Harwell (kharwell) 2018-01-16 17:44:26.131-0600

Unfortunately this is going to be really hard to diagnose with the given information. You say your cpu(s) are rising to close to 100% at the same time everyday for roughly the same period of time. If I am reading the above output correctly Asterisk was using ~32% of the cpu though. Is something else causing the cpu(s) to be consumed? Some other process, which then in turn slows down Asterisk? You are running in a VM so that program could even lie outside of your current VM. If multiple VMs are running on the machine at the same time some other process that kicks off everyday around that time could be eating up the CPU.

I would start by first making sure no other process is causing the slowdown. Check other processes running (databases, other programs, etc...) around that time on the parent system or in other VMs if applicable. If you are running other VMs can you have those switched off around the time of day the problem occurs?

Debug log files would be helpful too. Since it happens around the same time everyday it should be easy enough to escalate debugging on the system during that time. See [Collecting Debug Information|https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information] on the wiki for more information on how to do that
By: Asterisk Team (asteriskteam) 2018-01-31 12:00:00.497-0600

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines
By: Mohit Taneja (mtaneja712) 2018-02-01 04:19:05.228-0600

Hi,

We upgraded the system to 13.18 as suggested and after a week we are again getting the same frack errors which is causing High CPU. Please help us to fix it.
2018-01-31 09:54:54] ERROR[19073][C-0004576f] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x2ac1158
[2018-01-31 09:54:54] ERROR[19073][C-0004576f] astobj2.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x2ac1158 (0)
[2018-01-31 09:54:54] ERROR[22341][C-000459fc] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x2ac1158


[root@is01-astrsk-001 ~]# asterisk -rx "core show version"
Asterisk 13.18.3 built by mockbuild @ jenkins2.schmoozecom.net on a x86_64 running Linux on 2017-12-01 23:22:34 UTC

PFA complete snippit


By: Asterisk Team (asteriskteam) 2018-02-01 04:19:06.921-0600

This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable.
By: Mohit Taneja (mtaneja712) 2018-02-07 03:06:05.191-0600

Can someone please help us here. it's happening on daily basis and impact business.
Please help !!
By: Benjamin Keith Ford (bford) 2018-02-19 13:55:21.722-0600

As others have mentioned, we are going to need more information to move forward with this issue, namely:
# Isolate the process. Since it is on a VM, it would be extremely beneficial to know that Asterisk is the only thing consuming CPU. There are many external things that could cause the problem. We need to make sure it is Asterisk and not one of these other external things. Make sure Asterisk is the only process running when the issue occurs.
# Provide log files. If possible, turn debug and verbose up to 9 and provide the log files produced when this issue occurs. This could help track down the problem.
By: Asterisk Team (asteriskteam) 2018-03-06 12:00:01.027-0600

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines
By: Cirillo Ferreira (cirillor@lbv.org.br) 2018-10-06 11:30:24.168-0500

Maybe the fix to ASTERISK-27591 resolved this problem.