[Home]

Summary:ASTERISK-29814: Very High CPU uses only 80 con-currents calls on asterisk via AMI
Reporter:Amardeep (amarniit)Labels:
Date Opened:2021-12-22 05:23:15.000-0600Date Closed:2022-01-12 12:00:01.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:Applications/app_dial Applications/app_macro Applications/app_playback Channels/chan_pjsip Channels/chan_sip/General Codecs/codec_alaw
Versions:16.24.0 17.0.0 18.6.0 18.9.0 Frequency of
Occurrence		Frequent
Related
Issues:
Environment:CentOS 7Attachments:
Description:CentOS 7, 40 Core CPU very high Very high CPU use on in 80  calls via ami

We are using clean Asterisk 18.9.0 on Cent OS 7.
TWO SIP trunk connected
Telecom Provider
1. TATA : 60 Channels
2. VODAFONE : 90 Channels
audio codec uses : alaw & ulaw for both

server configuration
Manufacturer: HPE
Product Name: ProLiant DL380 Gen10
Family: ProLiant

RAM : 64 GB

CPU Info
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz
Stepping:              7
CPU MHz:               999.902
CPU max MHz:           3200.0000
CPU min MHz:           1000.0000
BogoMIPS:              4800.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              14080K
NUMA node0 CPU(s):     0-9,20-29
NUMA node1 CPU(s):     10-19,30-39

uname -a
Linux localhost.localdomain 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

when CPU load reach around 35 in TOP command most of calls voice issue.
our call flow is very simple
ami originate call from another server --- asterisk dial number--- play audio file--- google API audio to text---- and play another audio based on response.

Comments:By: Asterisk Team (asteriskteam) 2021-12-22 05:23:16.151-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/].
By: Joshua C. Colp (jcolp) 2021-12-22 05:37:57.788-0600

We require additional debug to continue with triage of your issue. Please follow the instructions on the wiki [1] for how to collect debugging information from Asterisk. For expediency, where possible, attach the debug with a '.txt' file extension so that the debug will be usable for further analysis.

Thanks!

[1] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information


By: Joshua C. Colp (jcolp) 2021-12-22 05:38:35.995-0600

You also need to explain how you are interfacing with Google, and what you are using in Asterisk.
By: Amardeep (amarniit) 2021-12-22 23:04:29.445-0600

yesterday our this server's asterisk is not working after "core restart now"
we had implement new server almost equal configuration

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                20
On-line CPU(s) list:   0-19
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz

RAM Size 16 GB

our call flow is like

server 1 originate call via ami and pass some call related information.

server 2 - where asterisk installed and telecom connected
accept the originate command and dial customer number.
after the answer the call.. play a audio from URL which location we receive  via ami in form of variable.
after this. we record user side response and send it to google via api and receive text from google. based on text received play the next audio.
after call hangup send all info to ami server via php api.

today at starting on calls, server load was 2 or 3 but as time goes now load reach around 22 where our cpu cores are 20..
on this condition most of calls have voice related issue.

logs as upload on this link
https://onlinevoipstore.com/asterisklog.zip  


By: Amardeep (amarniit) 2021-12-22 23:59:33.301-0600

Now calls are 100 and 20 core cpu load is 30
By: Amardeep (amarniit) 2021-12-23 01:08:03.962-0600

now load is 65
By: Benjamin Keith Ford (bford) 2021-12-29 08:03:20.934-0600

Can you get us a backtrace at the time CPU load is high?
Also, does CPU continually increase, or only when you add more calls?
By: Asterisk Team (asteriskteam) 2022-01-12 12:00:00.850-0600

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines