[Home]

Summary:ASTERISK-29385: Asterisk sometimes freezes
Reporter:Dan Erik Ollas (DanErikOllas)Labels:
Date Opened:2021-04-09 10:58:40Date Closed:2021-04-26 12:00:00
Priority:MajorRegression?
Status:Closed/CompleteComponents:Core/Platform
Versions:18.1.0 Frequency of
Occurrence		Occasional
Related
Issues:
Environment:It is installed on a Linux (Redhat) server. Asterisk is interacting with our CallCenter software using AGI-scripts. There are 5000–10000 calls/day. Asterisk restarts unexpectedly about once á month, last time 2021-04-08 Our asterisk is installed under /opt/asterisk181/ Attachments:( 0) core-brief.txt
( 1) core-full.txt
( 2) core-info.txt
( 3) core-locks.txt
( 4) core-thread1.txt
Description:Asterisk core freezes, drops all ongoing calls and restarts.
No useful info in log-files, only this:
----------------------------------------------------------------------------------------
{noformat}
/var/log/messages
Apr  8 16:02:06 xxx asterisk18: /opt/asterisk181/sbin/safe_asterisk: line 171: 25097 Segmentation fault      (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk"
-f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
Apr  8 16:02:06 xxx asterisk18: cat: /opt/asterisk181/var/run/asterisk/asterisk.pid: No such file or directory
Asterisk log-file messages:
.....
[2021-04-08 16:02:02.567] VERBOSE[28378][C-0000eb6d] res_musiconhold.c: Started music on hold, class 'dialtone', on channel 'PJSIP/tdc-0001c27f'
[2021-04-08 16:02:11.292] Asterisk 18.1.0 built by root @ xxx on a x86_64 running Linux on 2021-03-21 18:36:21 UTC
{noformat}
(Just a restart after 9 seconds of no logging)
----------------------------------------------------------------------------------------

But I managed to get the "core" from last crash and I have done "ast_coredumper" so I can provide these files:
core-thread1.txt
core-brief.txt
core-full.txt
core-locks.txt
core-info.txt

Really hope you can help. And we are willing to contribute with a significant donation if you can solve.


Regards,
/Dan Erik
Comments:By: Asterisk Team (asteriskteam) 2021-04-09 10:58:41.366-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/].
By: Kevin Harwell (kharwell) 2021-04-09 11:38:42.208-0500

[~DanErikOllas] Yes please attach the core-*.txt files you gathered to this issue.
By: Dan Erik Ollas (DanErikOllas) 2021-04-09 12:56:27.460-0500

Here are the files.
/Dan Erik
By: Kevin Harwell (kharwell) 2021-04-09 13:35:14.252-0500

Thanks, unfortunately the backtrace is missing debug symbols so there is no way to really tell what happened. You will need to rebuild Asterisk to include the debug symbols, or if installed via a package manager then check to see if the packager also supplies an Asterisk debug symbols package and install it.

Then when the issue occurs again re-run _ast_coredumper_ against the newly produced core file.

Please see [Getting a backtrace|https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace] wiki for more information.
By: Kevin Harwell (kharwell) 2021-04-09 13:39:43.110-0500

Also this is really odd as I see you've reported it as Asterisk 18.1 and in the log output it shows:
{noformat}
[2021-04-08 16:02:11.292] Asterisk 18.1.0 built by root @ xxx on a x86_64 running Linux on 2021-03-21 18:36:21 UTC
{noformat}
However, in the collected core-info it has a different version:
{noformat}
Asterisk 1.6.0.26 built by root @ servit7 on a x86_64 running Linux on 2015-11-13 20:14:13 UTC
{noformat}
It seems the core file might have been produced by an older version?

Also with regards to the version I'd recommend upgrading to the latest version of Asterisk 18 and see if the problem still persists. It's possible this problem has already been fixed.
By: Dan Erik Ollas (DanErikOllas) 2021-04-09 14:51:41.138-0500

Ok, thank you for info.
Really odd this with ver 1.6.0.26. Actually it is a previous version that we upgraded from for some time ago.
But this gives me a clue .... It can be part of the problem. Maybe I need to first really remove everything from old version (make distclean) and the recompile (with backtrace)

Regards,
/Dan Erik
By: Kevin Harwell (kharwell) 2021-04-09 15:10:26.346-0500

In all likelihood it's probably a core file then that was left over from when the previous version was installed/running.

But yeah anytime you rebuild Asterisk with a new version (well for sure a major version at least), then I'd recommend the following:
{noformat}
$ make uninstall
$ make distclean
$ ./configure <your options>
$ make
$ make menuselect # optional
$ make install
{noformat}

By: Kevin Harwell (kharwell) 2021-04-09 15:15:00.763-0500

Also if the core file being written out is written out using the same name every time then I'd also recommend removing (delete, move, or rename) the previously written core file once you're done with it. For some unknown reason (to me) I have found occasionally corrupted core files after one was overwritten.
By: Dan Erik Ollas (DanErikOllas) 2021-04-10 14:53:31.657-0500

One thing I'm sure of is that the core was produced the same minute as the crash.

I don't understand "rebuild Asterisk to include the debug symbols".
Does this mean turn on DEBUG_THREADS, DEBUG_FD_LEAKS?

Sorry, this is last question. I understand you are busy doing other stuff.

Can you recommend any consultant who can help us with this?
By: Kevin Harwell (kharwell) 2021-04-12 11:26:05.423-0500

No worries. To ensure [Asterisk is built with debug symbols|https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-PreparingAsteriskToProduceCoreFilesOnCrash]:
{quote}
This can be done by selecting the 'DONT_OPTIMIZE' option in the Compiler Flags submenu in the 'make menuselect' tree before building Asterisk.
{quote}

By: Asterisk Team (asteriskteam) 2021-04-26 12:00:00.656-0500

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines