[Home]

Summary:ASTERISK-28478: Crash performing "core reload" with modified res_config_sqlite3.conf
Reporter:Dennis (dbuteyn)Labels:
Date Opened:2019-07-14 02:52:19Date Closed:2019-07-24 18:59:25
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Resources/res_config_sqlite3
Versions:13.25.0 Frequency of
Occurrence
Constant
Related
Issues:
is related toASTERISK-28477 Crash when not specifying "dbfile" in res_config_sqlite3.conf
Environment:CentOS7Attachments:( 0) asterisk-config.tar.gz
( 1) config.log
Description:{code:title=original res_config_sqlite3.conf}
[asterisk]
dbfile => /var/lib/asterisk/somefile.sqlite3
{code}

{code:title=changed res_config_sqlite3.conf}
[asterisk]
dbfile => /var/lib/asterisk/anotherfile.sqlite3
{code}

{code:title=journalctl -u asterisk (starting from "core reload")}
Jul 14 07:46:06 erol asterisk[25998]: [2019-07-14 07:46:06] NOTICE[26113]: cdr.c:4459 cdr_toggle_runtime_options: CDR simple logging enabled.
Jul 14 07:46:06 erol asterisk[25998]: [2019-07-14 07:46:06] WARNING[26113]: res_digium_phone.c:606 reload: No Valid DPMA License found.
Jul 14 07:46:06 erol systemd[1]: asterisk.service: main process exited, code=killed, status=11/SEGV
{code}

{code:title=backtrace}
(gdb) bt
#0  0x00007fe6d082feb1 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fe656facb19 in db_destructor (obj=0x7fe6c400e338) at res_config_sqlite3.c:237
#2  0x000000000045c922 in internal_ao2_ref (func=0x61e1d3 <__FUNCTION__.9420> "__ao2_ref", line=523, file=0x61df78 "astobj2.c", delta=-1, user_data=0x7fe6c400e338)
   at astobj2.c:456
#3  __ao2_ref (user_data=user_data@entry=0x7fe6c400e338, delta=delta@entry=-1) at astobj2.c:523
#4  0x00007fe656facde6 in unref_db (db=<synthetic pointer>) at res_config_sqlite3.c:252
#5  update_realtime_sqlite3_db (cat=0x7fe6c4007990 "stuffs", config=0x7fe6c400d690, db=0x2ba59b8) at res_config_sqlite3.c:471
#6  parse_config (reload=reload@entry=1) at res_config_sqlite3.c:1293
#7  0x00007fe656fad0be in reload () at res_config_sqlite3.c:1312
#8  0x000000000053a4ca in ast_module_reload (name=name@entry=0x0) at loader.c:1058
#9  0x00000000004d07ca in handle_core_reload (e=<optimized out>, cmd=<optimized out>, a=<optimized out>) at cli.c:325
#10 0x00000000004d77ff in ast_cli_command_full (uid=uid@entry=0, gid=gid@entry=0, fd=fd@entry=23, s=s@entry=0x7fe6cce94600 "core reload") at cli.c:2854
#11 0x00000000004d792b in ast_cli_command_multiple_full (uid=0, gid=0, fd=23, size=12, s=0x7fe6cce948e0 "core reload") at cli.c:2880
#12 0x0000000000457a48 in netconsole (vconsole=vconsole@entry=0x8e6ee0 <consoles>) at asterisk.c:1554
#13 0x00000000005f978e in dummy_start (data=<optimized out>) at utils.c:1239
#14 0x00007fe6d082edd5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fe6cfb0bead in clone () from /lib64/libc.so.6
{code}
Comments:By: Asterisk Team (asteriskteam) 2019-07-14 02:52:20.204-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

By: Sean Bright (seanbright) 2019-07-15 07:28:27.660-0500

I'm not able to reproduce with Asterisk 13.27.1 (or 13 from git). Can you try upgrading to 13.27.1 and see if that resolves the problem?

By: Dennis (dbuteyn) 2019-07-16 06:27:56.874-0500

Yup. Issue persists with 13.27.1 from git.

For what its worth, it seems pthread_join is very upset that "db->syncthread" is NULL in db_stop_batch():

{code:title=(gdb) p *(realtime_sqlite3_db *) obj}
$1 = {
__field_mgr_pool = 0x0,
name = 0x8e5912 <__ast_string_field_empty_buffer+2> "",
filename = 0x8e5912
<__ast_string_field_empty_buffer+2> "",
__field_mgr = {
last_alloc = 0x1d52ef2 "/var/lib/asterisk/anotherfile.sqlite3",
header = 0x0
},
handle = 0x0,
syncthread = 0,
cond = {
__data = {
__lock = 0,
__futex = 0,
__total_seq = 0,
__wakeup_seq = 0,
__woken_seq = 0,
__mutex = 0x0,
__nwaiters = 0,
__broadcast_seq = 0
},
__size = '\000' <repeats 47 times>,
__align = 0
},
requirements = 0,
dirty = 0,
debug = 1,
exiting = 1,
wakeup = 1,
batch = 100
}
{code}

By: Sean Bright (seanbright) 2019-07-16 06:44:35.378-0500

Not able to reproduce. Can you attach your entire {{/etc/asterisk}} directory to this ticket (a {{tar.gz}} is fine).

Also, your {{config.log}} from your build.

By: Sean Bright (seanbright) 2019-07-16 07:50:09.028-0500

Also, from your backtrace:

{noformat}
#5  update_realtime_sqlite3_db (cat=0x7fe6c4007990 "stuffs", config=0x7fe6c400d690, db=0x2ba59b8) at res_config_sqlite3.c:471
{noformat}

That indicates a category called {{"stuffs"}} which is not present in either your original or changed configuration, so I'm not sure how it is possible that you got that backtrace with those configs.

By: Dennis (dbuteyn) 2019-07-18 06:53:29.891-0500

The "stuffs" discrepancy is because at the time I was in the middle of running tests for an unreleased feature for one of our products. Asterisk config was not entirely "clean" nor identical, but sufficiently so to reproduce the problem. (Problem being that "core reload" after making changes to res_config_sqlite3.conf would crash Asterisk)

I'll provide minimal Asterisk config files with pristine 13.27.1 build when able (our standard build contains a few unrelated patches for DAHDi and OpenR2). The server I was using is currently under test (for said feature) and I currently have no spare servers to molest. Please bear with me.

By: Dennis (dbuteyn) 2019-07-23 06:20:50.573-0500

Still crashes with pristine build, attached config.log and /etc/asterisk

By: Dennis (dbuteyn) 2019-07-24 03:43:43.922-0500

Asterisk no longer crashes with patch from ASTERISK-28477.

By: Friendly Automation (friendly-automation) 2019-07-24 18:59:26.306-0500

Change 11619 merged by Friendly Automation:
res_config_sqlite3: Only join threads that we started

[https://gerrit.asterisk.org/c/asterisk/+/11619|https://gerrit.asterisk.org/c/asterisk/+/11619]

By: Friendly Automation (friendly-automation) 2019-07-25 07:01:51.423-0500

Change 11618 merged by George Joseph:
res_config_sqlite3: Only join threads that we started

[https://gerrit.asterisk.org/c/asterisk/+/11618|https://gerrit.asterisk.org/c/asterisk/+/11618]

By: Friendly Automation (friendly-automation) 2019-07-25 07:02:22.165-0500

Change 11620 merged by George Joseph:
res_config_sqlite3: Only join threads that we started

[https://gerrit.asterisk.org/c/asterisk/+/11620|https://gerrit.asterisk.org/c/asterisk/+/11620]