[Home]

Summary:ASTERISK-22350: DUNDI - core dump on shutdown - segfault in sqlite3_reset from /usr/lib/libsqlite3.so.0
Reporter:Birger "WIMPy" Harzenetter (wimpy)Labels:
Date Opened:2013-08-20 16:58:58Date Closed:2013-12-19 10:40:01.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:Core/AstDB PBX/pbx_dundi
Versions:SVN Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) pastebin.txt
( 1) pastebin.txt
Description:segfault in sqlite3 after process_clearcache

Asterisk was running in foreground and was killed using ^C
Comments:By: Birger "WIMPy" Harzenetter (wimpy) 2013-08-20 17:01:24.146-0500

gdb output

By: Birger "WIMPy" Harzenetter (wimpy) 2013-08-20 18:20:29.237-0500

output of thread apply all bt
By: Rusty Newton (rnewton) 2013-08-21 16:47:23.142-0500

What branch was this reproduced in?

Do you have any steps for reproduction?

Can you also provide a VERBOSE and DEBUG log during the time of the shutdown?

Thanks!
By: Birger "WIMPy" Harzenetter (wimpy) 2013-08-21 21:06:14.172-0500

It happened when asterisk was running in foreground (asterisk -gc) and was killed by pressing ^C.

I cannot reproduce the fault (on todays version).

By: Corey Farrell (coreyfarrell) 2013-08-23 15:10:27.153-0500

Using Ctrl-C to shutdown causes SHUTDOWN_FAST.  Modules are not unloaded for SHUTDOWN_FAST or SHUTDOWN_NORMAL, but ast_register_atexit callbacks are run.

The 14k pastebin.txt shows that thread #1 is DUNDI trying to use ast_db_gettree.  Thread #12 is close_logger() waiting for the logger thread to end.  This means that astdb and most other things in the core have already been shutdown by ast_run_atexits.  Since asterisk exit's very shortly after ast_run_atexits() this is difficult to reproduce.
By: Corey Farrell (coreyfarrell) 2013-10-29 11:24:07.150-0500

This is an issue with asterisk 11+, probably 1.8 too.  I think backporting ast_register_cleanup and using it for db.c shutdown is the correct fix.  Otherwise pbx_dundi in 1.8/11 should use ast_register_atexit to kill it's background thread before db.c shuts down.
By: Alexander Hömig (ahoemig) 2013-11-06 05:57:30.220-0600

I have possible found the reason:

When the asterisk stops the "astdb_atexit"-Function will be called. This function calls "clean_statements()" which set the db-pointer in all prepared-Statements to null.
When at the same time a database operation occurs and uses a prepared-statement the segfault happens in sqlite3_reset.

{code:title=gdb-log|borderStyle=solid}
(gdb)
#0  0x4038d90e in sqlite3_reset (pStmt=0x82c7410) at sqlite3.c:63297
63297       sqlite3_mutex_enter(v->db->mutex);
(gdb) list
63292     int rc;
63293     if( pStmt==0 ){
63294       rc = SQLITE_OK;
63295     }else{
63296       Vdbe *v = (Vdbe*)pStmt;
63297       sqlite3_mutex_enter(v->db->mutex);
(gdb)
(gdb) print *((Vdbe*)pStmt)
$7 = {db = 0x0, aOp = 0x852dfa8, aMem = 0x852e1a0, apArg = 0x852e380, aColName = 0x0, pResultSet = 0x0, nMem = 9, nOp = 27,
 nOpAlloc = 51, nLabel = 2, aLabel = 0x0, nResColumn = 0, nCursor = 2, magic = 3053896648, zErrMsg = 0x0, pPrev = 0x0,
 pNext = 0x82c3ed0, apCsr = 0x852e388, aVar = 0x852e330, azVar = 0x852e380, nVar = 2, nzVar = 0, cacheCtr = 1, pc = -1, rc = 0,
 errorAction = 2 '\002', minWriteFileFormat = 255 '?', explain = 0, inVtabMethod = 0, changeCntOn = 1, expired = 0,
 runOnlyOnce = 0, usesStmtJournal = 0, readOnly = 0, isPrepareV2 = 0, doingRerun = 0, nChange = 0, btreeMask = 1, lockMask = 0,
 iStatement = 0, aCounter = {0, 0, 0}, startTime = 0, nFkConstraint = 0, nStmtDefCons = 0,
 zSql = 0x852a160 "`?R\bRT OR REPLACE INTO astdb (key, value) VALUES (?, ?)", pFree = 0x0, pFrame = 0x0, pDelFrame = 0x0,
 nFrame = 0, expmask = 0, pProgram = 0x0, nOnceFlag = 1, aOnceFlag = 0x852e390 ""}
(gdb)
{code}

Is it possible to set the prepared-statement-pointer to null in the function "clean_stmt" after sqlite3_finalize was called?