[Home]

Summary:ASTERISK-16656: [patch] Crash with autoclear=yes with MYSQL() application
Reporter:Wolfgang Pichler (wuwu)Labels:
Date Opened:2010-09-08 02:13:18Date Closed:2011-05-06 15:09:27
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Addons/app_mysql
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) issue17964_addon_1.6.2_svn.patch
Description:i am experiencing random segfaults on different asterisk 1.6.2.x (9-10) versions - always 64bit plattform - mostly centos 5.5 - one debian version.

the backtrace does look like this

#0  0x000072656c6c6143 in ?? ()
#1  0x00002aaaaaf22a91 in ?? () from /usr/lib64/mysql/libmysqlclient.so.15
#2  0x00002aaaaaf22e09 in my_net_read () from /usr/lib64/mysql/libmysqlclient.so.15
#3  0x00002aaaaaf1c6e2 in cli_safe_read () from /usr/lib64/mysql/libmysqlclient.so.15
#4  0x00002aaaaaf1d7d9 in ?? () from /usr/lib64/mysql/libmysqlclient.so.15
ASTERISK-1  0x00002aaaaaf1bd1e in mysql_real_query () from /usr/lib64/mysql/libmysqlclient.so.15
ASTERISK-2  0x00002aaab104397f in mysql_init () from /usr/lib/asterisk/modules/app_addon_sql_mysql.so
ASTERISK-3  0x00000000004b288d in pbx_exec (c=0x2aaaac074570, app=0x2aaab443d180, data=0x4062f990) at pbx.c:1348
ASTERISK-4  0x00000000004bdcdc in pbx_extension_helper (c=0x2aaaac074570, con=<value optimized out>, context=0x2aaaac074930 "enterPICA", exten=0x2aaaac074980 "118899", priority=15, label=0x0,
   callerid=0x2aaaac07f210 "00436763621526", action=E_SPAWN, found=0x4063200c, combined_find_spawn=1) at pbx.c:3711
ASTERISK-5  0x00000000004c24e2 in ast_spawn_extension (c=0x2aaaac074570, args=0x0) at pbx.c:4181
ASTERISK-6 __ast_pbx_run (c=0x2aaaac074570, args=0x0) at pbx.c:4275
ASTERISK-7 0x00000000004c3abb in pbx_thread (data=0x2aaab4498c40) at pbx.c:4562
ASTERISK-8 0x00000000004fdfbc in dummy_start (data=<value optimized out>) at utils.c:968
ASTERISK-9 0x000000384880673d in start_thread () from /lib64/libpthread.so.0
ASTERISK-10 0x00000038480d3d1d in clone () from /lib64/libc.so.6

I do make heavy use of the MYSQL functions - in the database there are also some stored procedures and triggers and functions - also there are some multi resultset results in my queries (where i only use the first resultset)

I now don't know exactly - will this be more a libmysqlclient bug - or is it more a asterisk bug.

The segfault does always show the same backtrace - so it should be more a libmysql specific bug - or i am wrong ?

Should i do compile the libmysqlclient for myself to get the debugging symbols ?

****** ADDITIONAL INFORMATION ******

MySQL Client Version used is

mysql-devel-5.0.77-4.el5_5.3

is libmysqlclient1.5 - there is already a libmysqlclient1.6 available - but can use it out of the box on my machine
Comments:By: Leif Madsen (lmadsen) 2010-09-15 15:04:08

Moved out of libpri as this is not a libpri issue.

By: Leif Madsen (lmadsen) 2010-09-15 15:09:04

First, your backtrace is not useful. You must compile Asterisk via the DONT_OPTIMIZE Compiler Flag to make backtraces useful.

Second, you probably shouldn't be using the packaged libmysqlclient library. It is best to use the latest release compiled from source and link against that. The same thing is recommended when using unixODBC.

Additionally, I would suggest you not use the MYSQL() dialplan application. Much more work has gone into res_odbc and func_odbc modules, and is the recommended method for MySQL interaction from the database. Not only is it more stable and better supported, your dialplan will become a lot simpler. In most cases 5-7 lines of dialplan for MYSQL() can be dropped into a single line since transaction support and all that jazz does not need to be done explicitly from the dialplan (it is handled by the module).

By: Leif Madsen (lmadsen) 2010-09-15 15:09:51

Thank you for your bug report. In order to move your issue forward, we require a backtrace from the core file produced after the crash. Please see the doc/backtrace.txt file in your Asterisk source directory.

Also, be sure you have DONT_OPTIMIZE enabled in menuselect within the Compiler Flags section, then:

make install

after enabling, reproduce the crash, and then execute the instructions in doc/backtrace.txt.

When complete, attach that file to this issue report. Thanks!

By: Wolfgang Pichler (wuwu) 2010-09-16 01:17:51

I will recompile asterisk tonight with DONT_OPTIMIZE enabled - and will then upload the next core dump.
I am using MYSQL instead of res_odbc - because i thought it will have less overhead using mysql directly - instead of going over odbc. And, normally - keep it simple and stupid is the best way.... But i did not tried res_odbc until yet - i will setup a test machine and replace there MYSQL with res_odbc - maybe it will solve my problems

I have found another repo for mysql - ourdelta - it seems that these builds are more up2date as the centos builds.

By: philipp2 (philipp2) 2010-09-16 05:21:13

Yesterday I turned on "autoclear=yes" in mysql.conf and since then I have had already two crashes (Asterisk 1.4.33.1, Debian 2.6.26-2-amd64, mysql  Ver 14.12 Distrib 5.0.51a):

kernel: [4981727.038185] asterisk[29436]: segfault at 49 ip 7f13cea3f9ba sp 40d12de0 error 4 in libmysqlclient.so.15.0.0[7f13ce9ba000+1bf000]
kernel: [4990799.437091] asterisk[1275]: segfault at 49 ip 7f9214bf99ba sp 40e0ae00 error 4 in libmysqlclient.so.15.0.0[7f9214b74000+1bf000]

So you might want to set autoclear=no and see if that helps.



By: Wolfgang Pichler (wuwu) 2010-09-17 07:54:33

autoclear is yes at my setup.

I have now changed it to no.

I will report back if it was helpful

Thanks

By: Wolfgang Pichler (wuwu) 2010-09-21 01:17:38

system is now running since 4 days without crash with autoclear=no - processed over 18.000 calls....

So it really seems that the autoclear=yes caused the problem.

The problem must occour somewhere in the asterisk-addon-mysql - at  mysql_ds_destroy or mysql_ds_fixup

mysql_close or mysql_free_result - when get called with null pointer will crash - or are the both functions safe to null pointers ?

Adding some debugging code to it would help - maybe i do find some time to play around with it.

By: Sean Bright (seanbright) 2011-05-05 20:38:31

Still running without a crash with autoclear=no?

By: Tilghman Lesher (tilghman) 2011-05-05 20:48:56

wuwu:  you aren't doing something silly like sharing a single MySQL connection across multiple channels, are you?  That is explicitly a forbidden behavior with autoclear=yes.

By: Sean Bright (seanbright) 2011-05-05 20:50:11

I've attached a patch that might fix this problem.  Please apply and test and report your results (with autoclear=yes) if you can.

By: Wolfgang Pichler (wuwu) 2011-05-05 21:53:20

system was running now stable for the last months with autoclear=no. The connection was not directly shared across multiple channels - it was shared along the call path - and maybe with the destination channel (in the hangup handler to write the last message - and to close it). I will test the patch next week - thanks seanbright.

By: Digium Subversion (svnbot) 2011-05-06 14:24:14

Repository: asterisk
Revision: 317837

U   branches/1.8/addons/app_mysql.c

------------------------------------------------------------------------
r317837 | russell | 2011-05-06 14:24:14 -0500 (Fri, 06 May 2011) | 11 lines

Fix a crash in the MySQL() application.

This code was not handling channel datastores safely.  The channel
must be locked.

(closes issue ASTERISK-16656)
Reported by: wuwu
Patches:
     issue17964_addon_1.6.2_svn.patch uploaded by seanbright (license 71)
Tested by: wuwu

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=317837

By: Digium Subversion (svnbot) 2011-05-06 14:25:37

Repository: asterisk
Revision: 317843

_U  trunk/
U   trunk/addons/app_mysql.c

------------------------------------------------------------------------
r317843 | russell | 2011-05-06 14:25:37 -0500 (Fri, 06 May 2011) | 18 lines

Merged revisions 317837 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.8

........
 r317837 | russell | 2011-05-06 14:24:11 -0500 (Fri, 06 May 2011) | 11 lines
 
 Fix a crash in the MySQL() application.
 
 This code was not handling channel datastores safely.  The channel
 must be locked.
 
 (closes issue ASTERISK-16656)
 Reported by: wuwu
 Patches:
       issue17964_addon_1.6.2_svn.patch uploaded by seanbright (license 71)
 Tested by: wuwu
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=317843

By: Sean Bright (seanbright) 2011-05-06 15:09:27

This was closed inadvertently, but if the patch doesn't fix your crash please re-open.