[Home]

Summary:ASTERISK-17817: [patch] Asterisk res_odbc causes SEGFAULT's if unable to connect to server.
Reporter:Ian Anderson (seadweller)Labels:
Date Opened:2011-05-08 09:38:11Date Closed:2011-05-23 09:34:32
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Resources/res_odbc
Versions:1.6.2.18 Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20110514__issue19250.diff.txt
( 1) backtrace_nonoptimized.txt
( 2) odbc_local_crash_full_bt.txt
( 3) odbc_remote_crash_full_bt.txt
( 4) res_odbc.conf
Description:I have res_odbc.conf configured with both a local and remote DSN.  If connectivity is lost to the either the local remote DSN, Asterisk SEGFAULT's.

Was seeing this issue intermittently, but when I place an IPTABLES rule to block outbound ODBC connections to the remote DSN or disable the local MySQL server, Asterisk will crash each time I start it up.

I also only tested with the local MySQL DSN listed in res_odbc.conf and odbc.ini and still had the same crashed with the local MySQL turned off.

A related bug is that it seems sometimes Asterisk loses contact with the local MySQL connection (via /var/lib/mysql/mysql.sock) and that also triggers this connection crash.

Desired outcome is that Asterisk should continue to run and periodically attempt reconnects.

CentOS 5.6 x86_64
Asterisk 1.6.2.18
MySQL 5.5.11
MySQL ODBC Connector 5.1.8

Full BT's attached.

****** ADDITIONAL INFORMATION ******

Debug log when blocking local MySQL (mysqld turned off):
---
[May  8 14:13:07] NOTICE[24556] res_odbc.c: Connecting asterisk-remote
[May  8 14:13:07] NOTICE[24556] res_odbc.c: res_odbc: Connected to asterisk-remote [asterisk-connector-remote]
[May  8 14:13:07] DEBUG[24556] res_odbc.c: odbc_release_obj2(0x2aaaac008a98) called (obj->txf = (nil))
[May  8 14:13:07] NOTICE[24556] res_odbc.c: Registered ODBC class 'asterisk-remote' dsn->[asterisk-connector-remote]
[May  8 14:13:07] NOTICE[24556] res_odbc.c: Connecting asterisk-local
[May  8 14:13:07] WARNING[24556] res_odbc.c: res_odbc: Error SQLConnect=-1 errno=2002 [unixODBC][MySQL][ODBC 5.1 Driver]Can't connect to local MySQL server through socket '/var/lib/mysq
[May  8 14:13:07] WARNING[24556] res_odbc.c: Failed to connect to asterisk-local
**** CRASH ****

GDB Backtrace when blocking local MySQL
---#0  ast_atomic_fetchadd_int (name=0x2aaaac2fb750 "asterisk-local", flags=<value optimized out>) at /usr/src/asterisk/asterisk-1.6.2.18/include/asterisk/lock.h:1988
#1  ast_odbc_request_obj2 (name=0x2aaaac2fb750 "asterisk-local", flags=<value optimized out>) at res_odbc.c:1197
#2  0x00002aaaaacefde8 in odbc_register_class () at res_odbc.c:970
#3  load_odbc_config () at res_odbc.c:861
#4  0x00002aaaaacf00f1 in load_module () at res_odbc.c:1694
ASTERISK-1  0x000000000049ad03 in start_resource (mod=0x2aaaac000020) at loader.c:743
ASTERISK-2  0x000000000049b839 in load_resource_list (load_order=0x7fff10863c30, global_symbols=1, mod_count=0x7fff10863c4c) at loader.c:921
ASTERISK-3  0x000000000049bc9a in load_modules (preload_only=0) at loader.c:1063
ASTERISK-4  0x0000000000434dd4 in main (argc=<value optimized out>, argv=0x7fff10866258) at asterisk.c:3754
ASTERISK-5  0x00000037d861d994 in __libc_start_main () from /lib64/libc.so.6
ASTERISK-6 0x000000000041b339 in pthread_attr_setdetachstate ()
ASTERISK-7 0x00007fff10866248 in ?? ()
ASTERISK-8 0x0000000000000000 in ?? ()


Debug log when blocking remote MySQL (via iptables):
---
[May  8 13:59:43] VERBOSE[24035] config.c:   == Parsing '/etc/asterisk/res_odbc.conf': [May  8 13:59:43] VERBOSE[24035] config.c:   == Found
[May  8 13:59:43] NOTICE[24035] res_odbc.c: Connecting asterisk-remote
[May  8 13:59:46] VERBOSE[24040] asterisk.c:     -- Remote UNIX connection
[May  8 13:59:53] WARNING[24035] res_odbc.c: res_odbc: Error SQLConnect=-1 errno=2003 [unixODBC][MySQL][ODBC 5.1 Driver]Can't connect to MySQL server on '<commented out>' (4)
[May  8 13:59:53] WARNING[24035] res_odbc.c: Failed to connect to asterisk-remote
**** CRASH ****

GDB Backtrace when blocking remote MySQL
---#0  ast_atomic_fetchadd_int (name=0x2aaaac2d3ec0 "asterisk-remote", flags=<value optimized out>) at /usr/src/asterisk/asterisk-1.6.2.18/include/asterisk/lock.h:1988
#1  ast_odbc_request_obj2 (name=0x2aaaac2d3ec0 "asterisk-remote", flags=<value optimized out>) at res_odbc.c:1197
#2  0x00002aaaaacefde8 in odbc_register_class () at res_odbc.c:970
#3  load_odbc_config () at res_odbc.c:861
#4  0x00002aaaaacf00f1 in load_module () at res_odbc.c:1694
ASTERISK-1  0x000000000049ad03 in start_resource (mod=0x2aaaac000020) at loader.c:743
ASTERISK-2  0x000000000049b839 in load_resource_list (load_order=0x7fffd87e5f30, global_symbols=1, mod_count=0x7fffd87e5f4c) at loader.c:921
ASTERISK-3  0x000000000049bc9a in load_modules (preload_only=0) at loader.c:1063
ASTERISK-4  0x0000000000434dd4 in main (argc=<value optimized out>, argv=0x7fffd87e8558) at asterisk.c:3754
ASTERISK-5  0x00000037d861d994 in __libc_start_main () from /lib64/libc.so.6
ASTERISK-6 0x000000000041b339 in pthread_attr_setdetachstate ()
ASTERISK-7 0x00007fffd87e8548 in ?? ()
ASTERISK-8 0x0000000000000000 in ?? ()
Comments:By: Ian Anderson (seadweller) 2011-05-08 11:18:00

Repeated the same setup on an Amazon EC2 instance and receive the same crash on startup connection failure.

Also installed 1.4.41 with same MySQL Connector and unixODBC... at startup Asterisk complains it can't connect, but continues to run as it should.

EDIT:  Turning *off* connection pooling in res_odbc.conf seems to resolve the issue in 1.6.  If a DSN is down, it will not SEGFAULT.


By: Leif Madsen (lmadsen) 2011-05-09 08:47:55

I am seeing some <value optimized out> which leads me to believe you don't have DONT_OPTIMIZE enabled in the Compiler Flags of menuselect.

Could you enable that flag, recompile and reinstall Asterisk, then reproduce the error, followed by providing the backtraces again? That may help the developers further.

Thanks!
By: Ian Anderson (seadweller) 2011-05-09 09:03:26

Non-optimized Asterisk backtrace uploaded.

I shutdown my local MySQL before starting Asterisk... so it crashes on the failed attempt to it.


By: Ian Anderson (seadweller) 2011-05-09 14:31:14

Thanks for the quick patch.  I turned connection pooling back on, disabled my MySQL server and no longer receive the crash on startup.

However, "odbc show" displays incorrect/misleading info.  There is no active connection but I see the following:

localhost*CLI> odbc show

ODBC DSN Settings
-----------------

 Name:   asterisk-local
 DSN:    asterisk-connector-local
 Pooled: Yes
 Limit:  100
 Connections in use: 1

localhost*CLI>


EDIT:  After running Asterisk and flapping my MySQL server a few times, I also see an incorrect Connection in use counter:

 Name:   asterisk-local
 DSN:    asterisk-connector-local
 Pooled: Yes
 Limit:  100
 Connections in use: 3
   - Connection 1: connected


By: Tilghman Lesher (tilghman) 2011-05-14 14:42:38

Patch updated.
By: Sebastian Gutierrez (sum) 2011-05-15 22:26:12

Tested the patch, solve the problem I had 0019294 (same as this).
By: Ian Anderson (seadweller) 2011-05-17 09:19:14

Seems to work well for me!  Thanks!
By: Digium Subversion (svnbot) 2011-05-23 09:34:23

Repository: asterisk
Revision: 320444

U   branches/1.6.2/res/res_odbc.c

------------------------------------------------------------------------
r320444 | tilghman | 2011-05-23 09:34:22 -0500 (Mon, 23 May 2011) | 8 lines

Don't crash when the connection fails.

(closes issue ASTERISK-17817)
Reported by: seadweller
Patches:
      20110514__issue19250.diff.txt uploaded by tilghman (license 14)
Tested by: seadweller, sum

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=320444
By: Digium Subversion (svnbot) 2011-05-23 09:34:27

Repository: asterisk
Revision: 320445

_U  branches/1.8/
U   branches/1.8/res/res_odbc.c

------------------------------------------------------------------------
r320445 | tilghman | 2011-05-23 09:34:27 -0500 (Mon, 23 May 2011) | 15 lines

Merged revisions 320444 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.6.2

........
 r320444 | tilghman | 2011-05-22 18:25:51 -0500 (Sun, 22 May 2011) | 8 lines
 
 Don't crash when the connection fails.
 
 (closes issue ASTERISK-17817)
  Reported by: seadweller
  Patches:
        20110514__issue19250.diff.txt uploaded by tilghman (license 14)
  Tested by: seadweller, sum
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=320445
By: Digium Subversion (svnbot) 2011-05-23 09:34:31

Repository: asterisk
Revision: 320446

_U  trunk/
U   trunk/res/res_odbc.c

------------------------------------------------------------------------
r320446 | tilghman | 2011-05-23 09:34:31 -0500 (Mon, 23 May 2011) | 22 lines

Merged revisions 320445 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.8

................
 r320445 | tilghman | 2011-05-22 18:34:57 -0500 (Sun, 22 May 2011) | 15 lines
 
 Merged revisions 320444 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.6.2
 
 ........
   r320444 | tilghman | 2011-05-22 18:25:51 -0500 (Sun, 22 May 2011) | 8 lines
   
   Don't crash when the connection fails.
   
   (closes issue ASTERISK-17817)
    Reported by: seadweller
    Patches:
          20110514__issue19250.diff.txt uploaded by tilghman (license 14)
    Tested by: seadweller, sum
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=320446