Summary:	ASTERISK-25102: res_config_odbc / libodbc - Asterisk core dumps with signal 6 when connection lost during query
Reporter:	Leandro Dardini (ldardini)	Labels:
Date Opened:	2015-05-19 10:23:04	Date Closed:
Priority:	Critical	Regression?
Status:	Open/New	Components:	Resources/res_config_odbc
Versions:	13.3.2	Frequency of Occurrence	Frequent
Related Issues:
Environment:	CentOS 7 64 bit. The same configuration on CentOS 6 64 bit works fine	Attachments:	( 0) backtrace.tar.gz ( 1) backtrace-2015-05-20T05_36.txt ( 2) backtrace-2015-05-23T16_57_46 ( 3) backtrace-2015-05-23T16_58_09 ( 4) backtrace-2015-05-23T16_59_53 ( 5) full-10minutes-around-16_55 ( 6) full-10minutes-around-5_36
Description:	Asterisk crashes with a signal 6. Asterisk activity is really low, but some ODBC commands are executed as well as the access to sipfriends table configured in extconfig.conf.
Comments:	By: Leandro Dardini (ldardini) 2015-05-19 10:29:48.131-0500 Several backtraces of asterisk core dumping, more available if needed By: Rusty Newton (rnewton) 2015-05-19 19:45:22.949-0500 Thanks, can you attach the backtraces directly (.txt) instead of inside an archive. This makes it faster for us when looking at issues. Can you also attach a full log with 'DEBUG' messages showing the minutes leading up to the crash? https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information Did this only happen the one time or are you able to reproduce? By: Leandro Dardini (ldardini) 2015-05-20 05:27:12.994-0500 I am not able to reproduce "on command", but it is enough to leave the server running and it happens quite regularly. These are the cores generated in the latest couple of days: -rw------- 1 root root  97951744 19 mag 03.00 /tmp/core.sip4.l33ttech.com-2015-05-19T03:00:04-0400 -rw------- 1 root root 114372608 19 mag 06.23 /tmp/core.sip4.l33ttech.com-2015-05-19T06:23:35-0400 -rw------- 1 root root  98893824 19 mag 06.46 /tmp/core.sip4.l33ttech.com-2015-05-19T06:46:20-0400 -rw------- 1 root root  98230272 19 mag 07.26 /tmp/core.sip4.l33ttech.com-2015-05-19T07:26:58-0400 -rw------- 1 root root  91078656 19 mag 07.44 /tmp/core.sip4.l33ttech.com-2015-05-19T07:44:32-0400 -rw------- 1 root root 100171776 19 mag 08.44 /tmp/core.sip4.l33ttech.com-2015-05-19T08:44:58-0400 -rw------- 1 root root  92536832 19 mag 09.00 /tmp/core.sip4.l33ttech.com-2015-05-19T09:00:29-0400 -rw------- 1 root root  96272384 19 mag 09.28 /tmp/core.sip4.l33ttech.com-2015-05-19T09:28:02-0400 -rw------- 1 root root  90959872 19 mag 09.34 /tmp/core.sip4.l33ttech.com-2015-05-19T09:34:57-0400 -rw------- 1 root root  92016640 19 mag 09.41 /tmp/core.sip4.l33ttech.com-2015-05-19T09:41:51-0400 -rw------- 1 root root  95158272 19 mag 09.55 /tmp/core.sip4.l33ttech.com-2015-05-19T09:55:49-0400 -rw------- 1 root root  96907264 19 mag 10.33 /tmp/core.sip4.l33ttech.com-2015-05-19T10:33:56-0400 -rw------- 1 root root  94621696 19 mag 10.53 /tmp/core.sip4.l33ttech.com-2015-05-19T10:53:02-0400 -rw------- 1 root root 105013248 19 mag 12.19 /tmp/core.sip4.l33ttech.com-2015-05-19T12:19:31-0400 -rw------- 1 root root  94027776 19 mag 12.37 /tmp/core.sip4.l33ttech.com-2015-05-19T12:37:20-0400 -rw------- 1 root root  90226688 19 mag 12.40 /tmp/core.sip4.l33ttech.com-2015-05-19T12:40:49-0400 -rw------- 1 root root  93683712 19 mag 13.01 /tmp/core.sip4.l33ttech.com-2015-05-19T13:01:29-0400 -rw------- 1 root root  93876224 19 mag 13.25 /tmp/core.sip4.l33ttech.com-2015-05-19T13:25:28-0400 -rw------- 1 root root  94683136 19 mag 13.44 /tmp/core.sip4.l33ttech.com-2015-05-19T13:44:35-0400 -rw------- 1 root root  92131328 19 mag 14.00 /tmp/core.sip4.l33ttech.com-2015-05-19T14:00:30-0400 -rw------- 1 root root  91521024 19 mag 14.06 /tmp/core.sip4.l33ttech.com-2015-05-19T14:06:54-0400 -rw------- 1 root root  92123136 19 mag 14.22 /tmp/core.sip4.l33ttech.com-2015-05-19T14:22:54-0400 -rw------- 1 root root 154939392 19 mag 16.34 /tmp/core.sip4.l33ttech.com-2015-05-19T16:34:16-0400 -rw------- 1 root root 117080064 19 mag 18.01 /tmp/core.sip4.l33ttech.com-2015-05-19T18:01:01-0400 -rw------- 1 root root  94699520 19 mag 18.01 /tmp/core.sip4.l33ttech.com-2015-05-19T18:01:17-0400 -rw------- 1 root root  92491776 19 mag 18.01 /tmp/core.sip4.l33ttech.com-2015-05-19T18:01:25-0400 -rw------- 1 root root  99024896 19 mag 18.50 /tmp/core.sip4.l33ttech.com-2015-05-19T18:50:09-0400 -rw------- 1 root root  95473664 19 mag 19.29 /tmp/core.sip4.l33ttech.com-2015-05-19T19:29:09-0400 -rw------- 1 root root  94138368 19 mag 20.00 /tmp/core.sip4.l33ttech.com-2015-05-19T20:00:59-0400 -rw------- 1 root root  91570176 19 mag 20.11 /tmp/core.sip4.l33ttech.com-2015-05-19T20:11:36-0400 -rw------- 1 root root 100417536 19 mag 21.16 /tmp/core.sip4.l33ttech.com-2015-05-19T21:16:23-0400 -rw------- 1 root root  90333184 19 mag 21.19 /tmp/core.sip4.l33ttech.com-2015-05-19T21:19:00-0400 -rw------- 1 root root  98185216 19 mag 22.54 /tmp/core.sip4.l33ttech.com-2015-05-19T22:54:06-0400 -rw------- 1 root root 101847040 19 mag 23.50 /tmp/core.sip4.l33ttech.com-2015-05-19T23:50:01-0400 -rw------- 1 root root  95322112 20 mag 00.48 /tmp/core.sip4.l33ttech.com-2015-05-20T00:48:34-0400 -rw------- 1 root root  91013120 20 mag 01.04 /tmp/core.sip4.l33ttech.com-2015-05-20T01:04:47-0400 -rw------- 1 root root 110129152 20 mag 04.27 /tmp/core.sip4.l33ttech.com-2015-05-20T04:27:04-0400 -rw------- 1 root root  98545664 20 mag 05.36 /tmp/core.sip4.l33ttech.com-2015-05-20T05:36:12-0400 I have attached the full debug log of 10 minutes across the last core dump and the corresponding backtrace. Just to void any hardware problem, the same install has been made also on another different server and the problem is happening with the same frequency. By: Leandro Dardini (ldardini) 2015-05-20 05:28:47.988-0500 Backtrace of the core at 5:36 and Asterisk full debug of 10 minutes from 5:30 By: Leandro Dardini (ldardini) 2015-05-20 05:30:39.749-0500 I have uploaded the backtrace of the latest core and the full debug of 10 minutes around that time By: Rusty Newton (rnewton) 2015-05-20 17:43:12.942-0500 Did this only start occurring when you upgraded between minor 13 versions? or did you do an upgrade from 11 to 13? Go ahead and post a few more traces and logs as you get them. It will be useful for the developers to compare a few. By: Leandro Dardini (ldardini) 2015-05-21 05:39:56.675-0500 This server was installed with asterisk 13.3.2, no other version has been tested on it. It starts getting the core dumps as soon as it was installed. Once noticed the bad behavior, a new different server has been installed, still with CentOS 7 64 bit, but the problem was the same. A third server was installed with CentOS 6 64 bit and it was correctly working so far. By: Matt Jordan (mjordan) 2015-05-21 09:03:59.516-0500 The fact that this issue does not occur with a different version of the library on a different version of CentOS is telling. My inclination is that this is not a bug in Asterisk, but is a regression in the {{libmyodbc5.so}} library. You should probably raise an issue on their issue tracker. I'm fine with holding this open for now, particularly if you want to go raise the issue with that project. I don't expect much movement on this issue, since at best we would be looking at implementing a workaround for that library, which is generally not a great idea. By: Leandro Dardini (ldardini) 2015-05-21 11:03:36.030-0500 I will do, but just before proceeding, I remembered I have another asterisk running on CentOS 7 64 bit and checked if that was having the same issues. It was not. It was asterisk 12.7.1. However it is a server not currently "hard working", so I am going to recompile asterisk 12.7.1 on the server where I am experiencing the core dumps to see if it is happening as well with older releases. By: Leandro Dardini (ldardini) 2015-05-23 16:06:25.188-0500 I can confirm it is happening also on latest asterisk 12. If the asterisk server is quiet, nothing happens, but once there is some traffic, the core dumps. By: Leandro Dardini (ldardini) 2015-05-23 17:34:36.029-0500 Three core dumps in a row By: Niklas Larsson (pnlarsson) 2017-03-13 09:11:12.741-0500 I have the same issue - doing a odbc thing after a lot of nothing for the asterisk makes it dump core (like rotating the logs). Solved it by running  a cron script doing {code} asterisk -rx 'odbc read ODBC_SQL_PING_CDR 3 exec' {code} every 10 minutes.