[Home]

Summary:ASTERISK-25740: Asterisk aborts after receiving a MySQL syntax error for adaptive_odbc CDR logging
Reporter:Chris Syria (csyria)Labels:
Date Opened:2016-02-02 12:46:11.000-0600Date Closed:2020-01-14 11:13:53.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:CDR/cdr_adaptive_odbc
Versions:13.6.0 Frequency of
Occurrence		Occasional
Related
Issues:
Environment:Asterisk is running on CentOS 7.2 64-bit The database server is running on another CentOS 7.2 box. MariaDB Server version: 5.5.44-MariaDBAttachments:( 0) backtrace_201602021206.txt
( 1) etc-asterisk-cdr_adaptive_odbc.conf.txt
( 2) etc-asterisk-cdr.conf.txt
( 3) etc-asterisk-res_odbc.conf.txt
( 4) etc-odbc.ini.txt
( 5) etc-odbcinst.ini.txt
( 6) messages-20160207.txt
( 7) Structure_of_CDR_MySQL_table.sql.txt
Description:On my 13.6.0 production box, we are seeing the Asterisk process crash and get restarted by safe_asterisk every few days. These crashes occur only at the end of calls, and appear to be related to adaptive_odbc logging for CDR.

In my /var/log/messages:
{code} HOSTNAME asterisk: /usr/sbin//safe_asterisk: line 164:  8661 Aborted                 (core dumped) nice -n $PRIORITY "${ASTSBINDIR}/asterisk" -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
{code}

This is the last log I see before it goes down:
{code}
WARNING[9159] res_odbc.c: SetConnectAttr (Txn isolation) returned an error: HY000: [MySQL][ODBC 5.2(w) Driver]You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '7' at line 1
WARNING[9159] res_odbc.c: SQL Execute returned an error -1: 08S01: [MySQL][ODBC 5.2(w) Driver][mysqld-5.5.44-MariaDB]MySQL server has gone away (76)
WARNING[9159] res_odbc.c: SQL Execute error -1! Verifying connection to asterisk-connector [asterisk-connector]...
WARNING[9159] res_odbc.c: Connection is down attempting to reconnect...
{code}

I suspect this is related to the ODBC logging because I see this in the backtrace:
{code}
#0  0x00007fa724d665f7 in raise () from /lib64/libc.so.6
#1  0x00007fa724d67ce8 in abort () from /lib64/libc.so.6
#2  0x00007fa724d5f566 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007fa724d5f612 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fa6e26ff43a in desc_free_paramdata () from /usr/lib64/libmyodbc5.so
#5  0x00007fa6e2704292 in my_SQLFreeStmtExtended () from /usr/lib64/libmyodbc5.so
#6  0x00007fa6e26fc4e5 in SQLDisconnect () from /usr/lib64/libmyodbc5.so
#7  0x00007fa7129ad46a in SQLDisconnect () from /lib64/libodbc.so.2
#8  0x00007fa712c0ae8b in odbc_obj_disconnect (obj=0x1f678f8) at res_odbc.c:1489
#9  0x00007fa712c07ef1 in ast_odbc_sanity_check (obj=0x1f678f8) at res_odbc.c:766
#10 0x00007fa712c07a04 in ast_odbc_prepare_and_execute (obj=0x1f678f8, prepare_cb=0x7fa6cab80c12 <generic_prepare>, data=0x7fa6dc032c68) at res_odbc.c:670
#11 0x00007fa6cab83682 in odbc_log (cdr=0x7fa6dc0603f0) at cdr_adaptive_odbc.c:742
#12 0x000000000049e386 in post_cdr (cdr=0x7fa6dc0603f0) at cdr.c:3271
#13 0x000000000049fc92 in cdr_detach (cdr=0x7fa6dc0603f0) at cdr.c:3568
#14 0x0000000000497270 in cdr_object_dispatch (cdr=0x7fa6dc0525b8) at cdr.c:1199
#15 0x000000000049a808 in handle_channel_cache_message (data=0x0, sub=0x9aee28, message=0x7fa71803b418) at cdr.c:2129
#16 0x00000000005ccf6c in router_dispatch (data=0x9b0068, sub=0x9aee28, message=0x7fa71803b418) at stasis_message_router.c:201
#17 0x00000000005bc3c5 in subscription_invoke (sub=0x9aee28, message=0x7fa71803b418) at stasis.c:433
#18 0x00000000005bcf01 in dispatch_exec_async (local=0x7fa722d8ac90) at stasis.c:695
#19 0x00000000005d72f3 in ast_taskprocessor_execute (tps=0x9aed38) at taskprocessor.c:776
#20 0x00000000005d5b42 in default_tps_processing_function (data=0x9b03b8) at taskprocessor.c:184
#21 0x00000000005ec2e1 in dummy_start (data=0x9aeb90) at utils.c:1237
#22 0x00007fa725a82dc5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fa724e2721d in clone () from /lib64/libc.so.6
{code}

I can not see any consistent pattern in the calls that cause this syntax error, nor can I reproduce it on demand. It seems to happen about every 5 days on average.

I realize that 13.6.0 has been superseded, but I haven't had the chance to schedule an outage window to upgrade to 13.7.0. I did search on some relevant terms in the tracker and didn't see a similar bug being filed.

(Also... my apologies to the community if this bug is inappropriate - I tried to follow the steps listed on the issue guidelines, but this is my first submission here).
Comments:By: Asterisk Team (asteriskteam) 2016-02-02 12:46:12.724-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Chris Syria (csyria) 2016-02-02 12:48:40.276-0600

This is the backtrace from the most recent occurrence of this issue.
By: Rusty Newton (rnewton) 2016-02-04 17:54:58.161-0600

bq.  I tried to follow the steps listed on the issue guidelines

Thanks! That is actually pretty rare around here (for any bug tracker really). We absolutely appreciate it.

Looks like you have provided almost everything we could ask for.

However a couple more things:

* Provide a debug log captured right before the crash. (a few minutes of logging or the last 10000 lines is fine) Follow these instructions: https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information
* Provide all of your CDR and ODBC configuration (don't forget to sanitize)
* Attach everything as .txt so it is easily accessible from our browsers.

Thanks again!

By: Chris Syria (csyria) 2016-02-10 16:29:04.682-0600

Uploaded requested configs.
Adaptive ODBC is the only backend being actively used:
{code}
Call Detail Record (CDR) settings
----------------------------------
 Logging:                    Enabled
 Mode:                       Simple
 Log unanswered calls:       Yes
 Log congestion:             No

* Registered Backends
 -------------------
   Adaptive ODBC
   cdr_manager (suspended)
   cdr-custom
{code}
By: Chris Syria (csyria) 2016-02-10 17:10:07.074-0600

Uploaded logs - Crash happens right around 10k lines in.

I hope this has this info needed. My logger.conf has this line from before it started happening:
{code}messages => notice,warning,error,debug,verbose{code}

but I didn't see any debug messages in the logged file.

If I need to enable more debugging, I'll provide that output as soon as another crash happens.

Additionally, I tried to sanitize this fairly heavily. If anything was lost in this process, I can provide more logs.
By: Rusty Newton (rnewton) 2016-02-16 19:47:00.849-0600

[~csyria] - Hmm, yeah it looks like you may have forgot to turn up the debug channel level.

Remember to follow the instructions [on the Collection Debug page|https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information] in regards to "core set debug 5" and "core set verbose 5".

You should immediately see "DEBUG" messages being written to the log file if your logger.conf configuration is correct.

Please attach a new set of debug along with the new crash trace the next time it happens. Thanks!
By: Asterisk Team (asteriskteam) 2016-03-02 12:00:01.701-0600

Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1].

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines