Summary:	ASTERISK-25833: Asterisk 13.7.2 crashes with error about libmysqlclient.so.18.0.0
Reporter:	Issam RACHDI (issam)	Labels:
Date Opened:	2016-03-04 05:20:21.000-0600	Date Closed:	2016-03-29 11:07:04
Priority:	Critical	Regression?
Status:	Closed/Complete	Components:
Versions:	13.7.2	Frequency of Occurrence	Occasional
Related Issues:
Environment:	Debian 8.3, kernel 3.16.0-4-amd64, pjproject 2.4.5	Attachments:	( 0) backtrace.txt
Description:	Hi, I have installed asterisk 13.7.2 yesterday, and so far it crashed 3 times. Each time, the logs in /var/log/asterisk just stops, and i find these errors in the syslog first crash {noformat} Mar  3 10:49:02 asterisk kernel: [47923.065444] asterisk[17500]: segfault at d ip 00007f6006bb7120 sp 00007f5f3e58ed38 error 6 in libmysqlclient.so.18.0.0[7f6006b67000+2b7000] {noformat} second crash {noformat} Mar  3 17:19:40 asterisk kernel: [71359.942310] asterisk[1524]: segfault at d ip 00007fed777bf120 sp 00007fecbf170cc8 error 6 in libmysqlclient.so.18.0.0[7fed7776f000+2b7000] {noformat} I then recompiled asterisk with the flag DONT_OPTIMIZE, i unchecked some modules that i didn't use, and re-installed asterisk. This morning, asterisk crashed again, with this message in syslog: {noformat} Mar  4 11:19:53 asterisk kernel: [50378.462268] traps: asterisk[21644] general protection ip:7f86b8134120 sp:7f86037406a8 error:0 in libmysqlclient.so.18.0.0[7f86b80e4000+2b7000] {noformat} I tried to produce core files by setting the option dumpcore = yes in asterisk.conf, but i don't know where to look for it, or if it is even generated. Note that i use func_odbc with mysql a lot in my dialplan, but i never had this issue with asterisk 11.
Comments:	By: Asterisk Team (asteriskteam) 2016-03-04 05:20:22.452-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Michael L. Young (elguero) 2016-03-04 07:38:05.501-0600 Maybe this page would help you to get the backtrace: https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace By: Issam RACHDI (issam) 2016-03-04 08:09:07.164-0600 Thanks for your response. I saw the link, but it says to start asterisk with the -g option, but in my case asterisk is started with an init script generated with "make config". I found these lines in the init script, {code:title=/etc/init.d/asterisk|borderStyle=solid}        if [ "x$COREDUMP" = "xyes" ]; then                ASTARGS="$ASTARGS -g"        fi {code} but the variable COREDUMP is not defined in the script. Should i define it and set it to the value "yes" ? By: Issam RACHDI (issam) 2016-03-04 08:38:34.815-0600 Asterisk crashed again, this time with a new message :) {noformat} [64667.308843] asterisk[22157]: segfault at 3c ip 00007f99c8887b1d sp 00007f9930ef4230 error 4 in libpjnath.so.2[7f99c88    73000+22000] {noformat} By: Michael L. Young (elguero) 2016-03-04 16:26:16.265-0600 You mentioned that you set {{dumpcore = yes}} in {{asterisk.conf}}.  I would look under the {{/tmp}} directory for a backtrace file. By: Issam RACHDI (issam) 2016-03-07 07:32:59.452-0600 I could finally get the core file. Apparently, when started from the init script, it doesn't generate the core file, at least in my case, even with the option dumpcore set to yes. I attached the {{backtrace.txt}} file. And this is the error that was in the syslog at the time of the crash {noformat} [193585.262431] asterisk[12788]: segfault at a ip 00007f2784b4f120 sp 00007f26cbf006e8 error 6 in libmysqlclient.so.18.0.0[7f2784aff000+2b7000] {noformat} By: Michael L. Young (elguero) 2016-03-07 13:47:13.559-0600 Did this only start in 13.7.2 or are you jumping to this version from 11? I found an email thread where someone was getting a segfault as well which looks very similar to what you have here.  http://lists.digium.com/pipermail/asterisk-users/2016-January/288228.html . By: Issam RACHDI (issam) 2016-03-07 14:38:48.550-0600 I jumped directly from 11 to 13.7.2. I just grabbed the last version and installed it. The difference between the old installation (asterisk 11) and the new (13) is that the 11 was installed in debian 7, and the 13 was installed in debian 8. I read the the thread, and it implies that the problem is with mysql, but the database server is in a different server that didn't change. Also i'm only connecting to mysql through odbc, and i don't use real time, so there shouldn't be a need to call mysql connector (i'm just guessing here). Is there a version known to not have this problem? Thank you for your time By: Michael L. Young (elguero) 2016-03-07 15:28:10.754-0600 Issam - The problem could be in the client library that is being used in debian 8 to communicate with the database server, not with the database server. The other thread talks about asterisk realtime but they are using ODBC in debian 8 to communicate with the database server just like you are using ODBC.  ODBC uses the client library libmysqlclient.so.18 in order to communicate with the MySQL server. {noformat} Asterisk <-> Unix ODBC <-> libmysqlclient.so.18 <-> MySQL Server {noformat} It is a shame that they never followed up on that email thread to confirm if that was the problem or not for them. By: Issam RACHDI (issam) 2016-03-08 01:17:59.272-0600 OK, thank you. I'll try to install another version of libmysqlclient and see what happens. By: Joshua C. Colp (jcolp) 2016-03-09 08:31:35.764-0600 Assigning this to you for feedback pending changing the MySQL client to see if it alters the result. By: Issam RACHDI (issam) 2016-03-16 09:59:35.884-0500 Ok, so first, there is something i didn't mention before, because i didn't think it was related. There are some users that connect to asterisk using WebRTC via wss. The main reason why we migrated to version 13 was a problem in v 11 with res_rtp and webrtc that caused asterisk to crash. So the first thing we did was to stop using WebRTC and use a normal softphone (while still using asterisk 13), and see what happens. And asterisk didn't crash for a week. Next, and since it was suggested here that the problem may be with mysql connector, i downloaded the last version of MySQL ODBC connector (5.3.4) for debian from [here|http://dev.mysql.com/downloads/connector/odbc/] , and i changed the driver used in {{/etc/odbcinst.ini}}. We tested yesterday and today by adding gradually WebRTC users, and today asterisk crashed again with this error {noformat} kernel: [976392.374990] asterisk[20721]: segfault at 3c ip 00007f7f519eab1d sp 00007f7e7a68ae60 error 4 in libpjnath.so.2[7f7f519d6000+22000] {noformat} I had this error once before. I don't have a backtrace now, i'll try to have one next time. I'm still waiting to see if the error about mysql will happen again. By: Issam RACHDI (issam) 2016-03-17 05:45:59.783-0500 So the error about libmysql didn't show up so far, but asterisk is still crashing with the error i mentioned in the comment above about libpjnath.so.2. So i'll just open a new issue for that error. By: Joshua C. Colp (jcolp) 2016-03-17 09:00:58.657-0500 The pjnath issue you reported is actually a duplicate of an existing one and was closed as a result. I'm leaving this in feedback since we're still waiting to see what happens with the updated MySQL client. By: Issam RACHDI (issam) 2016-03-29 06:05:32.624-0500 Asterisk didn't crash since i changed the odbc driver (about 2 weeks ago). Also the workaround for the other error (pjnath) seems to work fine. Thank you for your help. By: Joshua C. Colp (jcolp) 2016-03-29 11:07:04.727-0500 Since this appears to have been the MySQL connector and changing it has now resolved the issue I am closing this issue out.