[Home]

Summary:ASTERISK-23717: Segfault when loading XML documentation
Reporter:Hendrik Wedhorn (hwedhorn)Labels:
Date Opened:2014-05-05 09:14:59Date Closed:2014-06-12 13:52:31
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General Documentation
Versions:11.9.0 Frequency of
Occurrence		Constant
Related
Issues:
Environment:i586 - GLIBC_2.3.2Attachments:( 0) asterisk.conf
( 1) asterisk.inc
( 2) backtrace_new.txt
( 3) backtrace1190.txt
( 4) libxml-z_stream-misuse.diff
( 5) libxml-z_stream-misuse-gitfix.diff
( 6) valgrind.txt
Description:When loading the xml documentation the asterisk always dies with an segmentation fault. Although this should not influence each other, this only happens when the /etc/asterisk/asterisk.conf has an include to an another file.

- When the include in the asterisk.conf is removed the xml documentation is loaded perfectly fine. No segfault.
- When the xml documentation is remove the include in the asterisk.conf works perfectly fine. No segfault.

- The segfault always appears (100% reproducible.) when both is used.
Comments:By: Rusty Newton (rnewton) 2014-05-05 17:36:19.064-0500

I had Asterisk 12 handy and couldn't reproduce the crash after building Asterisk with a "make full" and loading it up.

I'll try in 11 when I get the chance. Can you post the relevant configs and included file, plus describe reproduction steps?

Thanks!
By: Hendrik Wedhorn (hwedhorn) 2014-05-06 02:53:19.829-0500

Hi, i will add the configs in a few moments.

Reproduction steps:

- Build with libxml2 2.6.16
- Start Asterisk 11.9.0 with /usr/sbin/asterisk -f -U www-data -G www-data -g -c
- Segfault
By: Hendrik Wedhorn (hwedhorn) 2014-05-06 02:56:41.516-0500

Start of Asterisk in foreground:

/usr/sbin/asterisk -vvvvvvvvvvvvvvvvvvvvvvvvvvvvvdddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
Parsing /etc/asterisk/asterisk.conf
Parsing /var/ast3/asterisk.inc
Seeding global EID '00:0d:48:38:03:91' from 'eth0' using 'siocgifhwaddr'
Privilege escalation protection disabled!
See https://wiki.asterisk.org/wiki/x/1gKfAQ for more details.
Asterisk 11.9.0, Copyright (C) 1999 - 2013 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Parsing /etc/asterisk/extconfig.conf
 == Parsing '/etc/asterisk/extconfig.conf': Found
 == Binding voicemail to pgsql/asterisk/voicemail_users
 == Binding queues to pgsql/asterisk/queue_table
 == Binding queue_members to pgsql/asterisk/queue_member_table
 == Binding queue_log to pgsql/asterisk/ast_queuecdr11
 == Binding meetme to pgsql/asterisk/meetme
 == Binding iaxpeers to pgsql/asterisk/iax_peers
 == Binding iaxusers to pgsql/asterisk/iax_users
Resetting translation matrix
Parsing /etc/asterisk/logger.conf
 == Parsing '/etc/asterisk/logger.conf': Found
Segmentation fault

By: Rusty Newton (rnewton) 2014-05-06 16:57:19.968-0500

I tested with Asterisk 11 SVN-branch-11-r413305 and the libxml2 package on CentOS:
{noformat}
Name        : libxml2
Arch        : i686
Version     : 2.7.6
Release     : 14.el6
{noformat}

I followed your instructions, used your configuration (except I didn't use a custom modules directory) and I couldn't get a crash.

Can you test with the SVN of Asterisk 11, and then test with the newer libxml2 ?

By: Hendrik Wedhorn (hwedhorn) 2014-05-13 02:22:32.523-0500

Ok, i will try a new version of libxml2 and give you a feedback in a couple of hours.
By: Hendrik Wedhorn (hwedhorn) 2014-05-13 09:27:48.127-0500

I still have the same problem with libxml2 2.7.6 :(
By: Walter Doekes (wdoekes) 2014-05-13 10:14:32.386-0500

Please press the Send-back button.
By: Rusty Newton (rnewton) 2014-05-20 09:32:28.540-0500

[~hwedhorn]

Can you get another backtrace making sure you have compiled with BETTER_BACKTRACES, DONT_OPTIMIZE and have installed whatever package provides the debug symbols for libxml2?
By: Walter Doekes (wdoekes) 2014-05-20 09:52:53.698-0500

@Rusty: I think he already got most of that. Just not the libxml2 recompiled in (unoptimized) debug mode.

@Hendrik: can you start asterisk from valgrind? Unrelated issues generally point to memory being overwritten.
By: Addix Internet Services GmbH (addix) 2014-05-21 04:02:37.537-0500

Im getting the following with valgrind:

==9380== Memcheck, a memory error detector.
==9380== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==9380== Using LibVEX rev 1854, a library for dynamic binary translation.
==9380== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==9380== Using valgrind-3.3.1-Debian, a dynamic binary instrumentation framework.
==9380== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==9380== For more details, rerun with: -v
==9380==
==9380== Conditional jump or move depends on uninitialised value(s)
==9380==    at 0x41BE82B: __xmlParserInputBufferCreateFilename (xmlIO.c:2521)
==9380==    by 0x41BE8E9: xmlParserInputBufferCreateFilename (xmlIO.c:2559)
==9380==    by 0x4191C10: xmlNewInputFromFile (parserInternals.c:1549)
==9380==    by 0x41BED12: xmlLoadExternalEntity (xmlIO.c:3941)
==9380==    by 0x41961D0: xmlCreateURLParserCtxt (parser.c:13357)
==9380==    by 0x41ACF80: xmlReadFile (parser.c:14510)
==9380==    by 0x81CC136: ast_xml_open (xml.c:63)
==9380==    by 0x81D1A5E: ast_xmldoc_load_documentation (xmldoc.c:2442)
==9380==    by 0x808FA89: main (asterisk.c:4227)
==9380==
==9380== Thread 19:
==9380== Conditional jump or move depends on uninitialised value(s)
==9380==    at 0x443289B: BN_cmp (in /usr/lib/libcrypto.so.1.0.0)
==9380==
==9380== Conditional jump or move depends on uninitialised value(s)
==9380==    at 0x443289D: BN_cmp (in /usr/lib/libcrypto.so.1.0.0)
==9380==
==9380== Conditional jump or move depends on uninitialised value(s)
==9380==    at 0x44327F2: BN_ucmp (in /usr/lib/libcrypto.so.1.0.0)
==9380==
==9380== Conditional jump or move depends on uninitialised value(s)
==9380==    at 0x4439ACD: BN_mod_inverse (in /usr/lib/libcrypto.so.1.0.0)
.
.
.
.
==9380==
==9380== Use of uninitialised value of size 4
==9380==    at 0x443D4AB: bn_mul_mont (in /usr/lib/libcrypto.so.1.0.0)
==9380==    by 0x390E5D25: ???
==9380==
==9380== Use of uninitialised value of size 4
==9380==    at 0x443D4A0: bn_mul_mont (in /usr/lib/libcrypto.so.1.0.0)
==9380==    by 0xB6E18164: ???
==9380==
==9380== More than 1000 different errors detected.  I'm not reporting any more.
==9380== Final error counts will be inaccurate.  Go fix your program!
==9380== Rerun with --error-limit=no to disable this cutoff.  Note
==9380== that errors may occur in your program without prior warning from
==9380== Valgrind, because errors are no longer being displayed.
==9380==


By: Walter Doekes (wdoekes) 2014-05-22 07:21:49.316-0500

Okay. So that valgrind output doesn't tell me much. Most likely because of all the BN_mod_inverse crap before.

You didn't rerun with --error-limit=no (or with exclude filters on libcrypto).

A couple of suggestions to try:
- rerun valgrind while attempting to get more info out of it
- try a more recent libxml2. the oldest version I run is 2.8.0, the newest is 2.9.1. yours are older
- if you can, remove the libxml2 package and compile it from source while making sure that optimization is disabled and debugging symbols are enabled
{{libxml2$ CFLAGS=-O0\ -g ./configure --with-debug}}
- try what happens if you remove BUILD_NATIVE from the asterisk build options (a 586 system is rather rare nowadays)
- a combination of the above
By: Addix Internet Services GmbH (addix) 2014-05-26 03:39:51.183-0500

Okay, i will do this today.
By: Addix Internet Services GmbH (addix) 2014-05-27 08:18:43.742-0500

>rerun valgrind while attempting to get more info out of it
There is no segmentation fault when starting asterisk with valgrind. The output to valgrind.txt is basically the same, just more errors of libcrypto
>try a more recent libxml2. the oldest version I run is 2.8.0, the newest is 2.9.1. yours are older
i will try using a newer version, but this is not easy in my enviroment. Rusty Newton was also using 2.7.6 with no issue.
>if you can, remove the libxml2 package and compile it from source while making sure that optimization is disabled and debugging symbols are enabled
l>ibxml2$ CFLAGS=-O0\ -g ./configure --with-debug
iam already building it from the source of libxml 2.7.6. Using these parameters does not change anything.
>try what happens if you remove BUILD_NATIVE from the asterisk build options (a 586 system is rather rare nowadays)
>a combination of the above
Im using an AMD Geode CPU but BUILD_NATIVE or any other optimization is not enabled.

By: Walter Doekes (wdoekes) 2014-05-27 08:23:06.506-0500

{quote}
>if you can, remove the libxml2 package and compile it from source while making sure that optimization is disabled and debugging symbols are enabled
l>ibxml2$ CFLAGS=-O0\ -g ./configure --with-debug
iam already building it from the source of libxml 2.7.6. Using these parameters does not change anything.
{quote}

The backtraces should include more info in the BT when in the libxml code.

This https://issues.asterisk.org/jira/secure/attachment/50048/backtrace1190.txt only shows function names, but no locals or arguments.
By: Addix Internet Services GmbH (addix) 2014-06-03 04:00:56.498-0500

So i got a new backtrace with libxml2 symbols
By: Walter Doekes (wdoekes) 2014-06-03 04:53:15.766-0500

Very well.

Looks like a bug in libxml-2.7.6, which is probably fixed in 2.7.7.

Could you try both of the patches that I attach (not simultaneously).

{noformat}
$ cd libxml2-2.7.6/
$ patch -p1 < the_patch.diff
{noformat}
By: Addix Internet Services GmbH (addix) 2014-06-03 08:16:13.447-0500

The patch libxml-z_stream-misuse.diff ist not working with 2.7.6:

libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I. -I. -I./include -I./include -D_REENTRANT -g -O0 -g -O0 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Winline -Wredundant-decls -Wp,-MD,.deps/xmlIO.pp -c xmlIO.c  -fPIC -DPIC -o .libs/xmlIO.o
cc1: warning: -Wuninitialized is not supported without -O
xmlIO.c: In function '__xmlParserInputBufferCreateFilename':
xmlIO.c:2524: warning: dereferencing 'void *' pointer
xmlIO.c:2524: error: request for member 'have' in something not a structure or union
xmlIO.c:2526: warning: dereferencing 'void *' pointer
xmlIO.c:2526: error: request for member 'next' in something not a structure or union
make[2]: *** [xmlIO.lo] Error 1
make[2]: Leaving directory `/mnt/sdb1/fec-astimax/libxml2-2.7.6'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/mnt/sdb1/fec-astimax/libxml2-2.7.6'
make: *** [all-recursive-am] Error 2


At the moment i cannot test 2.7.7 because of autoconf dependency....
By: Walter Doekes (wdoekes) 2014-06-03 08:26:26.566-0500

And what about the other (gitfix) patch?

And what is the value of ZLIB_VERNUM?

{noformat}
$ grep ZLIB_VERNUM /usr/include/zlib.h
#define ZLIB_VERNUM 0x1280
{noformat}
By: Addix Internet Services GmbH (addix) 2014-06-04 03:50:11.048-0500

Thanks a lot, its working now! In Detail:

The libxml-z_stream-misuse.diff is now compiling with an newer zlib, but still gets me an segfault.

The Patch  libxml-z_stream-misuse-gitfix.diff works fine. No segfault any more.
By: Walter Doekes (wdoekes) 2014-06-11 01:59:56.158-0500

[~rnewton]: so it appears libxml 2.7.6 is broken (in combination with certain zlib versions).

What do you want me to do? I'd say "Close" with a "Not my problem" resolution.
By: Addix Internet Services GmbH (addix) 2014-06-11 02:01:59.417-0500

Yep, close it, Thanks.
By: Rusty Newton (rnewton) 2014-06-12 13:52:22.545-0500

[~wdoekes] Thanks. I wasn't following closely on this one. Wasn't sure what was going on.

Close this out!