[Home]

Summary:ASTERISK-25168: Random Core Dumps on Asterisk 13.4 PJSIP, in ast_channel_name at channel_internal_api.c
Reporter:Carl Fortin (phonefxg)Labels:
Date Opened:2015-06-17 06:36:45Date Closed:2015-07-27 05:50:29
Priority:MajorRegression?
Status:Closed/CompleteComponents:Resources/res_pjsip
Versions:13.4.0 Frequency of
Occurrence		Frequent
Related
Issues:
Environment:Our software: Asterisk Realtime 13.4 PJSIP Driver mysql Ver 14.14 pjproject 2.3 spandsp 0.0.6 jansson 2.7 CentOS 6.6 64 bits on Vmware Number of endpoints : > 700 Numbers of calls : 3000/day Our Hardware: Phones : Cisco SPA514G FW: 7.5.7 ATA : Audiocodes MP124 T1 : Mediatrix 3532 ISDN to SIP gateway CPU : Quadcore Intel(R) Xeon(R) CPU E5-2650 RAM : 3 GBAttachments:( 0) backtrace_1108.txt
( 1) backtrace_20_11.txt
( 2) backtrace-1_1741.txt
( 3) backtrace-2_1741_24.txt
( 4) backtrace-3_1741_44.txt
( 5) backtrace-4_1742.txt
( 6) backtrace-5_1743_19.txt
( 7) backtrace5.txt
( 8) backtrace-6_1744_56.txt
( 9) backtrace-7_1745_08.txt
(10) Core_show_summary.txt
(11) jira_asterisk_25168_v13_test.patch
(12) jira_asterisk_25168_v13_test2.patch
(13) jira_asterisk_25168_v13.4.0_test.patch
(14) jira_asterisk_25168_v13.4.0_test2.patch
(15) Last_backtrace.txt
(16) mmlog
(17) myDebugLog_asterisk.zip
(18) myDebugLog2
(19) Putty_console_before_crash.txt
(20) valgrind_patch.txt
(21) valgrind.txt
Description:We are running Asterisk 13.4 PJSIP in a production environment and get random crashes. Everything seemed to be working fine for a couple of days and we now get crashes more and more.
I have also noticed that crashes happens even though no calls are made, so I don't think it has anything to do with the dialplan. I've really tested everything in a test machine before moving my dialplan to PJSIP. Most of the phones use direct media, so the load on asterisk is low.
I get no errors when doing asterisk -cvvvvvvvvvvvvvv.

[Edit by Rusty - For reference here I'm extracting from your debug log the last lines before the crash.]

{noformat}
[Jun 17 11:08:19] DEBUG[23136][C-0000020e] res_rtp_asterisk.c: Got RTCP report of 76 bytes
[Jun 17 11:08:19] DEBUG[23136][C-0000020e] acl.c: Not an IPv4 nor IPv6 address, cannot get port.
[Jun 17 11:08:19] DEBUG[23136][C-0000020e] netsock2.c: Splitting 'dti-asterisk.cegep-fxg.qc.ca' into...
[Jun 17 11:08:19] DEBUG[23136][C-0000020e] netsock2.c: ...host 'dti-asterisk.cegep-fxg.qc.ca' and port ''.
[Jun 17 11:08:19] DEBUG[23136][C-0000020e] acl.c: Not an IPv4 nor IPv6 address, cannot get port.
[Jun 17 11:08:19] DEBUG[23136][C-0000020e] acl.c: Attached to given IP address
[Jun 17 11:08:19] DEBUG[11851] netsock2.c: Splitting '206.80.250.101:40631' into...
[Jun 17 11:08:19] DEBUG[11851] netsock2.c: ...host '206.80.250.101' and port '40631'.
[Jun 17 11:08:19] DEBUG[11851] netsock2.c: Splitting '206.167.100.36:0' into...
[Jun 17 11:08:19] DEBUG[11851] netsock2.c: ...host '206.167.100.36' and port '0'.
[Jun 17 11:08:28] Asterisk 13.4.0 built by root @ dti-asterisk.cegep-fxg.qc.ca on a x86_64 running Linux on 2015-06-16 23:48:22 UTC
{noformat}

Excerpt from the trace

{noformat}Core was generated by `/usr/sbin/asterisk -f -vvvg -c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004c9c81 in ast_channel_name (chan=0x0) at channel_internal_api.c:476
476 DEFINE_STRINGFIELD_GETTER_FOR(name);
#0  0x00000000004c9c81 in ast_channel_name (chan=0x0) at channel_internal_api.c:476
No locals.
#1  0x00007f4b13724ffa in t38_automatic_reject (obj=0x7f4b441c50e8) at res_pjsip_t38.c:201
       session = 0x7f4b441c50e8
       datastore = 0x7f4b7c0fcc98
       session_media = 0x7f4b440d48b8
       __PRETTY_FUNCTION__ = "t38_automatic_reject"
#2  0x00000000005dc343 in ast_taskprocessor_execute (tps=0x7f4b440a7fe8) at taskprocessor.c:769
       local = {local_data = 0x2fc00649ca0, data = 0x649ca0}
       t = 0x7f4b20003000
       size = 0
       __PRETTY_FUNCTION__ = "ast_taskprocessor_execute"
#3  0x00000000005e481c in execute_tasks (data=0x7f4b440a7fe8) at threadpool.c:1157
       tps = 0x7f4b440a7fe8
#4  0x00000000005dc343 in ast_taskprocessor_execute (tps=0x2fad768) at taskprocessor.c:769
       local = {local_data = 0x7f4b01d07c90, data = 0x0}
       t = 0x7f4b20002fd0
       size = 0
       __PRETTY_FUNCTION__ = "ast_taskprocessor_execute"
{noformat}
Comments:By: Carl Fortin (phonefxg) 2015-06-17 06:37:39.166-0500

Backtrace from crash
By: Carl Fortin (phonefxg) 2015-06-17 06:46:41.036-0500

Backtrace from last core dump
By: Rusty Newton (rnewton) 2015-06-17 09:13:19.892-0500

Thank you for the crash report. However, we need more information to investigate the crash. Please provide:

1. A backtrace generated from a core dump using the instructions provided on the Asterisk wiki [1].
2. Specific steps taken that lead to the crash.
3. All configuration information necesary to reproduce the crash.

Thanks!

[1]: https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace


By: Rusty Newton (rnewton) 2015-06-17 09:14:13.653-0500

Specifically make sure to recompile using the compiler flags specified in the instructions. Then attach the new traces. I'll also post a link to instructions for gathering a debug log along with your trace.
By: Rusty Newton (rnewton) 2015-06-17 09:14:26.877-0500

We require additional debug to continue with triage of your issue. Please follow the instructions on the wiki [1] for how to collect debugging information from Asterisk. For expediency, where possible, attach the debug with a '.txt' file extension so that the debug will be usable for further analysis.

Thanks!

[1] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information


By: Rusty Newton (rnewton) 2015-06-17 09:14:51.775-0500

When you have the requested information please use the 'Send Back' or 'Enter Feedback' buttons to send the issue back to us.
By: Carl Fortin (phonefxg) 2015-06-17 12:17:57.210-0500

My full debug log when asterisk crashes at 11:08 AM.
By: Carl Fortin (phonefxg) 2015-06-17 12:18:25.504-0500

Backtraces at 11:08 AM
By: Carl Fortin (phonefxg) 2015-06-17 12:19:06.518-0500

Disconnected from the console at 11:08 AM.
By: Carl Fortin (phonefxg) 2015-06-17 12:29:07.999-0500

Here's the backtraces from our last core dump as well as a full debug log.
Look at line 135469 from the file myDebugLog_asterisk.

I have created the backtraces with DONT_OPTIMIZE and BETTER_BACKTRACES.

I have used the following command:

gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c /tmp/core.dti-asterisk.cegep-fxg.qc.ca-2015-06-17T11:08:24-0400  > /tmp/backtrace_1108.txt

As for the specific steps, it is really hard to know because it even happened during the night when there is no activities or calls on the server.It's really a ramdom thing,Asterisk restarted at 8:41 AM and after about a 1000 calls crashed again at 11:08 AM.

I will provide anything you need to help identify the cause.

Let me know if my backtraces are ok.



By: Carl Fortin (phonefxg) 2015-06-18 06:24:33.222-0500

I have disabled ACL after last crash which was yesterday at 11: 08 AM.

Take a look at this line from the debug log just before Asterisk crashed:

[Jun 17 11:08:19] DEBUG[23136][C-0000020e] acl.c: Not an IPv4 nor IPv6 address, cannot get port.


By: Carl Fortin (phonefxg) 2015-06-18 07:35:34.447-0500

It has nothing to do with acl since it crashed again at 8:30 AM.

By: Carl Fortin (phonefxg) 2015-06-18 11:42:00.927-0500

CPU is also getting very high when it happens and ram will go down as well.
By: Carl Fortin (phonefxg) 2015-06-18 15:36:37.461-0500

I see the bug seems to be linked to something else.
Do you have any idea what is causing this? If it's a module that I don't use I could disable it while the issue is taken care of.
I will have to go back to 13.3 and hope for the best if I cannot stop asterisk from crashing.
I'll be available if you need more testing.
By: Carl Fortin (phonefxg) 2015-06-20 12:34:28.484-0500

I have collected more information about the problem:

I have installed Asterisk version 13.3.2 with the exact same settings and it's a no go.
It won't even load. It's almost crashing right away as more phones try to register.
Same thing with the certified version 13.1-cert2.

I have a VM replica of our production machine with less than 10 endpoints and it's working great without any crashes.
The crashes seems to point to the number of contacts. Asterisk crashed more frequently as we kept adding new endpoints.
It crashed again in a Saturday morning when absolutely no ones is using their phones.(I have logs that proves it)

Since, I cannot leave over 700 users without a phone, I have switched back to Chansip temporarily while the issue is being investigated.

It'll be easy to go back to PJsip if you need me to. It just a matter of replacing the .conf files.
We keep writing in ps_endpoints and sippeers tables, so it's more safe in case of bug such as this one.

I hope it will help you identify the cause of the crashes.
By: Richard Mudgett (rmudgett) 2015-06-26 11:00:36.066-0500

Of the three backtrace files you have posted, you have two different crashes.

[^backtrace_1108.txt] - This crash is a result of the channel hanging up in the middle of the peer trying to do a T.38 reINVITE.  I have enough information to see what is going on for this crash.

I need more information for these two crashes:
[^backtrace5.txt] and [^Last_backtrace.txt] - These are the same kind of crash.  The system is trying to send a NOTIFY message but there is memory corruption because the crash happens while trying to do a malloc().  You probably will want to try MALLOC_DEBUG first as that does not bog the machine down too much while providing fairly good memory corruption debug information.  MALLOC_DEBUG also allows you to see where memory is being allocated in case of memory leaks.

These pages talk some about memory corruption crashes and how to collect debug information:
https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace
https://wiki.asterisk.org/wiki/display/AST/Valgrind
https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag
By: Carl Fortin (phonefxg) 2015-06-26 11:14:57.693-0500

So you will need another backtrace from me with MALLOC_DEBUG ?

What kind of notify? Is it a unsolicited MWI?

By: Richard Mudgett (rmudgett) 2015-06-26 11:32:54.238-0500

{quote}
So you will need another backtrace from me with MALLOC_DEBUG ?
{quote}
Yes.  I'll also need mmlog if it contains anything other than the "New session" message.

{quote}
What kind of notify? Is it a unsolicited MWI?
{quote}
The outgoing NOTIFY is in response to receiving a SUBSCRIBE request.
By: Carl Fortin (phonefxg) 2015-06-26 11:51:59.617-0500

Ok let me put the system back to pjsip and wait for a crash, It should not be too long.

By: Carl Fortin (phonefxg) 2015-06-26 17:05:45.471-0500

Well, it was fast this time. Asterisk has been up only a few minutes before crashing. Unfortunately, the mmlog only have new session in it.
I have 7 backtraces  as well as my full debug log.



By: Carl Fortin (phonefxg) 2015-06-26 17:08:24.531-0500

Backtraces from core dumps in chronological order with malloc.
mmlog and full debug log.
By: Carl Fortin (phonefxg) 2015-06-26 17:59:53.928-0500

Here's the file created by valgrind. I hope it can help troubleshooting this bug.
By: Richard Mudgett (rmudgett) 2015-06-26 18:31:11.097-0500

One of the nice things about MALLOC_DEBUG is that if something tries to use a pointer out of a destroyed object that pointer points to {{0xdeaddeaddeaddead}} on a 64 bit system and is just about guaranteed to be invalid memory.  That invalid pointer use causes an earlier crash in a useful location from a code debug standpoint.  All those new crashes are showing the same thing.  Trying to use a pointer from a destroyed datastore data object dealing with MWI subscriptions.

I can see something in the code and need to figure out how to fix it.  I should have a patch to try next week.
By: Carl Fortin (phonefxg) 2015-06-26 18:44:35.377-0500

Thank you Richard, I'll be ready to try the patch once it's ready.


By: Richard Mudgett (rmudgett) 2015-06-30 14:08:42.785-0500

[^jira_asterisk_25168_v13_test.patch] - This should fix both crashes.  I have placed some {{BUGBUG}} messages in a couple locations dealing with FAX.  Let me know which ones you see if you see any in the logs.

# The NULL channel crash.  Let me know if you see this message in the log:
{{BUGBUG T.38 auto reject timer expired without a session channel.}}
If it doesn't crash then that is proof positive that the NULL channel crash is fixed.  This crash happened when the caller called a FAX number and then hung up immediately after the call was answered.
# The MWI subscription crash.  I needed to give a datastore on the subscription a reference to the MWI subscription object to prevent it going away.  What makes this tricky is that it creates a circular object reference chain.  The circular reference can never go away without explicit code to break the loop such as when setting up the subscription fails or the subscription shuts down.
By: Carl Fortin (phonefxg) 2015-06-30 14:26:26.850-0500

I will apply the patch tonight and report back.
By: Carl Fortin (phonefxg) 2015-06-30 17:15:09.128-0500

I get one error when applying the patch:

[root@dti-asterisk asterisk-13.4.0-Patch]# patch -p1 -u  < jira_asterisk_25168_v13_test.patch
patching file res/res_pjsip_mwi.c
Hunk #2 FAILED at 499.
Hunk #3 succeeded at 625 (offset -4 lines).
Hunk #4 succeeded at 719 (offset -4 lines).
1 out of 4 hunks FAILED -- saving rejects to file res/res_pjsip_mwi.c.rej
patching file res/res_pjsip_session.c
patching file res/res_pjsip_t38.c

Am I doing something wrong?

By: Richard Mudgett (rmudgett) 2015-06-30 18:06:31.225-0500

The patch was for the v13 branch not v13.4.0.  I'll make one for v13.4.0.

By: Carl Fortin (phonefxg) 2015-06-30 18:07:55.318-0500

All right.
By: Richard Mudgett (rmudgett) 2015-06-30 18:44:58.164-0500

[^jira_asterisk_25168_v13.4.0_test.patch] - Is the v13.4.0 version of [^jira_asterisk_25168_v13_test.patch]
By: Carl Fortin (phonefxg) 2015-06-30 19:21:20.748-0500

I have applied the patch successfully but Asterisk crashed again with memory corruption  7 minutes after starting.
{noformat}
erisk: malloc(): memory corruption: 0x00007f5668053580 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3316c75e66]
/lib64/libc.so.6[0x3316c79904]
/lib64/libc.so.6(__libc_malloc+0x71)[0x3316c7a6b1]
/usr/lib64/libpj.so.2(+0x104c2)[0x7f56b93184c2]
/usr/lib64/libpj.so.2(pj_pool_allocate_find+0x76)[0x7f56b931dd76]
/usr/lib64/libpjsip.so.2(+0x10abe)[0x7f56bae71abe]
/usr/lib64/libpjsip-simple.so.2(pjsip_evsub_notify+0x1b5)[0x7f56bb0ad9d5]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0xa8b1)[0x7f56bb9f28b1]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0xae25)[0x7f56bb9f2e25]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0xb1e8)[0x7f56bb9f31e8]
/usr/lib64/libpjsip.so.2(pjsip_endpt_process_rx_data+0x123)[0x7f56bae7a623]
/usr/lib64/asterisk/modules/res_pjsip.so(+0x1ff11)[0x7f5684f2ef11]
/usr/sbin/asterisk(ast_taskprocessor_execute+0x10d)[0x5dfd52]
/usr/sbin/asterisk[0x5e6c91]
/usr/sbin/asterisk[0x5e817f]
/usr/sbin/asterisk[0x5e7f2f]
/usr/sbin/asterisk[0x5f3b66]
/lib64/libpthread.so.0[0x33170079d1]
/lib64/libc.so.6(clone+0x6d)[0x3316ce88fd]
======= Memory map: ========
00400000-00696000 r-xp 00000000 fd:00 530520                             /usr/sbin/asterisk
00896000-008a9000 rw-p 00296000 fd:00 530520                             /usr/sbin/asterisk
008a9000-008cf000 rw-p 00000000 00:00 0
0262d000-04b3f000 rw-p 00000000 00:00 0                                  [heap]
3316800000-3316820000 r-xp 00000000 fd:00 262196                         /lib64/ld-2.12.so
3316a1f000-3316a20000 r--p 0001f000 fd:00 262196                         /lib64/ld-2.12.so
3316a20000-3316a21000 rw-p 00020000 fd:00 262196                         /lib64/ld-2.12.so
3316a21000-3316a22000 rw-p 00000000 00:00 0
3316c00000-3316d8a000 r-xp 00000000 fd:00 262240                         /lib64/libc-2.12.so
3316d8a000-3316f8a000 ---p 0018a000 fd:00 262240                         /lib64/libc-2.12.so
3316f8a000-3316f8e000 r--p 0018a000 fd:00 262240                         /lib64/libc-2.12.so
3316f8e000-3316f8f000 rw-p 0018e000 fd:00 262240                         /lib64/libc-2.12.so
3316f8f000-3316f94000 rw-p 00000000 00:00 0
3317000000-3317017000 r-xp 00000000 fd:00 262262                         /lib64/libpthread-2.12.so
3317017000-3317217000 ---p 00017000 fd:00 262262                         /lib64/libpthread-2.12.so
3317217000-3317218000 r--p 00017000 fd:00 262262                         /lib64/libpthread-2.12.so
3317218000-3317219000 rw-p 00018000 fd:00 262262                         /lib64/libpthread-2.12.so
3317219000-331721d000 rw-p 00000000 00:00 0
3317400000-3317402000 r-xp 00000000 fd:00 262305                         /lib64/libdl-2.12.so
3317402000-3317602000 ---p 00002000 fd:00 262305                         /lib64/libdl-2.12.so
3317602000-3317603000 r--p 00002000 fd:00 262305                         /lib64/libdl-2.12.so
3317603000-3317604000 rw-p 00003000 fd:00 262305                         /lib64/libdl-2.12.so
3317800000-3317822000 r-xp 00000000 fd:00 262610                         /lib64/libncurses.so.5.7
3317822000-3317a21000 ---p 00022000 fd:00 262610                         /lib64/libncurses.so.5.7
3317a21000-3317a22000 rw-p 00021000 fd:00 262610                         /lib64/libncurses.so.5.7
3317c00000-3317c83000 r-xp 00000000 fd:00 262186                         /lib64/libm-2.12.so
3317c83000-3317e82000 ---p 00083000 fd:00 262186                         /lib64/libm-2.12.so
3317e82000-3317e83000 r--p 00082000 fd:00 262186                         /lib64/libm-2.12.so
3317e83000-3317e84000 rw-p 00083000 fd:00 262186                         /lib64/libm-2.12.so
3318000000-3318015000 r-xp 00000000 fd:00 262336                         /lib64/libz.so.1.2.3
3318015000-3318214000 ---p 00015000 fd:00 262336                         /lib64/libz.so.1.2.3
3318214000-3318215000 r--p 00014000 fd:00 262336                         /lib64/libz.so.1.2.3
3318215000-3318216000 rw-p 00015000 fd:00 262336                         /lib64/libz.so.1.2.3
3318400000-331841d000 r-xp 00000000 fd:00 262319                         /lib64/libselinux.so.1
331841d000-331861c000 ---p 0001d000 fd:00 262319                         /lib64/libselinux.so.1
331861c000-331861d000 r--p 0001c000 fd:00 262319                         /lib64/libselinux.so.1
331861d000-331861e000 rw-p 0001d000 fd:00 262319                         /lib64/libselinux.so.1
331861e000-331861f000 rw-p 00000000 00:00 0
3318800000-331888c000 r-xp 00000000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
331888c000-3318a8b000 ---p 0008c000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
3318a8b000-3318a8e000 rw-p 0008b000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
3318a8e000-3318a8f000 rw-p 00000000 00:00 0
3318c00000-3318c16000 r-xp 00000000 fd:00 262309                         /lib64/libresolv-2.12.so
3318c16000-3318e16000 ---p 00016000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e16000-3318e17000 r--p 00016000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e17000-3318e18000 rw-p 00017000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e18000-3318e1a000 rw-p 00000000 00:00 0
3319000000-3319002000 r-xp 00000000 fd:00 262338                         /lib64/libfreebl3.so
3319002000-3319201000 ---p 00002000 fd:00 262338                         /lib64/libfreebl3.so
3319201000-3319202000 r--p 00001000 fd:00 262338                         /lib64/libfreebl3.so
3319202000-3319203000 rw-p 00002000 fd:00 262338                         /lib64/libfreebl3.so
3319400000-3319407000 r-xp 00000000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319407000-3319607000 ---p 00007000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319607000-3319608000 r--p 00007000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319608000-3319609000 rw-p 00008000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319609000-3319637000 rw-p 00000000 00:00 0
3319800000-3319816000 r-xp 00000000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319816000-3319a15000 ---p 00016000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319a15000-3319a16000 rw-p 00015000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319c00000-3319ce8000 r-xp 00000000 fd:00 534145                         /usr/lib64/libstdc++.so.6.0.13*** glibc detected *** /usr/sbin/asterisk: malloc(): memory corruption: 0x0000000003714770 ***
{noformat}
{noformat}
======= Backtrace: =========
/lib64/libc.so.6[0x3316c75e66]
/lib64/libc.so.6[0x3316c79904]
/lib64/libc.so.6(__libc_malloc+0x71)[0x3316c7a6b1]
/usr/lib64/libpj.so.2(+0x104c2)[0x7f77d99184c2]
/usr/lib64/libpj.so.2(pj_pool_create_int+0x77)[0x7f77d991dfe7]
/usr/lib64/libpj.so.2(+0x16872)[0x7f77d991e872]
/usr/lib64/libpjsip.so.2(pjsip_endpt_create_pool+0x21)[0x7f77e544a901]
/usr/lib64/libpjsip.so.2(pjsip_tx_data_create+0x67)[0x7f77e5451b57]
/usr/lib64/libpjsip.so.2(pjsip_endpt_create_request_from_hdr+0xc2)[0x7f77e544d742]
/usr/lib64/libpjsip.so.2(pjsip_dlg_create_request+0x140)[0x7f77e5461380]
/usr/lib64/libpjsip-simple.so.2(pjsip_evsub_notify+0x91)[0x7f77e567d8b1]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0xa8b1)[0x7f77e5fc28b1]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0xb9aa)[0x7f77e5fc39aa]
/usr/sbin/asterisk[0x45d0c1]
/usr/sbin/asterisk(__ao2_callback+0x5f)[0x45d3f7]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0x9c81)[0x7f77e5fc1c81]
/usr/sbin/asterisk(ast_taskprocessor_execute+0x10d)[0x5dfd52]
/usr/sbin/asterisk[0x5e6c91]
/usr/sbin/asterisk[0x5e817f]
/usr/sbin/asterisk[0x5e7f2f]
/usr/sbin/asterisk[0x5f3b66]
/lib64/libpthread.so.0[0x33170079d1]
/lib64/libc.so.6(clone+0x6d)[0x3316ce88fd]
======= Memory map: ========
00400000-00696000 r-xp 00000000 fd:00 530520                             /usr/sbin/asterisk
00896000-008a9000 rw-p 00296000 fd:00 530520                             /usr/sbin/asterisk
008a9000-008cf000 rw-p 00000000 00:00 0
01a83000-03fc6000 rw-p 00000000 00:00 0                                  [heap]
3316800000-3316820000 r-xp 00000000 fd:00 262196                         /lib64/ld-2.12.so
3316a1f000-3316a20000 r--p 0001f000 fd:00 262196                         /lib64/ld-2.12.so
3316a20000-3316a21000 rw-p 00020000 fd:00 262196                         /lib64/ld-2.12.so
3316a21000-3316a22000 rw-p 00000000 00:00 0
3316c00000-3316d8a000 r-xp 00000000 fd:00 262240                         /lib64/libc-2.12.so
3316d8a000-3316f8a000 ---p 0018a000 fd:00 262240                         /lib64/libc-2.12.so
3316f8a000-3316f8e000 r--p 0018a000 fd:00 262240                         /lib64/libc-2.12.so
3316f8e000-3316f8f000 rw-p 0018e000 fd:00 262240                         /lib64/libc-2.12.so
3316f8f000-3316f94000 rw-p 00000000 00:00 0
3317000000-3317017000 r-xp 00000000 fd:00 262262                         /lib64/libpthread-2.12.so
3317017000-3317217000 ---p 00017000 fd:00 262262                         /lib64/libpthread-2.12.so
3317217000-3317218000 r--p 00017000 fd:00 262262                         /lib64/libpthread-2.12.so
3317218000-3317219000 rw-p 00018000 fd:00 262262                         /lib64/libpthread-2.12.so
3317219000-331721d000 rw-p 00000000 00:00 0
3317400000-3317402000 r-xp 00000000 fd:00 262305                         /lib64/libdl-2.12.so
3317402000-3317602000 ---p 00002000 fd:00 262305                         /lib64/libdl-2.12.so
3317602000-3317603000 r--p 00002000 fd:00 262305                         /lib64/libdl-2.12.so
3317603000-3317604000 rw-p 00003000 fd:00 262305                         /lib64/libdl-2.12.so
3317800000-3317822000 r-xp 00000000 fd:00 262610                         /lib64/libncurses.so.5.7
3317822000-3317a21000 ---p 00022000 fd:00 262610                         /lib64/libncurses.so.5.7
3317a21000-3317a22000 rw-p 00021000 fd:00 262610                         /lib64/libncurses.so.5.7
3317c00000-3317c83000 r-xp 00000000 fd:00 262186                         /lib64/libm-2.12.so
3317c83000-3317e82000 ---p 00083000 fd:00 262186                         /lib64/libm-2.12.so
3317e82000-3317e83000 r--p 00082000 fd:00 262186                         /lib64/libm-2.12.so
3317e83000-3317e84000 rw-p 00083000 fd:00 262186                         /lib64/libm-2.12.so
3318000000-3318015000 r-xp 00000000 fd:00 262336                         /lib64/libz.so.1.2.3
3318015000-3318214000 ---p 00015000 fd:00 262336                         /lib64/libz.so.1.2.3
3318214000-3318215000 r--p 00014000 fd:00 262336                         /lib64/libz.so.1.2.3
3318215000-3318216000 rw-p 00015000 fd:00 262336                         /lib64/libz.so.1.2.3
3318400000-331841d000 r-xp 00000000 fd:00 262319                         /lib64/libselinux.so.1
331841d000-331861c000 ---p 0001d000 fd:00 262319                         /lib64/libselinux.so.1
331861c000-331861d000 r--p 0001c000 fd:00 262319                         /lib64/libselinux.so.1
331861d000-331861e000 rw-p 0001d000 fd:00 262319                         /lib64/libselinux.so.1
331861e000-331861f000 rw-p 00000000 00:00 0
3318800000-331888c000 r-xp 00000000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
331888c000-3318a8b000 ---p 0008c000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
3318a8b000-3318a8e000 rw-p 0008b000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
3318a8e000-3318a8f000 rw-p 00000000 00:00 0
3318c00000-3318c16000 r-xp 00000000 fd:00 262309                         /lib64/libresolv-2.12.so
3318c16000-3318e16000 ---p 00016000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e16000-3318e17000 r--p 00016000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e17000-3318e18000 rw-p 00017000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e18000-3318e1a000 rw-p 00000000 00:00 0
3319000000-3319002000 r-xp 00000000 fd:00 262338                         /lib64/libfreebl3.so
3319002000-3319201000 ---p 00002000 fd:00 262338                         /lib64/libfreebl3.so
3319201000-3319202000 r--p 00001000 fd:00 262338                         /lib64/libfreebl3.so
3319202000-3319203000 rw-p 00002000 fd:00 262338                         /lib64/libfreebl3.so
3319400000-3319407000 r-xp 00000000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319407000-3319607000 ---p 00007000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319607000-3319608000 r--p 00007000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319608000-3319609000 rw-p 00008000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319609000-3319637000 rw-p 00000000 00:00 0
3319800000-3319816000 r-xp 00000000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319816000-3319a15000 ---p 00016000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319a15000-3319a16000 rw-p 00015000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1*** glibc detected *** /usr/sbin/asterisk: corrupted double-linked list: 0x00007f9294330a30 ***
{noformat}
{noformat}
======= Backtrace: =========
/lib64/libc.so.6[0x3316c75e66]
/lib64/libc.so.6[0x3316c762ed]
/lib64/libc.so.6[0x3316c78948]
/usr/sbin/asterisk[0x52fde8]
/usr/sbin/asterisk(ast_json_unref+0x5d)[0x53011d]
/usr/lib64/asterisk/modules/res_sorcery_astdb.so(+0x16c8)[0x7f92bc1e36c8]
/usr/lib64/asterisk/modules/res_sorcery_astdb.so(+0x180b)[0x7f92bc1e380b]
/usr/lib64/asterisk/modules/res_sorcery_astdb.so(+0x2485)[0x7f92bc1e4485]
/usr/sbin/asterisk[0x5c22a0]
/usr/sbin/asterisk(ast_sorcery_update+0xd6)[0x5c23b9]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0x7dd3)[0x7f92f8e06dd3]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0xa9b5)[0x7f92f8e099b5]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0xb9aa)[0x7f92f8e0a9aa]
/usr/sbin/asterisk[0x45d0c1]
/usr/sbin/asterisk(__ao2_callback+0x5f)[0x45d3f7]
/usr/lib64/asterisk/modules/res_pjsip_pubsub.so(+0x9c81)[0x7f92f8e08c81]
/usr/sbin/asterisk(ast_taskprocessor_execute+0x10d)[0x5dfd52]
/usr/sbin/asterisk[0x5e6c91]
/usr/sbin/asterisk[0x5e817f]
/usr/sbin/asterisk[0x5e7f2f]
/usr/sbin/asterisk[0x5f3b66]
/lib64/libpthread.so.0[0x33170079d1]
/lib64/libc.so.6(clone+0x6d)[0x3316ce88fd]
======= Memory map: ========
00400000-00696000 r-xp 00000000 fd:00 530520                             /usr/sbin/asterisk
00896000-008a9000 rw-p 00296000 fd:00 530520                             /usr/sbin/asterisk
008a9000-008cf000 rw-p 00000000 00:00 0
0107c000-035cd000 rw-p 00000000 00:00 0                                  [heap]
3316800000-3316820000 r-xp 00000000 fd:00 262196                         /lib64/ld-2.12.so
3316a1f000-3316a20000 r--p 0001f000 fd:00 262196                         /lib64/ld-2.12.so
3316a20000-3316a21000 rw-p 00020000 fd:00 262196                         /lib64/ld-2.12.so
3316a21000-3316a22000 rw-p 00000000 00:00 0
3316c00000-3316d8a000 r-xp 00000000 fd:00 262240                         /lib64/libc-2.12.so
3316d8a000-3316f8a000 ---p 0018a000 fd:00 262240                         /lib64/libc-2.12.so
3316f8a000-3316f8e000 r--p 0018a000 fd:00 262240                         /lib64/libc-2.12.so
3316f8e000-3316f8f000 rw-p 0018e000 fd:00 262240                         /lib64/libc-2.12.so
3316f8f000-3316f94000 rw-p 00000000 00:00 0
3317000000-3317017000 r-xp 00000000 fd:00 262262                         /lib64/libpthread-2.12.so
3317017000-3317217000 ---p 00017000 fd:00 262262                         /lib64/libpthread-2.12.so
3317217000-3317218000 r--p 00017000 fd:00 262262                         /lib64/libpthread-2.12.so
3317218000-3317219000 rw-p 00018000 fd:00 262262                         /lib64/libpthread-2.12.so
3317219000-331721d000 rw-p 00000000 00:00 0
3317400000-3317402000 r-xp 00000000 fd:00 262305                         /lib64/libdl-2.12.so
3317402000-3317602000 ---p 00002000 fd:00 262305                         /lib64/libdl-2.12.so
3317602000-3317603000 r--p 00002000 fd:00 262305                         /lib64/libdl-2.12.so
3317603000-3317604000 rw-p 00003000 fd:00 262305                         /lib64/libdl-2.12.so
3317800000-3317822000 r-xp 00000000 fd:00 262610                         /lib64/libncurses.so.5.7
3317822000-3317a21000 ---p 00022000 fd:00 262610                         /lib64/libncurses.so.5.7
3317a21000-3317a22000 rw-p 00021000 fd:00 262610                         /lib64/libncurses.so.5.7
3317c00000-3317c83000 r-xp 00000000 fd:00 262186                         /lib64/libm-2.12.so
3317c83000-3317e82000 ---p 00083000 fd:00 262186                         /lib64/libm-2.12.so
3317e82000-3317e83000 r--p 00082000 fd:00 262186                         /lib64/libm-2.12.so
3317e83000-3317e84000 rw-p 00083000 fd:00 262186                         /lib64/libm-2.12.so
3318000000-3318015000 r-xp 00000000 fd:00 262336                         /lib64/libz.so.1.2.3
3318015000-3318214000 ---p 00015000 fd:00 262336                         /lib64/libz.so.1.2.3
3318214000-3318215000 r--p 00014000 fd:00 262336                         /lib64/libz.so.1.2.3
3318215000-3318216000 rw-p 00015000 fd:00 262336                         /lib64/libz.so.1.2.3
3318400000-331841d000 r-xp 00000000 fd:00 262319                         /lib64/libselinux.so.1
331841d000-331861c000 ---p 0001d000 fd:00 262319                         /lib64/libselinux.so.1
331861c000-331861d000 r--p 0001c000 fd:00 262319                         /lib64/libselinux.so.1
331861d000-331861e000 rw-p 0001d000 fd:00 262319                         /lib64/libselinux.so.1
331861e000-331861f000 rw-p 00000000 00:00 0
3318800000-331888c000 r-xp 00000000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
331888c000-3318a8b000 ---p 0008c000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
3318a8b000-3318a8e000 rw-p 0008b000 fd:00 526822                         /usr/lib64/libsqlite3.so.0.8.6
3318a8e000-3318a8f000 rw-p 00000000 00:00 0
3318c00000-3318c16000 r-xp 00000000 fd:00 262309                         /lib64/libresolv-2.12.so
3318c16000-3318e16000 ---p 00016000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e16000-3318e17000 r--p 00016000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e17000-3318e18000 rw-p 00017000 fd:00 262309                         /lib64/libresolv-2.12.so
3318e18000-3318e1a000 rw-p 00000000 00:00 0
3319000000-3319002000 r-xp 00000000 fd:00 262338                         /lib64/libfreebl3.so
3319002000-3319201000 ---p 00002000 fd:00 262338                         /lib64/libfreebl3.so
3319201000-3319202000 r--p 00001000 fd:00 262338                         /lib64/libfreebl3.so
3319202000-3319203000 rw-p 00002000 fd:00 262338                         /lib64/libfreebl3.so
3319400000-3319407000 r-xp 00000000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319407000-3319607000 ---p 00007000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319607000-3319608000 r--p 00007000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319608000-3319609000 rw-p 00008000 fd:00 262341                         /lib64/libcrypt-2.12.so
3319609000-3319637000 rw-p 00000000 00:00 0
3319800000-3319816000 r-xp 00000000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319816000-3319a15000 ---p 00016000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319a15000-3319a16000 rw-p 00015000 fd:00 262616                         /lib64/libgcc_s-4.4.7-20120601.so.1
3319c00000-3319ce8000 r-xp 00000000 fd:00 534145                         /usr/lib64/libstdc++.so.6.0.13
3319ce8000-3319ee8000 ---p 000e8000 fd:00 534145                         /usr/lib64/libstdc++.so.6.0.13
3319ee8000-3319eef000 r--p 000e8000 fd:00 534145                         /usr/lib64/libstdc++.so.6.0.13
3319eef000-3319ef1000 rw-p 000ef000 fd:00 534145                         /usr/lib64/libstdc++.so.6.0.13
3319ef1000-3319f06000 rw-p 00000000 00:00 0
331a000000-331a01d000 r-xp 00000000 fd:00 262553                         /lib64/libtinfo.so.5.7
331a01d000-331a21d000 ---p 0001d000 fd:00 262553                         /lib64/libtinfo.so.5.7
331a21d000-331a221000 rw-p 0001d000 fd:00 262553                         /lib64/libtinfo.so.5.7
331a400000-331a403000 r-xp 00000000 fd:00 262324                         /lib64/libcom_err.so.2.1
331a403000-331a602000 ---p 00003000 fd:00 262324                         /lib64/libcom_err.so.2.1
331a602000-331a603000 r--p 00002000 fd:00 262324                         /lib64/libcom_err.so.2.1
331a603000-331a604000 rw-p 00003000 fd:00 262324                         /lib64/libcom_err.so.2.1
331a800000-331a816000 r-xp 00000000 fd:00 262277                         /lib64/libnsl-2.12.so
331a816000-331aa15000 ---p 00016000 fd:00 262277                         /lib64/libnsl-2.12.so
331aa15000-331aa16000 r--p 00015000 fd:00 262277                         /lib64/libnsl-2.12.so
331aa16000-331aa17000 rw-p 00016000 fd:00 262277                         /lib64/libnsl-2.12.so
331aa17000-331aa19000 rw-p 00000000 00:00 0
331ac00000-331ad37000 r-xp 00000000 fd:00 548294                         /usr/lib64/libnss3.so
331ad37000-331af36000 ---p 00137000 fd:00 548294                         /usr/lib64/libnss3.so
331af36000-331af3b000 r--p 00136000 fd:00 548294                         /usr/lib64/libnss3.so
331af3b000-331af3d000 rw-p 0013b000 fd:00 548294                         /usr/lib64/libnss3.so
331af3d000-331af3f000 rw-p 00000000 00:00 0
331b000000-331b062000 r-xp 00000000 fd:00 548307                         /usr/lib64/libtiff.so.3.9.4
331b062000-331b261000 ---p 00062000 fd:00 548307                         /usr/lib64/libtiff.so.3.9.4
331b261000-331b264000 rw-p 00061000 fd:00 548307                         /usr/lib64/libtiff.so.3.9.4
331b400000-331b404000 r-xp 00000000 fd:00 262521                         /lib64/libplc4.so
331b404000-331b603000 ---p 00004000 fd:00 262521                         /lib64/libplc4.so
331b603000-331b604000 r--p 00003000 fd:00 262521                         /lib64/libplc4.so
331b604000-331b605000 rw-p 00004000 fd:00 262521                         /lib64/libplc4.so
331b800000-331b8db000 r-xp 00000000 fd:00 262329                         /lib64/libkrb5.so.3.3
331b8db000-331bada000 ---p 000db000 fd:00 262329                         /lib64/libkrb5.so.3.3
331bada000-331bae4000 r--p 000da000 fd:00 262329                         /lib64/libkrb5.so.3.3
331bae4000-331bae6000 rw-p 000e4000 fd:00 262329                         /lib64/libkrb5.so.3.3
331bc00000-331bc29000 r-xp 00000000 fd:00 262323                         /lib64/libk5crypto.so.3.1
331bc29000-331be29000 ---p 00029000 fd:00 262323                         /lib64/libk5crypto.so.3.1
331be29000-331be2a000 r--p 00029000 fd:00 262323                         /lib64/libk5crypto.so.3.1
331be2a000-331be2b000 rw-p 0002a000 fd:00 262323                         /lib64/libk5crypto.so.3.1
331be2b000-331be2c000 rw-p 00000000 00:00 0
331c000000-331c002000 r-xp 00000000 fd:00 262307                         /lib64/libkeyutils.so.1.3
331c002000-331c201000 ---p 00002000 fd:00 262307                         /lib64/libkeyutils.so.1.3
331c201000-331c202000 r--p 00001000 fd:00 262307                         /lib64/libkeyutils.so.1.3
331c202000-331c203000 rw-p 00002000 fd:00 262307                         /lib64/libkeyutils.so.1.3
331c400000-331c43b000 r-xp 00000000 fd:00 548295                         /usr/lib64/libssl3.so
331c43b000-331c63a000 ---p 0003b000 fd:00 548295                         /usr/lib64/libssl3.so
331c63a000-331c63d000 r--p 0003a000 fd:00 548295                         /usr/lib64/libssl3.so
331c63d000-331c63e000 rw-p 0003d000 fd:00 548295                         /usr/lib64/libssl3.so
331c63e000-331c63f000 rw-p 00000000 00:00 0
331c800000-331c80a000 r-xp 00000000 fd:00 262321                         /lib64/libkrb5support.so.0.1
331c80a000-331ca09000 ---p 0000a000 fd:00 262321                         /lib64/libkrb5support.so.0.1
331ca09000-331ca0a000 r--p 00009000 fd:00 262321                         /lib64/libkrb5support.so.0.1
331ca0a000-331ca0b000 rw-p 0000a000 fd:00 262321                         /lib64/libkrb5support.so.0.1
331cc00000-331cc41000 r-xp 00000000 fd:00 262330                         /lib64/libgssapi_krb5.so.2.2
331cc41000-331ce41000 ---p 00041000 fd:00 262330                         /lib64/libgssapi_krb5.so.2.2
331ce41000-331ce42000 r--p 00041000 fd:00 262330                         /lib64/libgssapi_krb5.so.2.2
331ce42000-331ce44000 rw-p 00042000 fd:00 262330                         /lib64/libgssapi_krb5.so.2.2
331d000000-331d019000 r-xp 00000000 fd:00 548297                         /usr/lib64/libsasl2.so.2.0.23
331d019000-331d218000 ---p 00019000 fd:00 548297                         /usr/lib64/libsasl2.so.2.0.23
331d218000-331d219000 r--p 00018000 fd:00 548297                         /usr/lib64/libsasl2.so.2.0.23
331d219000-331d21a000 rw-p 00019000 fd:00 548297                         /usr/lib64/libsasl2.so.2.0.23
331d400000-331d5b9000 r-xp 00000000 fd:00 535568                         /usr/lib64/libcrypto.so.1.0.1e
331d5b9000-331d7b8000 ---p 001b9000 fd:00 535568                         /usr/lib64/libcrypto.so.1.0.1e
331d7b8000-331d7d3000 r--p 001b8000 fd:00 535568                         /usr/lib64/libcrypto.so.1.0.1e
331d7d3000-331d7df000 rw-p 001d3000 fd:00 535568                         /usr/lib64/libcrypto.so.1.0.1e
331d7df000-331d7e3000 rw-p 00000000 00:00 0
331d800000-331d80e000 r-xp 00000000 fd:00 262535                         /lib64/liblber-2.4.so.2.10.2
331d80e000-331da0d000 ---p 0000e000 fd:00 262535                         /lib64/liblber-2.4.so.2.10.2
331da0d000-331da0e000 r--p 0000d000 fd:00 262535                         /lib64/liblber-2.4.so.2.10.2
331da0e000-331da0f000 rw-p 0000e000 fd:00 262535                         /lib64/liblber-2.4.so.2.10.2
331e000000-331e062000 r-xp 00000000 fd:00 548292                         /usr/lib64/libssl.so.1.0.1e
331e062000-331e261000 ---p 00062000 fd:00 548292                         /usr/lib64/libssl.so.1.0.1e
331e261000-331e265000 r--p 00061000 fd:00 548292                         /usr/lib64/libssl.so.1.0.1e
331e265000-331e26c000 rw-p 00065000 fd:00 548292                         /usr/lib64/libssl.so.1.0.1e
331e400000-331e44d000 r-xp 00000000 fd:00 262628                         /lib64/libldap-2.4.so.2.10.2
331e44d000-331e64c000 ---p 0004d000 fd:00 262628                         /lib64/libldap-2.4.so.2.10.2
331e64c000-331e64e000 r--p 0004c000 fd:00 262628                         /lib64/libldap-2.4.so.2.10.2
331e64e000-331e650000 rw-p 0004e000 fd:00 262628                         /lib64/libldap-2.4.so.2.10.2
331e800000-331e949000 r-xp 00000000 fd:00 532801                         /usr/lib64/libxml2.so.2.7.6
331e949000-331eb48000 ---p 00149000 fd:00 532801                         /usr/lib64/libxml2.so.2.7.6
331eb48000-331eb51000 rw-p 00148000 fd:00 532801                         /usr/lib64/libxml2.so.2.7.6
331eb51000-331eb53000 rw-p 00000000 00:00 0
331f000000-331f032000 r-xp 00000000 fd:00 262393                         /lib64/libidn.so.11.6.1
331f032000-331f231000 ---p 00032000 fd:00 262393                         /lib64/libidn.so.11.6.1
331f231000-331f232000 rw-p 00031000 fd:00 262393                         /lib64/libidn.so.11.6.1
331f400000-331f427000 r-xp 00000000 fd:00 528241                         /usr/lib64/libssh2.so.1.0.1
331f427000-331f626000 ---p 00027000 fd:00 528241                         /usr/lib64/libssh2.so.1.0.1
331f626000-331f627000 rw-p 00026000 fd:00 528241                         /usr/lib64/libssh2.so.1.0.1
331f627000-331f628000 rw-p 00000000 00:00 0
38ac000000-38ac004000 r-xp 00000000 fd:00 262149                         /lib64/libuuid.so.1.3.0
38ac004000-38ac203000 ---p 00004000 fd:00 262149                         /lib64/libuuid.so.1.3.0
38ac203000-38ac204000 rw-p 00003000 fd:00 262149                         /lib64/libuuid.so.1.3.0
38ac400000-38ac402000 r-xp 00000000 fd:00 528827                         /usr/lib/libasteriskssl.so.1
38ac402000-38ac601000 ---p 00002000 fd:00 528827                         /usr/lib/libasteriskssl.so.1
38ac601000-38ac602000 rw-p 00001000 fd:00 528827                         /usr/lib/libasteriskssl.so.1
38ac800000-38ac829000 r-xp 00000000 fd:00 539943                         /usr/lib64/libedit.so.0.0.27
38ac829000-38aca29000 ---p 00029000 fd:00 539943                         /usr/lib64/libedit.so.0.0.27
38aca29000-38aca2c000 rw-p 00029000 fd:00 539943                         /usr/lib64/libedit.so.0.0.27
{noformat}

By: Carl Fortin (phonefxg) 2015-06-30 19:22:02.862-0500

Backtrace from crash with patch
By: Carl Fortin (phonefxg) 2015-06-30 19:29:07.251-0500

Valgrind output with patch applied.

By: Richard Mudgett (rmudgett) 2015-07-01 09:47:58

Valgrind is complaining that it cannot handle the number of threads Asterisk is using so it is not going to be much help unless you compile valgrind yourself with a larger thread handling capability as mentioned in the valgrind output.

The [^backtrace_20_11.txt] backtrace isn't much help as it is just showing memory corruption.  Please try again but using MALLOC_DEBUG instead of valgrind since valgrind isn't going to help in this situation because of the number of threads involved.
By: Carl Fortin (phonefxg) 2015-07-01 10:15:55.394-0500

Ok, I have recompiled with MALLOC_DEBUG and started asterisk at 10:57:51 AM. I hope we get something other than new session this time.

By: Carl Fortin (phonefxg) 2015-07-02 06:23:51.257-0500

The system has been up for 20 hours without problem, but I can see this in the mmlog:
I know for sure that the warning appeared in the log a couple of hours after asterisk started.


1435762675 - New session
WARNING: High fence violation of 0x7f6758071100 allocated at res_pjsip_dialog_info_body_generator.c dialog_info_to_string() line 168
WARNING: High fence violation of 0x7f674c8d6c20 allocated at res_pjsip_dialog_info_body_generator.c dialog_info_to_string() line 168


The bug is still really random, sometimes it crashes right after starting it and sometimes it's 1 day later.
I'm still waiting for the next crash to happen. I wish I could reproduce the bug in my test machine, it would be less painful than in a production environment.


By: Richard Mudgett (rmudgett) 2015-07-02 15:13:02.793-0500

[^jira_asterisk_25168_v13_test2.patch] and [^jira_asterisk_25168_v13.4.0_test2.patch] replace the earlier version of the corresponding patch.  The new patch adds a fix for the MALLOC_DEBUG high fence violations you detected.

In this particular case, the detected high fence violations will not cause a crash when MALLOC_DEBUG is enabled.  MALLOC_DEBUG prevents the off by one error from corrupting any critical memory.
By: Carl Fortin (phonefxg) 2015-07-02 18:53:07.148-0500

I've installed the new patch and started asterisk an hour ago and it's running fine so far.
By: Carl Fortin (phonefxg) 2015-07-05 05:59:58.797-0500

Hi Richard,

Here's our system uptime with the  test2.patch and asterisk 13.4:

System uptime: 2 days, 11 hours, 39 minutes, 28 seconds

I'ts running quite well. We never got asterisk to run that long with the new PJsip driver, so it's a good sign.


I'll be out of the office for 3 weeks and will be back on July 27. I'll keep asterisk running with the new driver while I'm gone to collect more data .
I did a small tool to switch quickly between sip drivers in case anything goes wrong while I'm away.

I suggest to keep the issue open until I get back. It will gives us a better idea of how stable asterisk is with the new patch.
I thank you again for all your great help and look forward to talk to you when I'm back.




By: Carl Fortin (phonefxg) 2015-07-05 06:02:40.262-0500

Here is the output of memory show summary to show you the memory allocations  used by our system.

By: Richard Mudgett (rmudgett) 2015-07-06 16:33:53.921-0500

Gerrit patch reviews for the various crashes.
For the v13 branch:
https://gerrit.asterisk.org/#/c/801/ - FAX automatick reject timer crash
https://gerrit.asterisk.org/#/c/802/ - MWI XML response body buffer overwrite error
https://gerrit.asterisk.org/#/c/803/ - MWI subscription memory corruption crash

For the master branch:
https://gerrit.asterisk.org/#/c/804/ - FAX automatic reject timer crash
https://gerrit.asterisk.org/#/c/805/ - MWI XML response body buffer overwrite error
https://gerrit.asterisk.org/#/c/806/ - MWI subscription memory corruption crash
By: Carl Fortin (phonefxg) 2015-07-27 05:48:46.024-0500

Our system has been running perfectly for over 3 weeks with the patches. Thanx a lot for your help.
By: Asterisk Team (asteriskteam) 2015-07-27 05:48:46.928-0500

This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable.
By: Carl Fortin (phonefxg) 2015-07-27 05:50:29.152-0500

The patches are working great.