[Home]

Summary:ASTERISK-20994: AMI command reception after app_confbridge.so unload results in crash
Reporter:Jeremy Kister (jkister)Labels:
Date Opened:2013-01-27 12:17:28.000-0600Date Closed:2013-03-27 10:43:08
Priority:CriticalRegression?
Status:Closed/CompleteComponents:Applications/app_confbridge General
Versions:11.2.1 Frequency of
Occurrence		Constant
Related
Issues:
must be completed before resolvingASTERISK-21005 Open Blockers for 11.3.0
Environment:debian 6.0.6Attachments:( 0) backtrace.txt
( 1) jira_asterisk_20994_v11.patch
( 2) module-unload-backtrace.txt
( 3) module-unload-console-debug.txt
( 4) module-unload-network.txt
Description:# asterisk is running and happy... then:
{code}
pbx1> cd /usr/local/src/asterisk/asterisk-11.2.1
pbx1> rm apps/app_confbridge.so
pbx1> make
make[1]: Entering directory `/usr/local/src/asterisk/asterisk-11.2.1'
  [LD] app_confbridge.o confbridge/conf_config_parser.o confbridge/conf_state.o confbridge/conf_state_empty.o confbridge/conf_state_inactive.o confbridge/conf_state_multi.o confbridge/conf_state_multi_marked.o confbridge/conf_state_single.o confbridge/conf_state_single_marked.o -> app_confbridge.so
make[1]: Leaving directory `/usr/local/src/asterisk/asterisk-11.2.1'
+--------- Asterisk Build Complete ---------+
+ Asterisk has successfully been built, and +
+ can be installed by running:              +
+                                           +
+                make install               +
+-------------------------------------------+
pbx1> cp apps/app_confbridge.so /usr/lib/asterisk/modules/
pbx1> ls /tmp/core*
ls: cannot access /tmp/core*: No such file or directory
pbx1> asterisk -r
Asterisk 11.2.1, Copyright (C) 1999 - 2012 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 11.2.1 currently running on pbx1 (pid = 16898)
pbx1*CLI>
Disconnected from Asterisk server
Asterisk cleanly ending (0).
Executing last minute cleanups
pbx1> ls /tmp/core*
core.pbx1.example.net-2013-01-27T13:15:04-0500
pbx1>
{code}
Comments:By: Matt Jordan (mjordan) 2013-01-28 11:54:52.095-0600

Did you unload the old app_confbridge module before removing it?

I'd expect bad things if you blow out a shared object library while Asterisk still technically has it loaded in memory.
By: Jeremy Kister (jkister) 2013-01-28 12:41:37.237-0600

i did not unload the old app_confbridge-

in asterisk 1.8, it was ok to unpack/install a new asterisk version right on top of the old version.  i expected it to be the same for 11.  perhaps it was just a coincidence that it worked on 1.8 ?

if this is the expected behavior, perhaps a NB in UPDATE.txt is needed.
By: Matt Jordan (mjordan) 2013-01-28 13:11:07.995-0600

There haven't been any changes in module loading/unloading. Deleting shared object libraries when something still has it loaded into memory just strikes me as unsafe in any situation, so I'm a little surprised that it worked just fine in the past.

What's interesting about this is that it crashed when executing 'logger mute silent'. I'm curious, if you do unload app_confbridge first (asterisk -rx "module unload app_confbridge.so") and then do the rm/cp, does it still crash?

{quote}
#0  0x00008816 in ?? ()
(gdb) bt
#0  0x00008816 in ?? ()
#1  0x080d9372 in ast_cli_command_multiple_full (uid=0, gid=14, fd=19, size=19, s=0xb52c30b8 "logger mute silent") at cli.c:2583
#2  0x08085944 in netconsole (vconsole=0x824a840) at asterisk.c:1341
#3  0x081b6405 in dummy_start (data=0xb54f5188) at utils.c:1028
#4  0xb724d955 in start_thread () from /lib/i686/cmov/libpthread.so.0
#5  0xb768b1de in clone () from /lib/i686/cmov/libc.so.6
{quote}
By: Jeremy Kister (jkister) 2013-01-28 20:16:19.524-0600

it appears that asterisk crashes on unload of the app_confbridge.so too-

the crash is not always immediate, but always within 30 seconds (with an idle server)


{code}
pbx1> asterisk -r
Asterisk 11.2.1, Copyright (C) 1999 - 2012 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 11.2.1 currently running on pbx1 (pid = 17063)
pbx1*CLI> module unload app_confbridge.so
Unloaded app_confbridge.so
 == Unregistered application 'ConfBridge'
 == Unregistered custom function CONFBRIDGE
 == Unregistered custom function CONFBRIDGE_INFO
 == Unregistered channel type 'ConfBridgeRec'
 == Manager unregistered action ConfbridgeList
 == Manager unregistered action ConfbridgeListRooms
 == Manager unregistered action ConfbridgeMute
 == Manager unregistered action ConfbridgeUnmute
 == Manager unregistered action ConfbridgeKick
 == Manager unregistered action ConfbridgeUnlock
 == Manager unregistered action ConfbridgeLock
 == Manager unregistered action ConfbridgeStartRecord
 == Manager unregistered action ConfbridgeStopRecord
pbx1*CLI>
Disconnected from Asterisk server
Asterisk cleanly ending (0).
Executing last minute cleanups
pbx1> Asterisk ended with exit status 139
Asterisk exited on signal 11.
Exited on signal 11
Automatically restarting Asterisk.
{code}

backtrace attached.
By: Jeremy Kister (jkister) 2013-01-28 20:25:02.082-0600

i've done this process [rm apps/app_XXX.so; make; cp apps/app_XXX.so /usr/lib/asterisk/modules; asterisk -r] and also [module unload app_XXX.so] with several other modules and I havent found any that asterisk gets upset about; this problem may be limited to app_confbridge.so vs. any module.

By: Jeremy Kister (jkister) 2013-01-31 12:04:48.938-0600

i've narrowed the "module unload" problem down a bit further.  *after* module unload app_confbridge.so AND my monitoring program comes along to log in with the AMI is when asterisk core dumps.

By: Rusty Newton (rnewton) 2013-02-05 19:13:07.642-0600

{quote}
i've narrowed the "module unload" problem down a bit further. after module unload app_confbridge.so AND my monitoring program comes along to log in with the AMI is when asterisk core dumps.
{quote}

Jeremy, can you attach an Asterisk full log (VERBOSE and DEBUG level 5 at least, including AMI debug) showing up to the crash? If the log doesn't show the AMI command received by Asterisk when it crashes, perhaps your application has a log of the last command sent?

By: Jeremy Kister (jkister) 2013-02-05 22:28:07.716-0600

no particular AMI query seems to trigger the core dump - just any ami query after unloading the module.
By: Jeremy Kister (jkister) 2013-02-05 22:29:01.338-0600

note that i've disabled chan_sip and res_xmpp simply because they were too chatty in the debug log.  I stopped the network capture very close to the time asterisk crashed.
By: Jeremy Kister (jkister) 2013-02-05 22:35:51.014-0600

not much help with the console debug, but info is attached.
By: Rusty Newton (rnewton) 2013-02-07 18:52:05.323-0600

Yup, blows up Asterisk. Reproduced in SVN-branch-11-r380384

1. module unload app_confbridge.so
2. any new AMI command from a current connection, or a login for a new connection will crash Asterisk.
By: Richard Mudgett (rmudgett) 2013-02-07 19:03:05.826-0600

[^jira_asterisk_20994_v11.patch] - Add missing unregister of the ConfbridgeSetSingleVideoSrc AMI action when ConfBridge is unloaded.
By: Rusty Newton (rnewton) 2013-02-07 19:18:22.743-0600

patched SVN-branch-11-r380974M, looks good, no more crashing.
By: Jeremy Kister (jkister) 2013-02-07 21:23:32.898-0600

confirmed your 20994_v11 patch fixes the module unload+ami command crash.

do we care about the original issue (where a make install in the source directory followed by 'asterisk -r') causes a crash ?

As per Matt's previous question (unload app_confbridge.so/make/make install/asterisk -r) now that unloading app_confbridge is fixed i can say that no, asterisk does not crash if i unload app_confbridge.so before make/make install/asterisk -r in the original source directory.
By: Jeremy Kister (jkister) 2013-02-08 11:41:48.375-0600

did something automated close this, or did someone miss my last question ?
By: Richard Mudgett (rmudgett) 2013-02-08 11:51:19.988-0600

It closed because I committed the patch.

I thought you had also said the original crash was fixed.  Is it fixed?
By: Jeremy Kister (jkister) 2013-02-08 11:53:28.618-0600

the original crash is not fixed - just the crash that I found while debugging the original crash.  As per Matt's [comment|https://issues.asterisk.org/jira/browse/ASTERISK-20994?focusedCommentId=202320&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-202320] it's questionable if we care about the original crash.  
By: Matt Jordan (mjordan) 2013-02-08 15:31:18.848-0600

No, it probably shouldn't crash on module unload.
By: Jeremy Kister (jkister) 2013-02-08 22:31:32.793-0600

The module unload crash has been fixed by rmudgett.

The original issue is:
{code}
cd /usr/local/src/asterisk-11.2.1
./configure ; make ; make install
/etc/init.d/asterisk start
{code}
...asterisk is happy

{code}
rm apps/app_confbridge.so
make
cp apps/app_confbridge.so /usr/lib/asterisk/modules/
asterisk -r
{code}

here asterisk crashes on "logger mute silent"

it is this issue that is the original issue and remains.

interestingly,
# the new .so made is identical to the original .so
# simply changing the inode is *not* the cause of the crash
# removing the .so while asterisk has it loaded is *not* the cause of the crash

{code}
cd /usr/lib/asterisk/modules
cp app_confbridge.so ..
rm app_confbridge.so
asterisk -r
{code}

.. asterisk does not crash - we continue:
{code}
cp ../app_confbridge.so .
asterisk -r
{code}

.. asterisk is still happy.

By: Walter Doekes (wdoekes) 2013-02-25 06:29:24.571-0600

@matt: Deleting the .so while loaded is safe. In fact, it is better than not doing it.

If you copy a new .so on top of the old one, strange things happen.
If you delete the old one first, the kernel still keeps the old copy around until all fd's to it are closed. This is what the installer does on make install.
By: Jeremy Kister (jkister) 2013-03-27 10:42:57.617-0500

as per walter's explenation, this should be closed fixed with rmudgett's patch.