Summary: | ASTERISK-22857: Deadlock: Locked Here: chan_iax2.c line 9756 (socket_read), when compiled with DEBUG_THREADS | ||||||
Reporter: | David Brillert (aragon) | Labels: | |||||
Date Opened: | 2013-11-15 08:40:26.000-0600 | Date Closed: | 2014-06-25 11:15:27 | ||||
Priority: | Critical | Regression? | No | ||||
Status: | Closed/Complete | Components: | Channels/chan_iax2 | ||||
Versions: | SVN 11.6.0 | Frequency of Occurrence | Frequent | ||||
Related Issues: |
| ||||||
Environment: | 64 bit CentOS, 4GB RAM | Attachments: | ( 0) backtrace.txt ( 1) core_show_locks_gdb_thread_apply_all_bt_full.txt ( 2) core_show_locks_svn_manual.txt ( 3) gdbcorebtbtfull.txt ( 4) iax2_debug.txt ( 5) valgrind.txt | ||||
Description: | Full deadlock no call processing, no SIP traffic.
GDB output DONT_OPTIMIZE DEBUG_THREADS BETTER_BACKTRACES thread apply all bt full.txt attached. The deadlock occurred overnight while the system was idle. It is happening on multiple production servers. | ||||||
Comments: | By: David Brillert (aragon) 2013-11-15 08:41:59.127-0600 Output from: core show locks gdb thread apply all bt gdb thread apply all bt full By: David Brillert (aragon) 2013-11-15 08:50:12.418-0600 I still see 'value optimized out' in the output. I just don't understand why this is happening when Asterisk is compiled with DONT_OPTIMIZE DEBUG_THREADS BETTER_BACKTRACES By: Matt Jordan (mjordan) 2013-11-18 09:16:40.069-0600 On symbols: It depends on if you've replaced all of the modules that were compiled previously with those that have the symbols. It's possible that some of the modules that are getting loaded weren't compiled with those options. The deadlock here appears to be occurring when {{defer_full_frame}} is called. Thread 0xb6f6bb90 is blocked while obtaining the lock for {{&to_here->lock}} during {{defer_full_frame}}, while at the same time it holds the {{active_list}} lock. Unfortunately, what I can't find given the current information is the thread that has itself locked, while holding the {{active_list}} lock. The {{core show locks}} doesn't show this information, which in itself is very odd (as it should), and the {{gdb}} backtrace doesn't show all of the information due to a lack of symbols. Looking at the code, if there was a call to {{signal_condition}} that occurred while {{active_list}}'s lock was held, I would expect this to occur - but I don't see that happening either. Just to confirm - do you have any patches to {{chan_iax2}}? By: David Brillert (aragon) 2013-11-18 09:43:23.621-0600 No custom patches to chan_iax2. Using iaxmodem and hylafax and faxopts... Just ran into another lock but not a full deadlock. Asterisk is able to process calls. {noformat} === Currently Held Locks ======================================================================= === === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked) === === Thread ID: 0xb755bb90 (tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()) === ---> Lock #0 (chan_iax2.c): MUTEX 4316 transmit_frame &iaxsl[fr->callno] 0x13689e0 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x812830a] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0x96) [0x8123d06] /usr/lib/asterisk/modules/chan_iax2.so [0x12a4c1d] /usr/sbin/asterisk [0x8199483] /usr/sbin/asterisk [0x81a9d71] /lib/libpthread.so.0 [0x2ef912] /lib/libc.so.6(clone+0x5e) [0x2197ce] === ------------------------------------------------------------------- === === Thread ID: 0xb73c3b90 (iax2_process_thread started at [12469] chan_iax2.c start_network_thread()) === ---> Waiting for Lock #0 (chan_iax2.c): MUTEX 12224 __iax2_poke_noanswer &iaxsl[callno] 0x13689e0 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x812830a] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0x96) [0x8123d06] /usr/lib/asterisk/modules/chan_iax2.so [0x12b94e3] /usr/lib/asterisk/modules/chan_iax2.so [0x12c9b19] /usr/sbin/asterisk(ast_module_reload+0x36a) [0x812155a] /usr/sbin/asterisk [0x80c824e] /usr/sbin/asterisk(ast_cli_command_full+0x159) [0x80c9a69] /usr/sbin/asterisk [0x8137fe8] /usr/sbin/asterisk [0x8134c42] /usr/sbin/asterisk [0x814272a] /usr/sbin/asterisk [0x819a6b9] /usr/sbin/asterisk [0x81a9d71] /lib/libpthread.so.0 [0x2ef912] /lib/libc.so.6(clone+0x5e) [0x2197ce] === --- ---> Locked Here: chan_iax2.c line 4316 (transmit_frame) === ------------------------------------------------------------------- === === Thread ID: 0xb71e3b90 (iax2_process_thread started at [12469] chan_iax2.c start_network_thread()) === ---> Waiting for Lock #0 (chan_iax2.c): MUTEX 2911 __find_callno &iaxsl[dcallno] 0x13689e0 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x812830a] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0x96) [0x8123d06] /usr/lib/asterisk/modules/chan_iax2.so [0x12b94e3] /usr/lib/asterisk/modules/chan_iax2.so [0x12c9b19] /usr/sbin/asterisk(ast_module_reload+0x36a) [0x812155a] /usr/sbin/asterisk [0x80c824e] /usr/sbin/asterisk(ast_cli_command_full+0x159) [0x80c9a69] /usr/sbin/asterisk [0x8137fe8] /usr/sbin/asterisk [0x8134c42] /usr/sbin/asterisk [0x814272a] /usr/sbin/asterisk [0x819a6b9] /usr/sbin/asterisk [0x81a9d71] /lib/libpthread.so.0 [0x2ef912] /lib/libc.so.6(clone+0x5e) [0x2197ce] === --- ---> Locked Here: chan_iax2.c line 4316 (transmit_frame) === ------------------------------------------------------------------- === === Thread ID: 0xb693fb90 (handle_tcptls_connection started at [ 322] tcptls.c ast_tcptls_server_root()) === ---> Lock #0 (loader.c): MUTEX 739 ast_module_reload &reloadlock 0x821b420 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x812830a] /usr/sbin/asterisk(__ast_pthread_mutex_trylock+0x97) [0x8125b87] /usr/sbin/asterisk(ast_module_reload+0xfc) [0x81212ec] /usr/sbin/asterisk [0x80c824e] /usr/sbin/asterisk(ast_cli_command_full+0x159) [0x80c9a69] /usr/sbin/asterisk [0x8137fe8] /usr/sbin/asterisk [0x8134c42] /usr/sbin/asterisk [0x814272a] /usr/sbin/asterisk [0x819a6b9] /usr/sbin/asterisk [0x81a9d71] /lib/libpthread.so.0 [0x2ef912] /lib/libc.so.6(clone+0x5e) [0x2197ce] === ---> Lock #1 (loader.c): MUTEX 779 ast_module_reload &(&module_list)->lock 0x821b3a8 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x812830a] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0x96) [0x8123d06] /usr/sbin/asterisk(ast_module_reload+0x1eb) [0x81213db] /usr/sbin/asterisk [0x80c824e] /usr/sbin/asterisk(ast_cli_command_full+0x159) [0x80c9a69] /usr/sbin/asterisk [0x8137fe8] /usr/sbin/asterisk [0x8134c42] /usr/sbin/asterisk [0x814272a] /usr/sbin/asterisk [0x819a6b9] /usr/sbin/asterisk [0x81a9d71] /lib/libpthread.so.0 [0x2ef912] /lib/libc.so.6(clone+0x5e) [0x2197ce] === ---> Waiting for Lock #2 (chan_iax2.c): MUTEX 12277 iax2_poke_peer &iaxsl[callno] 0x13689e0 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x812830a] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0x96) [0x8123d06] /usr/lib/asterisk/modules/chan_iax2.so [0x12b94e3] /usr/lib/asterisk/modules/chan_iax2.so [0x12c9b19] /usr/sbin/asterisk(ast_module_reload+0x36a) [0x812155a] /usr/sbin/asterisk [0x80c824e] /usr/sbin/asterisk(ast_cli_command_full+0x159) [0x80c9a69] /usr/sbin/asterisk [0x8137fe8] /usr/sbin/asterisk [0x8134c42] /usr/sbin/asterisk [0x814272a] /usr/sbin/asterisk [0x819a6b9] /usr/sbin/asterisk [0x81a9d71] /lib/libpthread.so.0 [0x2ef912] /lib/libc.so.6(clone+0x5e) [0x2197ce] === --- ---> Locked Here: chan_iax2.c line 4316 (transmit_frame) {noformat} By: David Brillert (aragon) 2013-11-18 09:59:14.176-0600 [2013-11-18 10:44:41] ERROR[28189]: chan_iax2.c:2470 peercnt_add: maxcallnumber limit of 2048 for 192.168.192.140 has been reached! That extension is a Digium IAXy. And then 'iax2 show peers' displays nothing (locked). iax2 reload does not recover the module, reload doesn't complete. Restarting Asterisk brings back my iax2 peers. By: David Brillert (aragon) 2013-11-18 10:06:07.640-0600 Maybe the symbols problem is caused by distcc (cache compiler). Rebuilding w/o distcc/ccache. I should have another lock within 24 hours. By: David Brillert (aragon) 2013-11-18 10:10:32.373-0600 Compared binary and no changes so not related to distcc etc. On each build, cleaned build directory etc.. .so no "old" modules By: David Brillert (aragon) 2013-11-18 12:34:01.668-0600 I enabled verbose to file and iax2 set debug on. Then ran a couple of iax2 commands to see why the maxcallnumber limit exceeded. Issuing the IAX2 commands resulted in a deadlock in IAX2 core show locks didn't respond and other CLI commands did not respond. So I attached with GDB Attached messages file and gdb output. So it looks like I can reproduce by running the IAXy for a little while and then executing some iax2 CLI commands. By: David Brillert (aragon) 2013-11-18 13:29:47.830-0600 Also saw a coredump today. Attached GDB core trace. By: David Brillert (aragon) 2013-11-19 09:33:36.045-0600 I ran Asterisk under Valgrind and reproduced a lock. Valgrind debug output is attached. These are the commands I used to lock IAX2 iax2 show peers Name/Username Host Mask Port Status Description 236/236 (null) (D) 255.255.255.255 0 UNKNOWN iaxmodem0 127.0.0.1 (S) 255.255.255.255 4570 OK (185 ms) iaxmodem1 127.0.0.1 (S) 255.255.255.255 4571 OK (224 ms) iaxmodem2 127.0.0.1 (S) 255.255.255.255 4572 OK (86 ms) iaxmodem3 127.0.0.1 (S) 255.255.255.255 4573 OK (166 ms) 610/610 192.168.192.140 (D) 255.255.255.255 4569 OK (115 ms) provision prune reload set show test unregister *CLI> iax2 show cache callnumber channels firmware netstats peer peers provisioning registry stats threads users *CLI> iax2 show threads IAX2 Thread Information Idle Threads: Thread 9: state=0, update=40, actions=407, func='' Thread 10: state=0, update=36, actions=426, func='' Thread 1: state=0, update=36, actions=411, func='' Thread 2: state=0, update=36, actions=435, func='' Thread 4: state=0, update=36, actions=420, func='' Thread 6: state=0, update=35, actions=421, func='' Thread 7: state=0, update=35, actions=409, func='' Thread 5: state=0, update=34, actions=412, func='' Thread 8: state=0, update=34, actions=422, func='' Thread 3: state=0, update=33, actions=429, func='' Active Threads: Dynamic Threads: 10 of 10 threads accounted for with 0 dynamic threads *CLI> iax2 show netstats -------- LOCAL --------------------- -------- REMOTE -------------------- Channel RTT Jit Del Lost % Drop OOO Kpkts Jit Del Lost % Drop OOO Kpkts FirstMsg LastMsg iax2 show peers 0 active IAX channels *CLI> iax2 show peers Name/Username Host Mask Port Status Description 236/236 (null) (D) 255.255.255.255 0 UNKNOWN iaxmodem0 127.0.0.1 (S) 255.255.255.255 4570 OK (420 ms) iaxmodem1 127.0.0.1 (S) 255.255.255.255 4571 OK (457 ms) iaxmodem2 127.0.0.1 (S) 255.255.255.255 4572 OK (161 ms) iaxmodem3 127.0.0.1 (S) 255.255.255.255 4573 OK (331 ms) 610/610 192.168.192.140 (D) 255.255.255.255 4569 OK (1176 ms) *CLI> [2013-11-19 10:01:36] NOTICE[16411]: chan_iax2.c:12219 __iax2_poke_noanswer: Peer 'iaxmodem2' is now UNREACHABLE! Time: 161 [2013-11-19 10:01:37] NOTICE[16412]: chan_iax2.c:12219 __iax2_poke_noanswer: Peer 'iaxmodem3' is now UNREACHABLE! Time: 331 [2013-11-19 10:01:41] NOTICE[16407]: chan_iax2.c:12219 __iax2_poke_noanswer: Peer 'iaxmodem0' is now UNREACHABLE! Time: 420 [2013-11-19 10:01:42] NOTICE[16409]: chan_iax2.c:12219 __iax2_poke_noanswer: Peer 'iaxmodem1' is now UNREACHABLE! Time: 457 [2013-11-19 10:01:45] NOTICE[16412]: chan_iax2.c:12219 __iax2_poke_noanswer: Peer '610' is now UNREACHABLE! Time: 1176 By: David Brillert (aragon) 2013-11-19 11:00:31.428-0600 I ran into a segfault yesterday but the backtrace looked useless. I ran a different gdb command today and values are still optimized out but the backtrace looks a lot more useful. backtrace.txt By: David Brillert (aragon) 2013-11-19 11:11:32.666-0600 @Matt Regarding the optimized out values, can I get some ideas from this? gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c core.31340 > /tmp/backtrace.txt warning: .dynamic section for "/usr/lib/libasteriskssl.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib/libdl.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib/libpthread.so.0" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib/libtermcap.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib/libltdl.so.3" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib/libnsl.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib/libgcrypt.so.11" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib/libgpg-error.so.0" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib/libutil.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: Can't read pathname for load map: Input/output error. warning: Can't read pathname for load map: Input/output error. warning: Can't read pathname for load map: Input/output error. warning: Can't read pathname for load map: Input/output error. warning: Can't read pathname for load map: Input/output error. warning: Can't read pathname for load map: Input/output error. By: David Brillert (aragon) 2013-11-19 11:13:45.479-0600 (gdb) bt #0 0x08089bb0 in __ao2_find () #1 0x08157df0 in ast_add_hint () #2 0x08161165 in ast_add_extension2_lockopt () #3 0x003d9a33 in pbx_load_config () at pbx_config.c:1644 #4 pbx_load_module () at pbx_config.c:1848 #5 0x0812155a in ast_module_reload () #6 0x080cdff9 in handle_reload () #7 0x080c9a69 in ast_cli_command_full () #8 0x023612af in cli_alias_passthrough (e=0xb711d1c4, cmd=-4, a=0xb76a0c04) at res_clialiases.c:131 #9 0x080c9a69 in ast_cli_command_full () #10 0x080c9cd5 in ast_cli_command_multiple_full () #11 0x080802d2 in netconsole () #12 0x081a9d71 in dummy_start () #13 0x002b8912 in start_thread () from /lib/libpthread.so.0 #14 0x0022b7ce in clone () from /lib/libc.so.6 (gdb) bt full #0 0x08089bb0 in __ao2_find () No symbol table info available. #1 0x08157df0 in ast_add_hint () No symbol table info available. #2 0x08161165 in ast_add_extension2_lockopt () No symbol table info available. #3 0x003d9a33 in pbx_load_config () at pbx_config.c:1644 __PRETTY_FUNCTION__ = "pbx_load_config" #4 pbx_load_module () at pbx_config.c:1848 con = <value optimized out> __PRETTY_FUNCTION__ = "pbx_load_module" #5 0x0812155a in ast_module_reload () No symbol table info available. #6 0x080cdff9 in handle_reload () No symbol table info available. #7 0x080c9a69 in ast_cli_command_full () No symbol table info available. #8 0x023612af in cli_alias_passthrough (e=0xb711d1c4, cmd=-4, a=0xb76a0c04) at res_clialiases.c:131 tmp = {cli_entry = {cmda = {0x0 <repeats 16 times>}, summary = 0x0, usage = 0x0, inuse = 0, module = 0x0, _full_cmd = 0x0, cmdlen = 0, args = 0, command = 0xb7114158 "reload", handler = 0, list = {next = 0x0}}, alias = 0x0, real_cmd = 0x0} generator = 0x0 line = <value optimized out> #9 0x080c9a69 in ast_cli_command_full () No symbol table info available. #10 0x080c9cd5 in ast_cli_command_multiple_full () No symbol table info available. #11 0x080802d2 in netconsole () No symbol table info available. #12 0x081a9d71 in dummy_start () No symbol table info available. #13 0x002b8912 in start_thread () from /lib/libpthread.so.0 No symbol table info available. #14 0x0022b7ce in clone () from /lib/libc.so.6 By: David Brillert (aragon) 2013-11-19 11:40:28.165-0600 I'm no expert but I would guess that the crash is happening when we do a reload and because we are using dynamic hints. We recently changes to using dynamic hints to reduce CPU load during reloads. By: David Brillert (aragon) 2013-11-20 13:35:50.111-0600 This is definitely a regression in SVN. Since I was able to reproduce easily with some CLI commands I tested some older revisions. The last stable Asterisk revision was 399516 which we installed on September 27th 2013. The only commit to chan_iax2 since that date is: Revision 401016 - Directory Listing Modified Tue Oct 15 19:57:06 2013 UTC (5 weeks ago) by rmudgett chan_iax2: Fix channel left locked in off nominal code path. So this IMHO must be the cause of the regression. By: David Brillert (aragon) 2013-11-21 08:34:51.521-0600 Reverted Revision 401016 and I can no longer reproduce the deadlock. By: Matt Jordan (mjordan) 2013-11-21 09:27:26.184-0600 Hm. That doesn't make a lot of sense. The offending code is this: {noformat} iax2_lock_owner(fr->callno); if (iaxs[fr->callno]) { if (iaxs[fr->callno]->owner) { struct ast_format_cap *orignative = ast_format_cap_dup(ast_channel_nativeformats(iaxs[fr->callno]->owner)); struct ast_format_cap *native = ast_channel_nativeformats(iaxs[fr->callno]->owner); if (orignative) { ast_format_cap_set(native, &f.subclass.format); if (ast_channel_readformat(iaxs[fr->callno]->owner)->id) { ast_set_read_format(iaxs[fr->callno]->owner, ast_channel_readformat(iaxs[fr->callno]->owner)); } ast_format_cap_copy(native, orignative); orignative = ast_format_cap_destroy(orignative); } ast_channel_unlock(iaxs[fr->callno]->owner); {noformat} Richard's patch moved the {{ast_channel_unlock}} from inside the {{if (orignative)}} call to outside of it. The only time that would have made any difference is if the channel did not have native formats on it (which is fairly off nominal) - in which case, the original code would have left the channel locked. I'm not sure how or why reverting this would cause any difference to the scenario you're seeing. By: David Brillert (aragon) 2013-11-21 12:47:43.908-0600 OK I was able to lock it up with and without commit 401016 so that's not it. It took much longer without 401016... Good news is that I appear to have a usable backtrace and core show locks from the fresh lock and a fresh checkout of SVN. By: Richard Mudgett (rmudgett) 2013-11-21 20:07:28.472-0600 The deadlock you captured is in the DEBUG_THREADS code itself. See ASTERISK-19463 and https://reviewboard.asterisk.org/r/2826/ Do you normally run with DEBUG_THREADS enabled? By: David Brillert (aragon) 2013-11-21 21:53:35.763-0600 Ideally I'd like to run with DEBUG_THREADS enabled, at least that way when I run into a deadlock I can get a trace right away. But to answer your question no. I do not normally run with DEBUG_THREADS. By: David Brillert (aragon) 2013-11-21 21:58:18.297-0600 Actually I was running DEBUG_THREADS to track down another problem that looked like a SIP deadlock, then I ran into this problem. By: Matt Jordan (mjordan) 2013-12-07 20:17:32.782-0600 David: can you try the patch on reviewboard? * {{wget https://reviewboard.asterisk.org/r/2826/diff/raw/ -O 2826.patch}} * {{patch -p0 < 2826.patch}} By: David Brillert (aragon) 2013-12-09 08:28:21.925-0600 Will do after the holiday break, in the New Year. By: David Brillert (aragon) 2014-01-02 12:43:21.760-0600 Before I test, how far is this patch from being committed? It seems to have some open issues... https://reviewboard.asterisk.org/r/2826/ rmudgett 1 month, 1 week ago (Nov. 25, 2013, 1:12 p.m.) Instead of using the pointer as a flag variable which cannot be changed atomically, use a volatile int as the flag variable. If you still have concernes then you can use ast_atomic_fetchadd_int() to examine and change the int flag. By: David M. Lee (dlee) 2014-01-02 14:08:39.023-0600 David B., Richard's comment is a performance concern, but I would rather make sure we get a correct patch before we start optimizing it. As far as committing it, that partially depends on whether it fixes your debug threads related deadlock or not :-) I wrote the patch trying to correct some debug thread related deadlocks we were seeing on our CI servers. But the patch didn't fix the deadlocks we were seeing, so I've held off on committing it. Your backtrace looks like the same sort of deadlock that this patch intends to fix. If the patch works for you, that's a good sign that it's correct and worth putting in. But if it doesn't I'll probably discard the code review since there would be something much more sneaky going on. By: David Brillert (aragon) 2014-01-09 09:49:02.476-0600 To reproduce the problem just connect a Digium IAXy and leave it connected and idle for less than 48 hours. By: Rusty Newton (rnewton) 2014-01-21 09:13:44.573-0600 David do you have any results from testing with the patch? By: David Brillert (aragon) 2014-01-27 12:23:58.380-0600 No I haven't tested the patch. Since I can't get Asterisk to give me a segfault with unoptimized backtraces no matter how I compile it. And because I provided easy steps to reproduce by compiling with DEBUG_THREADS and installing a Digium IAXy. A registered and idle IAXy is enough to drive Asterisk crazy within a day or two. You don't even have to load Asterisk with any traffic/channels. Here is the relevant iax2.conf [610] type = friend mohinterpret = default_default mohsuggest = default_default accountcode = 610 amaflags = default parkinglot = parkinglot_default username = 610 auth = md5 secret = scrubbed requirecalltoken = no host = dynamic language = en cc_agent_policy = generic cc_monitor_policy = generic cc_offer_timer = 20 callgroup = 1 pickupgroup = 1 dtmfmode = rfc2833 encryption = no qualify = 2000 transfer = no trunk = no disallow = all allow = ulaw allow = gsm context = default-default setvar = DYNAMIC_FEATURES=atxfer#fmcxfer#blindxfer#automon#automixmon#hookflash#supervisor#conference#automonmute setvar = EXTCONTEXT=default-default setvar = TRANSFER_CONTEXT=default-default setvar = AUTO_RECORDING=610 setvar = MONITOR_OPTION=wW setvar = INBOUND_GROUP=610@INCOMING_default setvar = SPYGROUP=default By: Matt Jordan (mjordan) 2014-06-25 11:15:27.677-0500 This issue has been reported a lot under a number of different names. To keep the issue tracker sane - and so everyone who runs into this problem can find the same solution - I'm closing this out as a duplicate of ASTERISK-22851. |