Summary: | ASTERISK-24238: Double free or corruption (!prev) - moh_files_generator | ||
Reporter: | Lee Webb (lwebb) | Labels: | |
Date Opened: | 2014-08-14 23:02:23 | Date Closed: | 2014-09-04 20:19:00 |
Priority: | Critical | Regression? | |
Status: | Closed/Complete | Components: | Resources/res_musiconhold |
Versions: | 1.8.20.1 | Frequency of Occurrence | One Time |
Related Issues: | |||
Environment: | CentOS 6.5 x86_64 | Attachments: | |
Description: | Hi Team,
We've recently experienced what I'm hoping is a once off Asterisk crash while using 1.8.20.1. {code} Aug 15 01:13:46 gsdcvoice07 asterisk: *** glibc detected *** /home/asterisk/asterisk-bin/sbin/asterisk: double free or corruption (!prev): 0x00000000021454d0 *** {code} The GDB back trace contained the following {code} Program terminated with signal 6, Aborted. #0 0x00007f1ab1eb0925 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.2.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-15.el6_5.1.x86_64 libcom_err-1.41.12-18.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libstdc++-4.4.7-4.el6.x86_64 libxml2-2.7.6-14.el6_5.2.x86_64 mysql-libs-5.5.38-1.el6.remi.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64 openssl-1.0.1e-16.el6_5.14.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f1ab1eb0925 in raise () from /lib64/libc.so.6 #1 0x00007f1ab1eb2105 in abort () from /lib64/libc.so.6 #2 0x00007f1ab1eee837 in __libc_message () from /lib64/libc.so.6 #3 0x00007f1ab1ef4166 in malloc_printerr () from /lib64/libc.so.6 #4 0x00007f1ab1ef6ca3 in _int_free () from /lib64/libc.so.6 #5 0x00000000004b6f9b in __frame_free (frame=0x21454d0, cache=1) at frame.c:370 #6 ast_frame_free (frame=0x21454d0, cache=1) at frame.c:382 #7 0x00007f1aadcec5ef in moh_files_generator (chan=0x7f1a1c075268, data=<value optimized out>, len=<value optimized out>, samples=<value optimized out>) at res_musiconhold.c:392 #8 0x000000000046eb60 in generator_force (data=0x7f1a1c075268) at channel.c:3120 #9 0x000000000047195f in __ast_read (chan=0x7f1a1c075268, dropaudio=<value optimized out>) at channel.c:3876 #10 0x0000000000476c5c in ast_read (chan=0x7f1a1c075268, timeout_ms=1000, cond=0x7f1a75619f45 <agent_cont_sleep>, data=0x201f320) at channel.c:4336 #11 ast_safe_sleep_conditional (chan=0x7f1a1c075268, timeout_ms=1000, cond=0x7f1a75619f45 <agent_cont_sleep>, data=0x201f320) at channel.c:1870 #12 0x00007f1a7561f55d in login_exec (chan=0x7f1a1c075268, data=<value optimized out>) at chan_agent.c:2183 #13 0x00000000004e9185 in pbx_exec (c=0x7f1a1c075268, app=0x1ff6a30, data=0x7f1a68296680 "500241051,s") at pbx.c:1446 #14 0x00000000004f3b3c in pbx_extension_helper (c=0x7f1a1c075268, con=0x0, context=0x7f1a1c0757c0 "agent-login", exten=0x7f1a1c075810 "s", priority=9, label=0x7f1a1c0757c0 "agent-login", callerid=0x7f1a1c0e7ad0 "61295898015", action=E_SPAWN, found=0x7f1a68298cfc, combined_find_spawn=1) at pbx.c:4489 #15 0x00000000004fa222 in ast_spawn_extension (c=0x7f1a1c075268, args=0x0) at pbx.c:5127 #16 __ast_pbx_run (c=0x7f1a1c075268, args=0x0) at pbx.c:5230 #17 0x00000000004fbb72 in pbx_thread (data=<value optimized out>) at pbx.c:5571 #18 0x000000000053298c in dummy_start (data=<value optimized out>) at utils.c:1010 #19 0x00007f1ab12779d1 in start_thread () from /lib64/libpthread.so.0 #20 0x00007f1ab1f66b5d in clone () from /lib64/libc.so.6 {code} Unfortunately we didn't have DONT_OPTIMISE as a compilation option so I can't provide any better information at this time & this is the first instance of the issue we've seen so I can't be certain whether it might happen again. Interestingly on the same platform we're also occasionally deadlocking issues within Asterisk too which I'm trying to troubleshoot, however I'm not sure whether that might be related. Any help would be appreciated. | ||
Comments: | By: Matt Jordan (mjordan) 2014-08-17 18:01:26.053-0500 This may have been fixed by {{r410043}}, which was released in 1.8.27.0: {quote} moh: fix a refcount error with realtime MOH I observed a crash in res_musiconhold on an Asterisk 11 system using realtime MOH. Investigation of the backtrace showed a corrupt mohclass, implying that it got destroyed before the code expected it to. I went looking for reference counting errors that could have caused this crash and this patch this result. It contains 2 changes. 1) Remove a usless block of code that was impossible to reach. There was even a comment indicating that it was impossible to reach. The conditional includes "!ast_test_flag(global_flags, MOH_CACHERTCLASSES)" and it's inside of an if block with the opposite check "ast_test_flag(global_flags, MOH_CACHERTCLASSES)". There's no good reason to keep it around. 2) A similar block to #1 contained a reference counting error. It stores state->class in the local variable mohclass without increasing its reference count. The reference count on mohclass is decremented at the end of the function. This block of code probably very rarely runs, which would help explain why this system was working fine for many months before experiencing a crash. Review: https://reviewboard.asterisk.org/r/3282/ {quote} Try upgrading to the latest version of Asterisk 1.8. If you continue to experience this problem, you'll need to run Asterisk under valgrind to pinpoint the source of the memory corruption. By: Rusty Newton (rnewton) 2014-09-04 20:18:33.194-0500 This issue will now be closed, but should you be able to reproduce with a more recent version, then please re-open the issue. Thanks! |