| Summary: | ASTERISK-27338: Excessive refcount 100000 reached on ao2 object | ||||
| Reporter: | Abhay Gupta (agupta) | Labels: | |||
| Date Opened: | 2017-10-13 06:25:57 | Date Closed: | 2019-04-17 15:34:53 | ||
| Priority: | Major | Regression? | |||
| Status: | Closed/Complete | Components: | Bridges/bridge_softmix | ||
| Versions: | 13.17.2 | Frequency of Occurrence | |||
| Related Issues: |
| ||||
| Environment: | Ubuntu 16.04 with Asterisk 13.17.2 using ARI and bridge for mixing | Attachments: | ( 0) frack.log | ||
| Description: | System crash with
[Oct 12 14:32:07] ERROR[21298][C-0001ad35] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x557464afd838 [Oct 12 14:32:07] WARNING[7128][C-0001aabd] channel.c: Exceptionally long queue length queuing to Local/agentmanual@xyz-00017532;1 [Oct 12 14:32:07] ERROR[21298][C-0001ad35] astobj2.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x557464afd838 | ||||
| Comments: | By: Asterisk Team (asteriskteam) 2017-10-13 06:25:58.959-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Abhay Gupta (agupta) 2017-10-13 06:27:54.656-0500 bt , bt bull and thread apply all bt full of the crash . #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58 #1 0x00007f7663ad53ea in __GI_abort () at abort.c:89 #2 0x00007f7663b170d0 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f7663c2cf80 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f7663b20c9a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7f7663c2d090 "double free or corruption (out)", action=3) at malloc.c:5048 #4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3904 #5 0x00007f7663b24d8c in __GI___libc_free (mem=<optimized out>) at malloc.c:2984 #6 0x0000557463e35acd in __ast_bt_get_symbols (addresses=0x7f740dfffeb0, num_frames=13) at backtrace.c:175 #7 0x0000557463f24217 in ast_log_backtrace () at logger.c:2053 #8 0x0000557463ffb8b9 in __ast_assert_failed (condition=0, condition_str=0x7f74adf86bc0 "Excessive refcount 100000 reached on ao2 object 0x557464afd838", file=0x55746400e47b "astobj2.c", line=518, function=0x55746400e718 <__FUNCTION__.8898> "__ao2_ref") at utils.c:2473 #9 0x0000557463e2a0d9 in internal_ao2_ref (user_data=0x557464afd838, delta=1, file=0x55746400e47b "astobj2.c", line=518, func=0x55746400e718 <__FUNCTION__.8898> "__ao2_ref") at astobj2.c:435 #10 0x0000557463e2a463 in __ao2_ref (user_data=0x557464afd838, delta=1) at astobj2.c:518 #11 0x0000557463f030db in ast_frdup (f=0x7f75f00860b8) at frame.c:343 #12 0x0000557463e5279e in ast_bridge_channel_queue_frame (bridge_channel=0x7f760835bc10, fr=0x7f75f00860b8) at bridge_channel.c:989 #13 0x00007f759edcd335 in softmix_mixing_loop (bridge=0x7f73da33ac80) at bridge_softmix.c:1041 #14 0x00007f759edcd79e in softmix_mixing_thread (data=0x7f75f001a5e0) at bridge_softmix.c:1115 #15 0x0000557463ff834f in dummy_start (data=0x7f75f002a720) at utils.c:1238 #16 0x00007f766490f6ca in start_thread (arg=0x7f74adf8b700) at pthread_create.c:333 #17 0x00007f7663ba6caf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105 By: Richard Mudgett (rmudgett) 2017-10-13 11:11:39.572-0500 The backtrace isn't too helpful because it shows the ao2 object has to be a format and formats are referenced in a lot of places. Certain formats like slin, ulaw, and alaw can get a lot of references normally. I imagine the ao2 object that reached the excessive ref count is the slin format since the code was trying to write a frame toward a softmix bridge channel after mixing audio. About how many channels did you have in confbridge/softmix bridges at the time? Do many normal calls on the system have to do format translations? How is the system used? By: Abhay Gupta (agupta) 2017-10-13 11:22:08.693-0500 The system is used as predictive dialer and at any time not more then 100 channels are in softbridge. Since the codec used is ulaw and recording is also happening so slin must be getting used for recording. By: Steven Sedory (stevensedory) 2017-10-13 19:48:03.541-0500 Are you running this server as a VM on KVM by chance? If so, are you using proxmox? By: Abhay Gupta (agupta) 2017-10-13 19:50:37.644-0500 No it is a dedicated standalone server By: George Joseph (gjoseph) 2017-10-23 08:26:10.675-0500 Was this a single occurrence of the crash or is it continuing to happen? By: Abhay Gupta (agupta) 2017-10-25 08:15:56.868-0500 This issue has not repeated By: Kevin Harwell (kharwell) 2017-10-25 12:22:42.616-0500 Thanks for the report! Unfortunately at this time we do not have enough information to move forward with this issue. Please provide an exact description of the scenario you were running when the problem occurred. The executing dialplan and config files involved are also required. Information on the usage of your system would also be helpful. By: Asterisk Team (asteriskteam) 2017-11-08 12:00:01.074-0600 Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1]. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines | ||||