| Summary: | ASTERISK-28947: Segmentation fault in mixmonitor_ds_destroy | ||||
| Reporter: | Robert Sutton (rsutton@noojee.com.au) | Labels: | |||
| Date Opened: | 2020-06-11 23:03:49 | Date Closed: | 2021-01-06 16:48:06.000-0600 | ||
| Priority: | Major | Regression? | |||
| Status: | Closed/Complete | Components: | Applications/app_mixmonitor | ||
| Versions: | 16.10.0 | Frequency of Occurrence | Occasional | ||
| Related Issues: |
| ||||
| Environment: | ubuntu 18.04, asterisk 16.10, pjsip | Attachments: | |||
| Description: | This has occurred twice in about 10 days.
{noformat} Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". bCore was generated by `/usr/sbin/asterisk -f -g -U asterisk -g -G shared -vvv'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI_abort () at abort.c:107 107 abort.c: No such file or directory. [Current thread is 1 (Thread 0x7f88d8855700 (LWP 14549))] (gdb) bt #0 __GI_abort () at abort.c:107 #1 0x00007f89f6454897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f89f6581b9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181 #2 0x00007f89f645b90a in malloc_printerr (str=str@entry=0x7f89f65837a8 "munmap_chunk(): invalid pointer") at malloc.c:5350 #3 0x00007f89f6462ecc in munmap_chunk (p=0x7f88ec01a700) at malloc.c:2846 #4 __GI___libc_free (mem=0x7f88ec01a710) at malloc.c:3117 #5 0x0000561aa218d2dd in __ast_free (ptr=0x7f88ec01a710, file=0x7f89ab9f6185 "app_mixmonitor.c", lineno=454, func=0x7f89ab9f7230 <__PRETTY_FUNCTION__.15608> "mixmonitor_ds_destroy") at astmm.c:1588 #6 0x00007f89ab9eeea5 in mixmonitor_ds_destroy (data=0x7f88ec062190) at app_mixmonitor.c:454 #7 0x0000561aa220f8d4 in ast_datastore_free (datastore=0x7f88ec00b250) at datastore.c:74 #8 0x0000561aa21d1d54 in ast_channel_destructor (obj=0x7f89c0ac4530) at channel.c:2243 #9 0x0000561aa218eaea in __ao2_ref (user_data=0x7f89c0ac4530, delta=-1, tag=0x7f89ab9f64c5 "", file=0x7f89ab9f6185 "app_mixmonitor.c", line=1309, func=0x7f89ab9f73f0 <__PRETTY_FUNCTION__.16078> "handle_cli_mixmonitor") at astobj2.c:614 #10 0x00007f89ab9f4955 in handle_cli_mixmonitor (e=0x7f89abbf8440 <cli_mixmonitor>, cmd=-4, a=0x7f88d8853e20) at app_mixmonitor.c:1309 #11 0x0000561aa21ffb3e in ast_cli_command_full (uid=-1, gid=-1, fd=285, s=0x7f88ec027479 "mixmonitor start PJSIP/trunk-0000372B \"/var/spool/asterisk/monitor/1591853283.17660-xxxxxxx-xxxxxxxxx-I-2.wav\",'',\"curl 'http://127.0.0.1:8080/rest/recordingMigration/migrate?guid=1591853314407-9"...) at cli.c:2833 #12 0x0000561aa2372d70 in action_command (s=0x7f88d8854c40, m=0x7f88d88543d0) at manager.c:5241 #13 0x0000561aa237903b in process_message (s=0x7f88d8854c40, m=0x7f88d88543d0) at manager.c:6612 #14 0x0000561aa2379a5d in do_message (s=0x7f88d8854c40) at manager.c:6825 #15 0x0000561aa2379f0f in session_do (data=0x7f89700094d0) at manager.c:6930 #16 0x0000561aa2301f6b in handle_tcptls_connection (data=0x7f89700094d0) at tcptls.c:274 #17 0x0000561aa23145e2 in dummy_start (data=0x7f8970004d80) at utils.c:1249 #18 0x00007f89f72fb6db in start_thread (arg=0x7f88d8855700) at pthread_create.c:463 #19 0x00007f89f64ec88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 {noformat} | ||||
| Comments: | By: Asterisk Team (asteriskteam) 2020-06-11 23:03:50.884-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: Sean Bright (seanbright) 2020-06-12 15:25:12.583-0500 Did you not have this problem with 16.9? By: Robert Sutton (rsutton@noojee.com.au) 2020-06-15 00:06:04.205-0500 This is system has not run any prior versions of asterisk 16. By: Kevin Harwell (kharwell) 2020-06-15 16:46:48.165-0500 It seems the channel (PJSIP/trunk-0000372B) starting the mixmonitor goes away (hungup?). Do you know what kind of call scenario is happening at the time? For instance, a basic call Alice calls Bob, or is this a channel spying on another, etc...? Also can you attach the log from an occurrence? Anything of note (error or warning) in it before the crash? By: Robert Sutton (rsutton@noojee.com.au) 2020-06-15 20:12:25.659-0500 Sorry the logs got rolled due to an unrelated crash. I've subsequently moved to logging that will persist. I still have the core dump, but the logs are lost. From what I recall about the logs there was nothing interesting in the logs prior. I can't say what this call was doing, but 90+% of the calls on this system are inbound. They enter an AGI which simulates queuing, playing messages and hold music. The agent channel is originated into dialplan via manager and then into AGI where the agent channel is bridged to the inbound call. It is quite common for the channel to be subsequently transferred (attended) to a 3rd external party. By: Kevin Harwell (kharwell) 2020-06-16 17:13:22.903-0500 Since you still have the core file can you run the ast_coredumper with the following options against it: {noformat} $ /var/lib/asterisk/scripts/ast_coredumper --tarball-coredumps --no-default-search <path_to_coredump> {noformat} The resulting tarball will be too large to attach here. Instead upload it to the provider of your choice (for instance Google Drive, DropBox, etc... somewhere we can retrieve it) and email the link to asteriskteam@digium.com with the following subject: ASTERISK-28947 - Coredumps. Thanks! By: Kevin Harwell (kharwell) 2020-06-17 15:15:06.790-0500 Alas I was able to replicate this issue. This happens when the channel being monitored is hung up after the datastore has been setup, but prior to monitor starting itself. It's a narrow window, but if you force a sleep just before the call to _startmon_ in the _launch_monitor_thread_ function in app_mixmonitor.c then you can give yourself time to hangup the channel after starting the mixmonitor from the CLI. So steps I took: 1) alter app_mixmonitor with something like the following: {noformat} diff --git a/apps/app_mixmonitor.c b/apps/app_mixmonitor.c index 09148ccf1d..4e54052454 100644 --- a/apps/app_mixmonitor.c +++ b/apps/app_mixmonitor.c @@ -984,6 +985,7 @@ static int launch_monitor_thread(struct ast_channel *chan, const char *filename, if (writevol) mixmonitor->audiohook.options.write_volume = writevol; + while (sleep(5) > 0); if (startmon(chan, &mixmonitor->audiohook)) { ast_log(LOG_WARNING, "Unable to add '%s' spy to channel '%s'\n", mixmonitor_spy_type, ast_channel_name(chan)); {noformat} 2) Called into the Echo application from an endpoint 3) Started mixmonitor from the CLI: {noformat} *CLI> mixmonitor start <your channel name> /tmp/test.wav {noformat} 4) hung up the channel on the endpoint 5) Crash | ||||