[Home]

Summary:ASTERISK-28926: core dump trying to free null channel snapshot
Reporter:Robert Sutton (rsutton@noojee.com.au)Labels:
Date Opened:2020-06-01 18:44:39Date Closed:
Priority:MinorRegression?
Status:Open/NewComponents:Core/Stasis
Versions:16.10.0 Frequency of
Occurrence		One Time
Related
Issues:
Environment:ubuntu 16.04, docker, pjsipAttachments:
Description:Asterisk core dumped.

This is the first usable core dump I've got, but probably the second occurrence in about 5 days of running on a system with about 4000 calls a day.

Here is the backtrace and the cli output immediately before the crash.

{noformat}

Core was generated by `/usr/sbin/asterisk -f -g -U asterisk -g -G shared -vvv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000005be436 in reset_field (p=0x0) at stringfields.c:83
[Current thread is 1 (Thread 0x7f3721b83700 (LWP 22344))]
#0  0x00000000005be436 in reset_field (p=0x0) at stringfields.c:83
No locals.
#1  0x00000000005be493 in __ast_string_field_free_memory (mgr=0x7f3778cee7e8, pool_head=0x7f3778cee728, cleanup_type=AST_STRINGFIELD_DESTROY, file=0x69fa59 "stasis_channels.c", lineno=226, func=0x6a0150 <__PRETTY_FUNCTION__.15295> "channel_snapshot_dtor") at stringfields.c:111
       idx = 18
       cur = 0x0
       preserve = 0x0
       __PRETTY_FUNCTION__ = "__ast_string_field_free_memory"
#2  0x00000000005ab945 in channel_snapshot_dtor (obj=0x7f3778cee728) at stasis_channels.c:226
       __res__ = -1
       snapshot = 0x7f3778cee728
       __PRETTY_FUNCTION__ = "channel_snapshot_dtor"
#3  0x0000000000462a41 in __ao2_ref (user_data=0x7f3778cee728, delta=-1, tag=0x6a0b81 "", file=0x6a0b70 "stasis_message.c", line=137, func=0x6a0bd0 <__PRETTY_FUNCTION__.9493> "stasis_message_dtor") at astobj2.c:614
       obj = 0x7f3778cee710
       obj_mutex = 0xa
       obj_rwlock = 0x1
       obj_lockobj = 0xe
       current_value = 0
       ret = 1
       weakproxy = 0x0
       lock_state = 0x0
       __PRETTY_FUNCTION__ = "__ao2_ref"
#4  0x0000000000462d81 in __ao2_cleanup_debug (obj=0x7f3778cee728, tag=0x6a0b81 "", file=0x6a0b70 "stasis_message.c", line=137, function=0x6a0bd0 <__PRETTY_FUNCTION__.9493> "stasis_message_dtor") at astobj2.c:670
No locals.
#5  0x00000000005b3f59 in stasis_message_dtor (obj=0x7f37784b4a58) at stasis_message.c:137
       message = 0x7f37784b4a58
       __PRETTY_FUNCTION__ = "stasis_message_dtor"
#6  0x0000000000462a41 in __ao2_ref (user_data=0x7f37784b4a58, delta=-1, tag=0x69f56f "", file=0x69f560 "stasis_cache.c", line=787, func=0x69f8c0 <__PRETTY_FUNCTION__.9773> "stasis_cache_update_dtor") at astobj2.c:614
       obj = 0x7f37784b4a40
       obj_mutex = 0x666fa0 <__PRETTY_FUNCTION__.8509>
       obj_rwlock = 0x666d3a
       obj_lockobj = 0xf10650
       current_value = 0
       ret = 1
       weakproxy = 0x0
       lock_state = 0x78a9b298 <error: Cannot access memory at address 0x78a9b298>
       __PRETTY_FUNCTION__ = "__ao2_ref"
#7  0x0000000000462d81 in __ao2_cleanup_debug (obj=0x7f37784b4a58, tag=0x69f56f "", file=0x69f560 "stasis_cache.c", line=787, function=0x69f8c0 <__PRETTY_FUNCTION__.9773> "stasis_cache_update_dtor") at astobj2.c:670
No locals.
#8  0x00000000005aa255 in stasis_cache_update_dtor (obj=0x7f3778352f88) at stasis_cache.c:787
       update = 0x7f3778352f88
       __PRETTY_FUNCTION__ = "stasis_cache_update_dtor"
#9  0x0000000000462a41 in __ao2_ref (user_data=0x7f3778352f88, delta=-1, tag=0x6a0b81 "", file=0x6a0b70 "stasis_message.c", line=137, func=0x6a0bd0 <__PRETTY_FUNCTION__.9493> "stasis_message_dtor") at astobj2.c:614
       obj = 0x7f3778352f70
       obj_mutex = 0x7f3710feeb80
       obj_rwlock = 0x0
       obj_lockobj = 0x666a62
       current_value = 0
       ret = 1
       weakproxy = 0x0
       lock_state = 0x6adc50 <__PRETTY_FUNCTION__.16256> "check_retire_linkedid"
       __PRETTY_FUNCTION__ = "__ao2_ref"
#10 0x0000000000462d81 in __ao2_cleanup_debug (obj=0x7f3778352f88, tag=0x6a0b81 "", file=0x6a0b70 "stasis_message.c", line=137, function=0x6a0bd0 <__PRETTY_FUNCTION__.9493> "stasis_message_dtor") at astobj2.c:670
No locals.
#11 0x00000000005b3f59 in stasis_message_dtor (obj=0x7f3778275bc8) at stasis_message.c:137
       message = 0x7f3778275bc8
       __PRETTY_FUNCTION__ = "stasis_message_dtor"
#12 0x0000000000462a41 in __ao2_ref (user_data=0x7f3778275bc8, delta=-1, tag=0x69dca9 "", file=0x69dca0 "stasis.c", line=1260, func=0x69e5f0 <__PRETTY_FUNCTION__.15985> "dispatch_exec_async") at astobj2.c:614
       obj = 0x7f3778275bb0
       obj_mutex = 0x5f994a <cel_snapshot_update_cb+186>
       obj_rwlock = 0x7f3778275bc8
       obj_lockobj = 0x7f3778275bc8
       current_value = 0
       ret = 1
       weakproxy = 0x0
       lock_state = 0x5f97fb <cel_channel_app_change+92> "\205\300u&H\213U\350H\213E\370A\271"
       __PRETTY_FUNCTION__ = "__ao2_ref"
#13 0x0000000000462d81 in __ao2_cleanup_debug (obj=0x7f3778275bc8, tag=0x69dca9 "", file=0x69dca0 "stasis.c", line=1260, function=0x69e5f0 <__PRETTY_FUNCTION__.15985> "dispatch_exec_async") at astobj2.c:670
No locals.
#14 0x00000000005a0fa0 in dispatch_exec_async (local=0x7f3721b82d90) at stasis.c:1260
       sub = 0xf138a0
       message = 0x7f3778275bc8
       __PRETTY_FUNCTION__ = "dispatch_exec_async"
#15 0x00000000005c5720 in ast_taskprocessor_execute (tps=0xf13a80) at taskprocessor.c:1235
       local = {local_data = 0xf138a0, data = 0x7f3778275bc8}
       t = 0x7f37789ff920
       size = 1
       __PRETTY_FUNCTION__ = "ast_taskprocessor_execute"
#16 0x00000000005c2755 in default_tps_processing_function (data=0xf13a20) at taskprocessor.c:209
       listener = 0xf13a20
       tps = 0xf13a80
       pvt = 0xf139a0
       sem_value = 4550509
       res = 0
       __PRETTY_FUNCTION__ = "default_tps_processing_function"
#17 0x00000000005d83b7 in dummy_start (data=0xf13b60) at utils.c:1249
       __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {0, 1758957914214798609, 0, 140730941917087, 139874765650368, 0, 1758957914189632785, -1871437870580984559}, __mask_was_saved = 0}}, __pad = {0x7f3721b82ef0, 0x0, 0x0, 0x0}}
       __cancel_routine = 0x456f72 <ast_unregister_thread>
       __cancel_arg = 0x7f3721b83700
       __not_first_call = 0
       ret = 0x0
       a = {start_routine = 0x5c26b0 <default_tps_processing_function>, data = 0xf13a20, name = 0xf13930 "default_tps_processing_function started at [  226] taskprocessor.c default_listener_start()"}
       __PRETTY_FUNCTION__ = "dummy_start"
#18 0x00007f37858cd6ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#19 0x00007f3784b6241d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

{noformat}

{noformat}

2020-06-01T03:29:34.421805155Z     -- PJSIP/156-00001c44 answered
2020-06-01T03:29:34.421899492Z     -- Executing [activity-agi@noojee-activity:1] Set("PJSIP/156-00001c44", "AGIEXITONHANGUP=yes") in new stack
2020-06-01T03:29:34.421973608Z     -- Executing [activity-agi@noojee-activity:2] AGI("PJSIP/156-00001c44", "agi://127.0.0.1/activityAgi") in new stack
2020-06-01T03:29:34.424289633Z     -- Started music on hold, class 'default', on channel 'PJSIP/156-00001c44'
2020-06-01T03:29:34.427812487Z     -- Stopped music on hold on PJSIP/156-00001c44
2020-06-01T03:29:34.437148571Z     -- Channel PJSIP/trunk-00001c22 left 'simple_bridge' basic-bridge <0cafa268-e000-4f47-97b9-346d7d24b3b5>
2020-06-01T03:29:34.437999967Z   == MixMonitor close filestream (mixed)
2020-06-01T03:29:34.458724354Z   == Executing [curl 'http://127.0.0.1:8080/migrate?guid=1590982116181-39577&apiKey=xxxxxx']
2020-06-01T03:29:34.458747553Z     -- Channel PJSIP/439-00001c28 left 'simple_bridge' basic-bridge <0cafa268-e000-4f47-97b9-346d7d24b3b5>
2020-06-01T03:29:34.464649163Z     -- Manager 'njcontact' from 127.0.0.1, hanging up channel: PJSIP/439-00001c28
2020-06-01T03:29:34.465632969Z   == Spawn extension (noojee-activity, activity-agi, 2) exited non-zero on 'PJSIP/439-00001c28'
2020-06-01T03:29:34.466225994Z   == MixMonitor close filestream (mixed)
2020-06-01T03:29:34.478103254Z   == Executing [curl 'http://127.0.0.1:8080/migrate?guid=1590982150337-57634&apiKey=xxxxxxx']
2020-06-01T03:29:34.501728181Z   == Begin MixMonitor Recording PJSIP/trunk-00001c36
2020-06-01T03:29:34.501751500Z     -- AGI Script Executing Application: (bridge) Options: (PJSIP/trunk-00001c36,x)
2020-06-01T03:29:34.503194668Z     -- Stopped music on hold on PJSIP/trunk-00001c36
2020-06-01T03:29:34.503693399Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-06-01T03:29:34.503825442Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-06-01T03:29:34.506482069Z   == Spawn extension (routesv2-inbound, 61xxxxxxx, 3) exited non-zero on 'Surrogate/PJSIP/trunk-00001c36'
2020-06-01T03:29:34.507290851Z     -- Channel PJSIP/trunk-00001c36 joined 'simple_bridge' basic-bridge <b9deaef7-b11b-4b96-b7c0-965367873941>
2020-06-01T03:29:34.509225551Z     -- Channel PJSIP/156-00001c44 joined 'simple_bridge' basic-bridge <b9deaef7-b11b-4b96-b7c0-965367873941>
2020-06-01T03:29:34.521695983Z   == MixMonitor close filestream (mixed)
 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-06-01T03:29:34.533300587Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-06-01T03:29:34.538066243Z   == Executing [curl 'http://127.0.0.1:8080//migrate?guid=1590982174482-69989&apiKey=key']
2020-06-01T03:29:34.575074690Z   == Begin MixMonitor Recording PJSIP/trunk-00001c36
 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-06-01T03:29:34.590945947Z                                  Dload  Upload   Total   Spent    Left  Speed
100     2  100     2    0     0      3      0 --:--:-- --:--:-- --:--:--     3
100     2  100     2    0     0      3      0 --:--:-- --:--:-- --:--:--     3
2020-06-01T03:29:35.036287693Z OKOK  == End MixMonitor Recording PJSIP/trunk-00001c22
2020-06-01T03:29:35.037034104Z   == End MixMonitor Recording PJSIP/trunk-00001c36
100     2  100     2    0     0      3      0 --:--:-- --:--:-- --:--:--     3
2020-06-01T03:29:35.094448559Z OK  == End MixMonitor Recording PJSIP/trunk-00001c36
2020-06-01T03:29:36.358127323Z     -- AGI Script Executing Application: (StopPlayTones) Options: ()
2020-06-01T03:29:36.381212263Z     -- AGI Script Executing Application: (StopMixMonitor) Options: ()
2020-06-01T03:29:36.447050513Z     -- Called 203
2020-06-01T03:29:36.482589523Z     -- AGI Script Executing Application: (StopMixMonitor) Options: ()
2020-06-01T03:29:36.482932885Z     -- AGI Script Executing Application: (mixmonitor) Options: (/var/spool/asterisk/monitor/1590982170.9688-0448783681-0385086542-I-1.wav,,curl 'http://127.0.0.1:8080/migrate?guid=1590982176481-71198&apiKey=key')
2020-06-01T03:29:36.483176325Z   == Begin MixMonitor Recording PJSIP/trunk-00001c43
2020-06-01T03:29:36.500968719Z     -- Started music on hold, class 'queue', on channel 'PJSIP/trunk-00001c43'
2020-06-01T03:29:36.718278032Z     -- PJSIP/203-00001c45 is ringing
2020-06-01T03:29:36.718913372Z     -- PJSIP/203-00001c45 is ringing
2020-06-01T03:29:37.139635788Z     -- PJSIP/204-00001c42 answered
2020-06-01T03:29:37.139671394Z     -- Executing [activity-agi@noojee-activity:1] Set("PJSIP/204-00001c42", "AGIEXITONHANGUP=yes") in new stack
2020-06-01T03:29:37.140416679Z     -- Executing [activity-agi@noojee-activity:2] AGI("PJSIP/204-00001c42", "agi://127.0.0.1/activityAgi") in new stack
2020-06-01T03:29:37.147429692Z   == MixMonitor close filestream (mixed)
2020-06-01T03:29:37.162755901Z   == Executing [curl 'http://127.0.0.1:8080/rest/recordingMigration/migrate?guid=1590982164163-64347&apiKey=xxxxx']
2020-06-01T03:29:37.189195159Z     -- Stopped music on hold on PJSIP/trunk-00001c3b
2020-06-01T03:29:37.189217087Z   == Spawn extension (routesv2-inbound, 61xxxxxxxxx, 3) exited non-zero on 'PJSIP/trunk-00001c3b'
2020-06-01T03:29:37.190307657Z [Jun  1 13:29:37] WARNING[2190]: app_mixmonitor.c:988 launch_monitor_thread: Unable to add 'MixMonitor' spy to channel 'PJSIP/trunk-00001c3b'
2020-06-01T03:29:37.191300323Z     -- AGI Script Executing Application: (bridge) Options: (PJSIP/trunk-00001c3b,x)
2020-06-01T03:29:37.192167730Z *** Error in `/usr/sbin/asterisk': munmap_chunk(): invalid pointer: 0x00007f3778cee7d0 ***
2020-06-01T03:29:37.211586771Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-06-01T03:29:37.211612013Z                                  Dload  Upload   Total   Spent    Left  Speed
100     2  100     2    0     0      3      0 --:--:-- --:--:-- --:--:--     3
2020-06-01T03:29:39.252571941Z OK2020-06-01T03:29:40.153796408Z Asterisk 16.10.0, Copyright (C) 1999 - 2018, Digium, Inc. and others.
2020-06-01T03:29:40.153838211Z Created by Mark Spencer <markster@digium.com>
2020-06-01T03:29:40.153847253Z Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
2020-06-01T03:29:40.153855036Z This is free software, with components licensed under the GNU General Public
2020-06-01T03:29:40.153862849Z License version 2 and other licenses; you are welcome to redistribute it under
2020-06-01T03:29:40.153870603Z certain conditions. Type 'core show license' for details.
2020-06-01T03:29:40.153878195Z =========================================================================

{noformat}


Comments:By: Asterisk Team (asteriskteam) 2020-06-01 18:44:40.148-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.
By: George Joseph (gjoseph) 2020-06-02 08:23:08.207-0500

Robert, If you still have the actual coredump, can you run ast_coredumper on it with the following options...

{{ast_coredumper --tarball-coredumps --no-default-search <path_to_codedump>}}

The tarball will be large and contain sensitive info so don't attach it here.  Instead upload it to the hosting service of your choice, like Google Drive, DropBox, etc. and email the link to asteriskteam@digium.com with a subject of "ASTERISK-28926: Coredump".

By: Robert Sutton (rsutton@noojee.com.au) 2020-06-03 07:17:54.156-0500

I've emailed the link to the core dump as requested.