Summary:	ASTERISK-24592: Random crashes with preceding ERROR "Failed to encode JSON object"
Reporter:	Anton Evzhakov (Anber)	Labels:
Date Opened:	2014-12-03 13:59:11.000-0600	Date Closed:	2018-01-02 08:30:28.000-0600
Priority:	Major	Regression?
Status:	Closed/Complete	Components:	Core/Stasis
Versions:	12.7.1	Frequency of Occurrence	Frequent
Related Issues:	is related to ASTERISK-24590 Segmentation fault in get_multiple_fields_as_var_list
Environment:	Linux astratel 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux	Attachments:	( 0) app.log ( 1) ASTERISK-24592.backtrace.txt ( 2) ASTERISK-24592.log ( 3) debug_log_luke1 ( 4) debug_log_luke2 ( 5) debug_log_luke3
Description:	Asterisk randomly crashes on encode ARI events. Debug log and backtrace in attach.
Comments:	By: Matt Jordan (mjordan) 2014-12-03 14:09:13.399-0600 Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks! By: Anton Evzhakov (Anber) 2014-12-04 03:10:41.828-0600 It crashed on one in a thousand calls and I can't catch it via valgrind, because asterisk with valgrind works very slowly. Can i do something else? By: Richard Mudgett (rmudgett) 2014-12-04 16:07:48.221-0600 You can try {{MALLOC_DEBUG}} and see if anything shows up in the {{mmlog}} file.  The {{MALLOC_DEBUG}} menuselect option is a poor man's valgrind that doesn't reduce performance too much. https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag By: Rusty Newton (rnewton) 2014-12-19 09:20:02.502-0600 Anton we haven't heard from you in a couple weeks. Will you be able to get the {{MALLOC_DEBUG}} output? By: Anton Evzhakov (Anber) 2014-12-22 03:52:34.652-0600 I tried reproduce this bug on 12.8 with MALLOC_DEBUG, but I couldn't. I have many errors "ari/ari_websockets.c: Failed to encode JSON object" in log, but asterisk doesn't crash. Typical log before crash [2014-12-22 12:02:26.990] ERROR[19261] ari/ari_websockets.c: Failed to encode JSON object [2014-12-22 12:02:26.990] DEBUG[19240] ari/ari_websockets.c: Examining ARI event: {"application":"Lobby","type":"RecordingFinished","recording":{"state":"done","format":"wav","name":"TaskWithCustomer/e1aa3fd6-8ed9-411c-a66b-b0c0e5185e3b/2014/12/22/12_2_11-d5585b34-ab1b-4201-89e8-b2f0cedc598c","target_uri":"bridge:d5585b34-ab1b-4201-89e8-b2f0cedc598c"}} I think that problem reproduces when system sends message to two or more subscribers on very fast system (I can reproduce it on our production server with 24x Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz, but I can't on VPS). By: Matt Jordan (mjordan) 2014-12-22 13:18:49.713-0600 Can you provide a script and configuration to reproduce the problem? There are supposed to be locks around converting the JSON representation of an event to a string (and they are present in {{ast_json_dump_string_format}}), so whatever is causing the issue will take reproduction on a bug marshal's system. By: Anton Evzhakov (Anber) 2014-12-22 15:20:46.757-0600 I can provide a chain of queries and events which leads to the crash. Two apps and two channels: Lobby (with channel_A) and Incomings (with channel_B). In my case "Failed to encode JSON object" shows only for events RecordingStarted and RecordingFinished, and crashes only on message RecordingFinished. Application log in attach. By: Rusty Newton (rnewton) 2015-01-06 13:59:08.144-0600 Thanks. The application log is helpful, but after looking at it we'll still need some further help to track this down. Can you provide: * Another application log, *but* with a corresponding Asterisk full log (including DEBUG and VERBOSE messages, gathered as indicated in the [collecting debug instructions|https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information] ). It is important that all of the available logger channels be included and the verbosity of debug and verbose channels are turned up high, to five or above. * Your script or scripts making the calls and if possible a complete configuration environment for reproducing the issue. If anything, at least the scripts. By: Joshua C. Colp (jcolp) 2017-12-18 09:20:05.255-0600 Were you able to get the information that [~rnewton] mentioned and does this occur under Asterisk 13? By: Asterisk Team (asteriskteam) 2018-01-02 08:30:28.119-0600 Suspended due to lack of activity. This issue will be automatically re-opened if the reporter posts a comment. If you are not the reporter and would like this re-opened please create a new issue instead. If the new issue is related to this one a link will be created during the triage process. Further information on issue tracker usage can be found in the Asterisk Issue Guidlines [1]. [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines By: Lucas Tardioli Silveira (ltardioli) 2018-12-28 10:47:51.656-0600 Hey guys! I'm facing a similar issue. I'm attaching the logs I got, maybe it can be useful for the investigation. The version I'm using is: Asterisk certified/13.18-cert2 By: Lucas Tardioli Silveira (ltardioli) 2018-12-28 10:50:48.795-0600 Those are the logs I've got following the instrunctions I've found here: https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information By: Joshua C. Colp (jcolp) 2018-12-28 10:51:38.473-0600 [~ltardioli] That version is not supported. Please use a supported version of Asterisk and file a new issue with logs. Please note that if you have a support agreement then contact Digium support to file an issue, otherwise certified will not receive any fixes that may come about. By: Lucas Tardioli Silveira (ltardioli) 2018-12-28 10:56:40.881-0600 Okay! Thank you, Joshua! I'm gonna update it to the version 16. By: Lucas Tardioli Silveira (ltardioli) 2018-12-28 11:15:50.008-0600 Indeed it seems to be working fine on Asterisk 16. Thanks!