Summary:	ASTERISK-26078: core: Memory leak in logging
Reporter:	Etienne Lessard (hexanol)	Labels:
Date Opened:	2016-05-31 14:58:16	Date Closed:	2016-06-01 16:13:02
Priority:	Major	Regression?	Yes
Status:	Closed/Complete	Components:	Core/Logging
Versions:	13.8.0 13.9.1	Frequency of Occurrence	Constant
Related Issues:	is duplicated by ASTERISK-26150 Memory leakage in stringfields.c and logger.c
Environment:	Debian 8 amd64	Attachments:	( 0) jira_asterisk_26078_v13.patch ( 1) mmlog ( 2) mmsummary1 ( 3) mmsummary2 ( 4) valgrind.txt
Description:	There seems to be a new memory leak since Asterisk 13.8 or 13.9. Running a simple call center scenario with sipp (a new call into a queue every few seconds, that is answered by a queue member, etc), I'm seeing asterisk memory usage grow noticeably (a little bit less than 100 MB/hour). Looking at a diff from two "memory show summary" commands taken 30 minutes apart, it looks like the leak comes from logger.c / stringfields.c. I've attached the output of two "memory show summary". I've also attached the mmlog file, but I doubt this will be useful since it's mostly empty (guess I've done something wrong). I'm already aware of ASTERISK-25262, but this is not it: I've explicitly disabled the CEL in my current tests. I've also seen ASTERISK-25905, but this is not it (I've also try manually running malloc_trim, it gave nothing). Maybe it's similar to ASTERISK-26020, but there's not much info in that ticket. Thank you
Comments:	By: Asterisk Team (asteriskteam) 2016-05-31 14:58:17.578-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Etienne Lessard (hexanol) 2016-05-31 15:03:00.368-0500 I also had a valgrind output from a previous run, so I'll also attach it even though I'm not sure if it helps (that said, we do see something related to logger.c and stringfields.c, e.g.): {code} ==11069== 1,409,504 (626,672 direct, 782,832 indirect) bytes in 39,167 blocks are definitely lost in loss record 9,148 of 9,151 ==11069==    at 0x402B0D5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==11069==    by 0x820ECF6: _ast_calloc (in /usr/sbin/asterisk) ==11069==    by 0x81F9203: calloc_wrapper (in /usr/sbin/asterisk) ==11069==    by 0x81F9CB2: __ast_calloc_with_stringfields (in /usr/sbin/asterisk) ==11069==    by 0x816328A: ast_log_full (in /usr/sbin/asterisk) ==11069==    by 0x8163ED8: ast_log_callid (in /usr/sbin/asterisk) ==11069==    by 0x8164235: __ast_verbose_ap (in /usr/sbin/asterisk) ==11069==    by 0x816429A: __ast_verbose (in /usr/sbin/asterisk) ==11069==    by 0x8191D33: pbx_extension_helper (in /usr/sbin/asterisk) ==11069==    by 0x8194DF6: ast_spawn_extension (in /usr/sbin/asterisk) ==11069==    by 0x8195A9F: __ast_pbx_run (in /usr/sbin/asterisk) ==11069==    by 0x81971CA: pbx_thread (in /usr/sbin/asterisk) {code} My tests were done on Asterisk 13.9.1. By: Etienne Lessard (hexanol) 2016-05-31 17:18:10.236-0500 It seems like the logmsg_free function in logger.c is missing a call to "ast_string_field_free_memory". I'm guessing this memory leak was a side effect of [commit f6f4cf459f43f072604927209b39646f84aaa2e2|https://github.com/asterisk/asterisk/commit/f6f4cf459f43f072604927209b39646f84aaa2e2]. I don't know if other parts of the source code are affected by a similar issue. By: Richard Mudgett (rmudgett) 2016-05-31 18:22:42.723-0500 The following cel_custom.c:free_config() cdr_syslog.c:free_config() cdr_custom.c:free_config() are also missing the ast_string_field_free_memory() call. By: Richard Mudgett (rmudgett) 2016-06-01 13:36:36.351-0500 [^jira_asterisk_26078_v13.patch] - This patch adds the missing ast_string_field_free_memory() call to logger.c, cel_custom.c, cdr_syslog.c, and cdr_custom.c. Please report if this takes care of the memory leak. By: Etienne Lessard (hexanol) 2016-06-01 13:52:47.070-0500 {quote} Please report if this takes care of the memory leak. {quote} The patch in logger.c does take care of the memory leak I was observing.