[Home]

Summary:ASTERISK-18265: CLONE - [patch] Memory Leak in app_queue
Reporter:Fred Schroeder (apten)Labels:
Date Opened:2011-08-12 01:27:11Date Closed:2011-08-23 13:37:42
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Applications/app_queue
Versions:1.8.5.0 Frequency of
Occurrence
Related
Issues:
is a clone ofASTERISK-15862 [patch] Memory Leak in app_queue
Environment:Attachments:( 0) 17081v2.patch
( 1) jira_asterisk_18265_v1.8_config.patch
( 2) jira_asterisk_18265_v1.8_q_ref_leaks.patch
( 3) memory_show_summary_after_queue_show.log
( 4) memory_show_summary_before_queue_show.log
( 5) queue_leak.patch
( 6) queue_mem_allocations.txt
( 7) queue_mem_check_output.txt
( 8) queue_mem_check.sh
( 9) valgrind_after_queue_show_with_queue_leak_patch.log
(10) valgrind_after_startup_with_queue_leak_patch.log
(11) valgrind_significant_change.log
(12) valgrind.txt
Description:Hello,
on a system with lots of realtime queues I have a problem with very high memory consumption in app_queue.
According to "memory show summary", there is a high number of allocations in
/usr/src/asterisk-1.6.0.26/include/asterisk/strings.h:390 (>5.000.000 allocations per Day) as well as high memory consumption in app_queue.c:1449 in only a few allocations (>1GB in 778 allocations).

This issue is simply reproducible by typing "queue show" several times in the cli and watch the allocation summary.
The more (realtime?)queues are configured the faster grows the memory consumption after each "queue show" command.

****** ADDITIONAL INFORMATION ******

This issue seems to be in all 1.6 Versions.
I also checked this issue with 1.6.0.3, 1.6.0.7, 1.6.0.25 and 1.6.1.18.
Comments:By: Fred Schroeder (apten) 2011-08-12 01:29:30.488-0500

The patch has not been applied to Asterisk 1.8.x branch / trunk. So I see the same memory allocation problem with Asterisk 1.8.5 too
Any chance that the patch will make it to this branch, too?

By: Mark Michelson (mmichelson) 2011-08-12 13:24:42.663-0500

The patch that was committed in the 1.6.x versions of Asterisk actually fixed two issues. One was not applicable to 1.8 but the other is. queue_leak.patch fixes the portion applicable to 1.8

By: Thomas Arimont (tomaso) 2011-08-15 07:07:33.064-0500

Mark,
o.k., the queue_leak.patch fixes one of the issues (the 'real' memory leak).
What about the other portion of the 17081v2.patch?
I assume it is fixed within the string field API implementation, isn'it?



By: Thomas Arimont (tomaso) 2011-08-17 09:43:31.436-0500

The results of our tests with your provided patch 'queue_leak.patch' looks better, but we do have still a significant memory leak when requesting queue informations using "queue show". In our specific environment/configuration we have a leak of approx. 680 kbytes (detected by 'ps -aux') without the patch, and approx. 164 kbytes when the patch was applied, after repeating 'queue show' about 60 times.

Please see attached valgrind leak-check informations, especially 'valgrind_significant_change.log'.
It seems to me that there is a memory-leak problem with realtime loading in general (not only 'load_realtime_queue' (app_queue.c) is involved, but also find_peer (chan_sip.c)).

Additional information: Buildhost and running host is a 64-Bit-Quadcore System with Scientific Linux 6 (redhat gcc 4.4.4-13). The running Asterisk is a non-debug version.

By: Michael Spiceland (mspiceland) 2011-08-18 17:15:29.855-0500

Thomas Arimont, Thank you for the valgrind info.  Can you please enable the MALLOC_DEBUG and DONT_OPTIMIZE compiler flags (using make menuselect).  You will then have a CLI command 'memory show summary' that you can run before and after.  This should help resolve some ambiguity and to track down the location of the leak.  Could you please do this and post the results?

By: Thomas Arimont (tomaso) 2011-08-19 08:06:40.892-0500

Here are the requested logs. You can see increasing allocations in module res_config_mysql.c, which would fit to the valgrind output.
The allocated objects in realtime_mysql() created by ast_variable_new()/ast_config_new() seems not to get deallocated (ast_variables_destroy(),ast_config_destroy()) again by the calling function load_realtime_queue() in app_queue.c.


By: Malcolm Davenport (mdavenport) 2011-08-22 09:38:25.419-0500

res_config_mysql is marked under extended support.  Support status of Asterisk components, including modules, is listed on the Wiki here:  https://wiki.asterisk.org/wiki/display/AST/Asterisk+Module+Support+States

Can the problem be reproduced using the res_config_odbc connector?"

By: Richard Mudgett (rmudgett) 2011-08-22 10:40:48.440-0500

[^jira_asterisk_18265_v1.8_config.patch] fixes a memory leak in ast_load_realtime().  It is the only leak I can see in the code path of queue show besides the [^queue_leak.patch].  The leak I found though does not seem to fit the leak characteristics indicated by the log files.

By: Richard Mudgett (rmudgett) 2011-08-22 14:30:26.213-0500

[^jira_asterisk_18265_v1.8_q_ref_leaks.patch] fixes some queue reference leaks that can be turned into memory leaks if the queue is ever removed from configuration.

By: Thomas Arimont (tomaso) 2011-08-23 12:13:21.857-0500

Richard,
with the latest patches (especially jira_asterisk_18265_v1.8_config.patch) I can not reproduce the mem leak anymore. Great!

I'm only a little bit confused about your comment 'The leak I found though does not seem to fit the leak characteristics indicated by the log files'? In my opinion my latest log file 'valgrind_significant_change.log' does fit to the leak you fixed in ast_load_realtime()/jira_asterisk_18265_v1.8_config.patch, doesn't it?  

By: Richard Mudgett (rmudgett) 2011-08-23 12:46:48.551-0500

Apparently it was how I was interpreting the valgrind loss report.  I was thinking that valgrind was indicating a loss of 52k as one unit or a chain of blocks lost at one time.  Valgrind indicated two such 52k losses.  The leak I found could only lose *one* ast_variable list node per call.

Anyway, I'll commit the remaining patch and close the issue.