[Home]

Summary:ASTERISK-19596: Memory Leak of 8gigs RAM.
Reporter:Jared Marcomb (jrad)Labels:
Date Opened:2012-03-26 13:18:10Date Closed:2017-12-13 09:00:04.000-0600
Priority:CriticalRegression?
Status:Closed/CompleteComponents:Core/BuildSystem
Versions:10.2.1 Frequency of
Occurrence
Frequent
Related
Issues:
Environment:CentOS6 * Intel Core i7-2600K Sandy Bridge * ASUS P8H67-I *PNY Optima 8GB 240-Pin DDR3 SDRAM DDR3 1333 * OCZ Agility 2 OCZSSD2Attachments:( 0) core.5137backtrace.txt
( 1) mmlog
( 2) modules.conf
Description:Asterisk v10.2.1 - Trying to locate a memory leak that brings down the service once a day during production work.

Will attach logs.
Comments:By: Matt Jordan (mjordan) 2012-03-27 11:19:15.385-0500

Please do the following:
1. In menuselect, under Compiler Flags, select MALLOC_DEBUG
2. Rebuild asterisk and install
3. Run as normal.  You should see a line on the Asterisk CLI that says:

Asterisk Malloc Debugger Started (see /var/log/asterisk/mmlog))

This is a log of the memory allocations... it can obviously grow rather large.  So you won't want to wait until the machine has run out of memory before stopping this.
4. Monitor the memory usage of Asterisk on the machine.  When you observe memory being consumed, you can use the CLI commands (only available when compiled with this option) "memory show summary" and "memory show allocations" to watch and see what is being allocated over time.  (NOTE: memory show allocations isn't nearly as useful)

Once you have an idea that Asterisk is indeed actively leaking memory, stop Asterisk and provide the CLI capture or logs demonstrating what is growing over time, as well as the mmlog generated by Asterisk.

By: Jared Marcomb (jrad) 2012-03-27 12:43:53.203-0500

Information requested through IRC.

By: Jared Marcomb (jrad) 2012-03-27 12:59:14.086-0500

I have uploaded requested docs from IRC.

It should be noted that running with the memory compiler flag as requested caused this core dump.

Unsure if there is an issue with the memory compiler flag and my system or part of the reason we have a memory leak in the first place.

By: Matt Jordan (mjordan) 2012-04-02 14:01:11.910-0500

Jared:

Looking at the backtrace, it appears as if you're using res_config_ldap.  It doesn't surprise me (although it is unfortunate) that MALLOC_DEBUG didn't handle that module well, given how the LDAP mods array is allocated and handled.  I took a look through res_config_ldap and, at first glance, it does not appear as if things are being leaked in that area of code - I would guess that the crash you had when you turned on MALLOC_DEBUG was not related to the memory leak you're experiencing.  Note that res_config_ldap is in extended support, which means bugs in it are typically handled by the Asterisk development community.

This makes it rather difficult to debug why you have a memory leak.  You may want to consider a different real time backend, and see if that alleviates the situation.  Alternatively, you may want to profile the machine and determine what operations are taking place when the memory consumption increases.  If you believe you've detected something the users have done and can document what they did, and collect a DEBUG log illustrating it, as well as the jump in memory usage, that could help point towards what in your system is leaking memory.

As a last resort, you could attempt to use valgrind, although in a production environment I wouldn't expect that to be a very simple task.

Matt

By: Andrew Latham (lathama) 2013-01-16 12:23:42.820-0600

Could be related to ASTERISK-17386

By: Sean Bright (seanbright) 2017-12-13 09:00:05.139-0600

This was most likely fixed by https://gerrit.asterisk.org/#/c/5065/ which was released in Asterisk 13.15.0