[Home]

Summary:ASTERISK-23070: Memory leak when using CLI through AMI
Reporter:Denis (den159)Labels:
Date Opened:2013-12-26 10:01:58.000-0600Date Closed:2014-04-21 11:43:47
Priority:CriticalRegression?
Status:Closed/CompleteComponents:Core/ManagerInterface
Versions:12.0.0 Frequency of
Occurrence
Related
Issues:
Environment:Centos 6.5 on Oracle VM VirtualBoxAttachments:( 0) extconfig.conf
( 1) manager.conf
Description:Calling CLI commands through AMI interface causes memory leak (sending AMI actions does not cause leak), each call CLI command through AMI increase RAM buffer by 8 kb. I am using Astxx lib (by Matthew A. Nicholson http://matt-land.com/astxx/) as AMI caller, memory usage increases when executed
action::command command(str_cmd);
command(*connectionn);
Comments:By: Matt Jordan (mjordan) 2013-12-30 10:05:22.308-0600

I think we need some more information to be able to reproduce this accurately.

Which CLI commands are you executing? For those CLI commands, what data is returned?

Please also attach a sanitized {{manager.conf}}.
By: Denis (den159) 2014-01-02 05:37:06.456-0600

manager conf in realtime MySQL table:
manager.conf general enabled yes
manager.conf general port 5038
manager.conf general bindaddr 0.0.0.0
manager.conf manager secret xxxxxx
manager.conf manager deny 0.0.0.0/0.0.0.0
manager.conf manager permit 192.168.1.0/255.255.255.0
manager.conf manager read system,call,command,agent,user,dtmf,reporting,originate,agi,cc,aoc
manager.conf manager write system,call,command,agent,user,dtmf,reporting,originate,agi,cc,aoc
manager.conf manager eventfilter Event: Registry
manager.conf manager eventfilter Event: PeerStatus
manager.conf manager eventfilter Event: Dial
manager.conf manager eventfilter Event: DialBegin
manager.conf manager eventfilter Event: DialEnd
manager.conf manager eventfilter Event: Newstate
manager.conf manager eventfilter Event: NewCallerid
manager.conf manager eventfilter Event: Newchannel
manager.conf manager eventfilter Event: PresenceStatus
manager.conf manager eventfilter Event: Hangup

In short on base of above mentioned Astxx, i have compiled .dll for third party windows application 1C, I also tried compile the same .dll on pure WinSock2 in both cases memory leak presents. Active using AMI with 4 clients, increase RAM buffer for ~3 hours from ~8400 kb to ~20000 kb, below you can see details:

system booted
[root@localhost ~]# free
            total       used       free     shared    buffers     cached
Mem:       1030584     166060     864524          0       8384      89516

then registered trunk, registered peer, connect and login AMI manager, and now we have

Mem:       1030584     166372     864212          0       8444      89616

it's OK, and now trying to send CLI over AMI

sent CLI command: core show settings

recived:

PBX Core settings
-----------------
 Version:                     12.0.0
 Build Options:               LOADABLE_MODULES, BUILD_NATIVE, OPTIONAL_API
 Maximum calls:               Not set
 Maximum open file handles:   Not set
 Verbosity:                   3
 Debug level:                 0
 Maximum load average:        0.000000
 Minimum free memory:         0 MB
 Startup time:                16:45:52
 Last reload time:            16:45:52
 System:                      Linux/2.6.32-431.1.2.0.1.el6.i686 built by root on i686 2013-12-31 08:53:35 UTC
 System name:                
 Entity ID:                   08:00:27:15:3e:e7
 Default language:            en
 Language prefix:             Enabled
 User name and group:         /
 Executable includes:         Disabled
 Transcode via SLIN:          Enabled
 Internal timing:             Enabled
 Transmit silence during rec: Disabled
 Generic PLC:                 Enabled
 Min DTMF duration::          80

* Subsystems
 -------------
 Manager (AMI):               Enabled
 Web Manager (AMI/HTTP):      Disabled
 Call data records:           Disabled
 Realtime Architecture (ARA): Enabled

* Directories
 -------------
 Configuration file:          
 Configuration directory:     /etc/asterisk
 Module directory:            /usr/lib/asterisk/modules
 Spool directory:             /var/spool/asterisk
 Log directory:               /var/log/asterisk
 Run/Sockets directory:       /var/run/asterisk
 PID file:                    /var/run/asterisk/asterisk.pid
 VarLib directory:            /var/lib/asterisk
 Data directory:              /var/lib/asterisk
 ASTDB:                       /var/lib/asterisk/astdb
 IAX2 Keys directory:         /var/lib/asterisk/keys
 AGI Scripts directory:       /var/lib/asterisk/agi-bin

now memory state
            total       used       free     shared    buffers     cached
Mem:       1030584     166400     864184          0       8452      89616

repeat this exercise 5 times over ~3 sec. pause, now memory state
            total       used       free     shared    buffers     cached
Mem:       1030584     166536     864048          0       8500      89652

logoff and disconnect AMI manager for 10 minutes...
...now connect and login AMI manager

now memory state
            total       used       free     shared    buffers     cached
Mem:       1030584     167032     863552          0       8564      89684

10 minutes do nothing...
...10 minutes have passed, during which nothing happens but we recieved several event messages:

Event: Registry
ChannelType: SIP
Domain: qwerty.cnt.ru
Privilege: system,all
Status: Registered
Username: xxxxxxxxxx


Event: PeerStatus
Address: 192.168.1.25:5065
ChannelType: SIP
Peer: SIP/101
PeerStatus: Registered
Privilege: system,all


Event: Registry
ChannelType: SIP
Domain: qwerty.cnt.ru
Privilege: system,all
Status: Registered
Username: xxxxxxxxxx


Event: Registry
ChannelType: SIP
Domain: qwerty.cnt.ru
Privilege: system,all
Status: Registered
Username: xxxxxxxxxx


Event: Registry
ChannelType: SIP
Domain: qwerty.cnt.ru
Privilege: system,all
Status: Registered
Username: xxxxxxxxxx


Event: PeerStatus
Address: 192.168.1.25:5065
ChannelType: SIP
Peer: SIP/101
PeerStatus: Registered
Privilege: system,all

now memory state
            total       used       free     shared    buffers     cached
Mem:       1030584     167032     863552          0       8596      89684

as we see recieving messages over AMI also inrease RAM buffer

now sent CLI command: sip show peers

recived:
Name/username             Host                                    Dyn Forcerport Comedia    ACL Port     Status      Description                      Realtime
101/101                   192.168.1.25                             D  No         Yes         A  5065     Unmonitored                                  Cached RT
1 sip peers [Monitored: 0 online, 0 offline Unmonitored: 1 online, 0 offline]

now memory state
            total       used       free     shared    buffers     cached
Mem:       1030584     167032     863552          0       8604      89684

repeat this exercise 5 times over ~3 sec. pause, now memory state
            total       used       free     shared    buffers     cached
Mem:       1030584     167032     863552          0       8644      89684


By: Matt Jordan (mjordan) 2014-01-30 10:55:30.192-0600

MySQL realtime is an extended support module and is support by the community. It is known to have several problems; a memory leak would not surprise me.

Can you reproduce this same leak with a static configuration of manager.conf?
By: Denis (den159) 2014-01-30 12:03:00.175-0600

I`ll try maybe on the next week, and will report You immediately
By: Denis (den159) 2014-02-06 02:52:01.697-0600

Exact same result with static manager configuration (conf files attached). RAM buffer does not grows when sending those comands through CLI interface in console, that is the reason why this question appeared, maybe my fears are groundless but would like to understand.
By: Denis (den159) 2014-02-08 01:19:33.681-0600

Also today i wrote my own socket for Centos where Asterisk runs, and tried to send CLI commands and recieve responses over it using key -x (asterisk -x core show settings) etc., all works fine - no memory leak, no RAM buffer increase.
By: Matt Jordan (mjordan) 2014-02-24 12:51:42.193-0600

Just to double check, how are you getting the memory output?

If you're simply looking at system memory usage, there's no guarantee that the memory leak isn't in the AMI library you're using to connect to Asterisk. One way to double check that would be to compile Asterisk with MALLOC_DEBUG enabled, and then use the CLI commands it affords to check the memory usage:

{{memory show summary}} - this will show the bytes allocated by the various components
{{memory atexit list}} - show leaked memory on exit


By: Denis (den159) 2014-02-28 03:02:07.773-0600

In example added above for memory output i use command "free" in Centos console
[root@localhost ~]# free
    total   used   free   shared buffers cached
Mem: 1030584 166060 864524 0      8384    89516

Today i`ve compiled Asterisk 12.1.0-rc1 with MALLOC_DEBUG enabled, and after that Asterisk even cant start, with error:
[root@localhost asterisk-12.1.0-rc1]# asterisk -vvvvvvvvvvvvr
Segmentation fault.

If it may help You, I can upload hole preinstalled virtual machine (3GB) with my configurations for Oracle VM VirtualBox.
By: Rusty Newton (rnewton) 2014-03-18 11:02:06.296-0500

Well, you need to get MALLOC_DEBUG working before going further on the memory issue.

I can't reproduce the MALLOC_DEBUG crash by simply compiling with it in the latest SVN-branch-12-r410844.

Open up a new issue for the crash with MALLOC_DEBUG and then, you might go ahead and test with the latest SVN of 12, then get a backtrace from the crash if you can still repeat it:  [Getting a Backtrace|https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace] and post a link to your VM. I do use VirtualBox so I could take quick look at it.
By: Rusty Newton (rnewton) 2014-04-03 18:40:31.944-0500

Denis will you be able to provide a link to your virtual machine image? Otherwise I don't know what else we can do here since we can reproduce the issue.
By: Denis (den159) 2014-04-04 05:27:57.669-0500

Hi Rusty! To avoid misunderstandings i`ve made RDP access for you on my Win2003 server, where you can fast to see and test full configuration and copy VM from there if nesessary. How can i send private message for you ?
By: Rusty Newton (rnewton) 2014-04-21 11:43:33.624-0500

Denis, I think this is turning into a bit of a support issue. You may have a legitimate bug, but it will be up to you to show us how to reproduce the problem easily so that we can investigate. We have so far been unable to reproduce the problem.


I won't be able to RDP to your system to look at VMs there. If we don't have others reporting the issue, and we can't reproduce it, I can't spent any further time looking into it when we have a hundred other issues to look at.

I understand your crash with MALLOC_DEBUG is preventing you from gathering some further information. You should first file a new issue on that crash, with a [backtrace per the wiki instructions|https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace]  and we'll try to get that resolved and then we can re-open this issue if necessary. If that time comes, feel free to contact a bug marshal in #asterisk-bugs on irc.freenode.net to ask for a re-open. I'm going to close this issue out until we get further information.


By: Rusty Newton (rnewton) 2014-04-21 13:20:43.031-0500

Additionally here is a [link to the MALLOC_DEBUG|https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag] documentation on the wiki.
By: Denis (den159) 2014-04-21 14:11:41.482-0500

OK, i understand, if this issue becomes critical for us, i`ll try to find more time to gather more information and details so that we can solve this problem.