Summary: | ASTERISK-23070: Memory leak when using CLI through AMI | ||
Reporter: | Denis (den159) | Labels: | |
Date Opened: | 2013-12-26 10:01:58.000-0600 | Date Closed: | 2014-04-21 11:43:47 |
Priority: | Critical | Regression? | |
Status: | Closed/Complete | Components: | Core/ManagerInterface |
Versions: | 12.0.0 | Frequency of Occurrence | |
Related Issues: | |||
Environment: | Centos 6.5 on Oracle VM VirtualBox | Attachments: | ( 0) extconfig.conf ( 1) manager.conf |
Description: | Calling CLI commands through AMI interface causes memory leak (sending AMI actions does not cause leak), each call CLI command through AMI increase RAM buffer by 8 kb. I am using Astxx lib (by Matthew A. Nicholson http://matt-land.com/astxx/) as AMI caller, memory usage increases when executed
action::command command(str_cmd); command(*connectionn); | ||
Comments: | By: Matt Jordan (mjordan) 2013-12-30 10:05:22.308-0600 I think we need some more information to be able to reproduce this accurately. Which CLI commands are you executing? For those CLI commands, what data is returned? Please also attach a sanitized {{manager.conf}}. By: Denis (den159) 2014-01-02 05:37:06.456-0600 manager conf in realtime MySQL table: manager.conf general enabled yes manager.conf general port 5038 manager.conf general bindaddr 0.0.0.0 manager.conf manager secret xxxxxx manager.conf manager deny 0.0.0.0/0.0.0.0 manager.conf manager permit 192.168.1.0/255.255.255.0 manager.conf manager read system,call,command,agent,user,dtmf,reporting,originate,agi,cc,aoc manager.conf manager write system,call,command,agent,user,dtmf,reporting,originate,agi,cc,aoc manager.conf manager eventfilter Event: Registry manager.conf manager eventfilter Event: PeerStatus manager.conf manager eventfilter Event: Dial manager.conf manager eventfilter Event: DialBegin manager.conf manager eventfilter Event: DialEnd manager.conf manager eventfilter Event: Newstate manager.conf manager eventfilter Event: NewCallerid manager.conf manager eventfilter Event: Newchannel manager.conf manager eventfilter Event: PresenceStatus manager.conf manager eventfilter Event: Hangup In short on base of above mentioned Astxx, i have compiled .dll for third party windows application 1C, I also tried compile the same .dll on pure WinSock2 in both cases memory leak presents. Active using AMI with 4 clients, increase RAM buffer for ~3 hours from ~8400 kb to ~20000 kb, below you can see details: system booted [root@localhost ~]# free total used free shared buffers cached Mem: 1030584 166060 864524 0 8384 89516 then registered trunk, registered peer, connect and login AMI manager, and now we have Mem: 1030584 166372 864212 0 8444 89616 it's OK, and now trying to send CLI over AMI sent CLI command: core show settings recived: PBX Core settings ----------------- Version: 12.0.0 Build Options: LOADABLE_MODULES, BUILD_NATIVE, OPTIONAL_API Maximum calls: Not set Maximum open file handles: Not set Verbosity: 3 Debug level: 0 Maximum load average: 0.000000 Minimum free memory: 0 MB Startup time: 16:45:52 Last reload time: 16:45:52 System: Linux/2.6.32-431.1.2.0.1.el6.i686 built by root on i686 2013-12-31 08:53:35 UTC System name: Entity ID: 08:00:27:15:3e:e7 Default language: en Language prefix: Enabled User name and group: / Executable includes: Disabled Transcode via SLIN: Enabled Internal timing: Enabled Transmit silence during rec: Disabled Generic PLC: Enabled Min DTMF duration:: 80 * Subsystems ------------- Manager (AMI): Enabled Web Manager (AMI/HTTP): Disabled Call data records: Disabled Realtime Architecture (ARA): Enabled * Directories ------------- Configuration file: Configuration directory: /etc/asterisk Module directory: /usr/lib/asterisk/modules Spool directory: /var/spool/asterisk Log directory: /var/log/asterisk Run/Sockets directory: /var/run/asterisk PID file: /var/run/asterisk/asterisk.pid VarLib directory: /var/lib/asterisk Data directory: /var/lib/asterisk ASTDB: /var/lib/asterisk/astdb IAX2 Keys directory: /var/lib/asterisk/keys AGI Scripts directory: /var/lib/asterisk/agi-bin now memory state total used free shared buffers cached Mem: 1030584 166400 864184 0 8452 89616 repeat this exercise 5 times over ~3 sec. pause, now memory state total used free shared buffers cached Mem: 1030584 166536 864048 0 8500 89652 logoff and disconnect AMI manager for 10 minutes... ...now connect and login AMI manager now memory state total used free shared buffers cached Mem: 1030584 167032 863552 0 8564 89684 10 minutes do nothing... ...10 minutes have passed, during which nothing happens but we recieved several event messages: Event: Registry ChannelType: SIP Domain: qwerty.cnt.ru Privilege: system,all Status: Registered Username: xxxxxxxxxx Event: PeerStatus Address: 192.168.1.25:5065 ChannelType: SIP Peer: SIP/101 PeerStatus: Registered Privilege: system,all Event: Registry ChannelType: SIP Domain: qwerty.cnt.ru Privilege: system,all Status: Registered Username: xxxxxxxxxx Event: Registry ChannelType: SIP Domain: qwerty.cnt.ru Privilege: system,all Status: Registered Username: xxxxxxxxxx Event: Registry ChannelType: SIP Domain: qwerty.cnt.ru Privilege: system,all Status: Registered Username: xxxxxxxxxx Event: PeerStatus Address: 192.168.1.25:5065 ChannelType: SIP Peer: SIP/101 PeerStatus: Registered Privilege: system,all now memory state total used free shared buffers cached Mem: 1030584 167032 863552 0 8596 89684 as we see recieving messages over AMI also inrease RAM buffer now sent CLI command: sip show peers recived: Name/username Host Dyn Forcerport Comedia ACL Port Status Description Realtime 101/101 192.168.1.25 D No Yes A 5065 Unmonitored Cached RT 1 sip peers [Monitored: 0 online, 0 offline Unmonitored: 1 online, 0 offline] now memory state total used free shared buffers cached Mem: 1030584 167032 863552 0 8604 89684 repeat this exercise 5 times over ~3 sec. pause, now memory state total used free shared buffers cached Mem: 1030584 167032 863552 0 8644 89684 By: Matt Jordan (mjordan) 2014-01-30 10:55:30.192-0600 MySQL realtime is an extended support module and is support by the community. It is known to have several problems; a memory leak would not surprise me. Can you reproduce this same leak with a static configuration of manager.conf? By: Denis (den159) 2014-01-30 12:03:00.175-0600 I`ll try maybe on the next week, and will report You immediately By: Denis (den159) 2014-02-06 02:52:01.697-0600 Exact same result with static manager configuration (conf files attached). RAM buffer does not grows when sending those comands through CLI interface in console, that is the reason why this question appeared, maybe my fears are groundless but would like to understand. By: Denis (den159) 2014-02-08 01:19:33.681-0600 Also today i wrote my own socket for Centos where Asterisk runs, and tried to send CLI commands and recieve responses over it using key -x (asterisk -x core show settings) etc., all works fine - no memory leak, no RAM buffer increase. By: Matt Jordan (mjordan) 2014-02-24 12:51:42.193-0600 Just to double check, how are you getting the memory output? If you're simply looking at system memory usage, there's no guarantee that the memory leak isn't in the AMI library you're using to connect to Asterisk. One way to double check that would be to compile Asterisk with MALLOC_DEBUG enabled, and then use the CLI commands it affords to check the memory usage: {{memory show summary}} - this will show the bytes allocated by the various components {{memory atexit list}} - show leaked memory on exit By: Denis (den159) 2014-02-28 03:02:07.773-0600 In example added above for memory output i use command "free" in Centos console [root@localhost ~]# free total used free shared buffers cached Mem: 1030584 166060 864524 0 8384 89516 Today i`ve compiled Asterisk 12.1.0-rc1 with MALLOC_DEBUG enabled, and after that Asterisk even cant start, with error: [root@localhost asterisk-12.1.0-rc1]# asterisk -vvvvvvvvvvvvr Segmentation fault. If it may help You, I can upload hole preinstalled virtual machine (3GB) with my configurations for Oracle VM VirtualBox. By: Rusty Newton (rnewton) 2014-03-18 11:02:06.296-0500 Well, you need to get MALLOC_DEBUG working before going further on the memory issue. I can't reproduce the MALLOC_DEBUG crash by simply compiling with it in the latest SVN-branch-12-r410844. Open up a new issue for the crash with MALLOC_DEBUG and then, you might go ahead and test with the latest SVN of 12, then get a backtrace from the crash if you can still repeat it: [Getting a Backtrace|https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace] and post a link to your VM. I do use VirtualBox so I could take quick look at it. By: Rusty Newton (rnewton) 2014-04-03 18:40:31.944-0500 Denis will you be able to provide a link to your virtual machine image? Otherwise I don't know what else we can do here since we can reproduce the issue. By: Denis (den159) 2014-04-04 05:27:57.669-0500 Hi Rusty! To avoid misunderstandings i`ve made RDP access for you on my Win2003 server, where you can fast to see and test full configuration and copy VM from there if nesessary. How can i send private message for you ? By: Rusty Newton (rnewton) 2014-04-21 11:43:33.624-0500 Denis, I think this is turning into a bit of a support issue. You may have a legitimate bug, but it will be up to you to show us how to reproduce the problem easily so that we can investigate. We have so far been unable to reproduce the problem. I won't be able to RDP to your system to look at VMs there. If we don't have others reporting the issue, and we can't reproduce it, I can't spent any further time looking into it when we have a hundred other issues to look at. I understand your crash with MALLOC_DEBUG is preventing you from gathering some further information. You should first file a new issue on that crash, with a [backtrace per the wiki instructions|https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace] and we'll try to get that resolved and then we can re-open this issue if necessary. If that time comes, feel free to contact a bug marshal in #asterisk-bugs on irc.freenode.net to ask for a re-open. I'm going to close this issue out until we get further information. By: Rusty Newton (rnewton) 2014-04-21 13:20:43.031-0500 Additionally here is a [link to the MALLOC_DEBUG|https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag] documentation on the wiki. By: Denis (den159) 2014-04-21 14:11:41.482-0500 OK, i understand, if this issue becomes critical for us, i`ll try to find more time to gather more information and details so that we can solve this problem. |