[Home]

Summary:ASTERISK-26505: Using res_odbc module for querying a longBLOB column causes a big memory usage
Reporter:Leandro Dardini (ldardini)Labels:
Date Opened:2016-10-26 11:24:36Date Closed:
Priority:MinorRegression?
Status:Open/NewComponents:Resources/res_odbc
Versions:13.2.0 13.4.0 13.11.2 Frequency of
Occurrence		Constant
Related
Issues:
Environment:CentOS 6.x 64 bit, Mysql 5.1Attachments:( 0) dtmf.issue26505.txt
( 1) full.issue26505.txt
( 2) issue26505-memory_show_allocations.gz
( 3) issue26505-memory_show_summary.txt
( 4) issue26505-ps-postdial.txt
( 5) issue26505-ps-predial.txt
( 6) issue26505-strace-memory.txt
( 7) messages.issue26505.txt
( 8) mmlog
Description:Please note I have listed the versions where I tried this. Not listed version were not tested.

If I have a table with a longBLOB field (but probably it happens for any type of BLOB column) and I define in func_odbc.conf an entry like

{noformat}
[READ_EXTENSIONS]
dsn=asterisk1,asterisk2
synopsis=Retrieve the extension data
readsql=SELECT * FROM ex_extensions where id='${ARG1}'
{noformat}

Using a function like

{noformat}
Set(HASH(phone)=${ODBC_READ_EXTENSIONS(56)});
{noformat}

The same command, using any other type of column, gives no problems at all.

Determines an increase of virtual memory usage in Asterisk... I have seen it getting up to 30 GB on a medium sized server.

When this abnormal memory usage happens, asterisk continue to work, but it cannot spawn external processes, like from res_agi.c:

{noformat}
[2016-10-26 10:04:18] WARNING[26880][C-000002bb] res_agi.c: Failed to fork(): Cannot allocate memory
{noformat}

Comments:By: Asterisk Team (asteriskteam) 2016-10-26 11:24:36.931-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].
By: Rusty Newton (rnewton) 2016-10-26 16:55:17.438-0500

Hey Leandro! I hope you are doing well.

Can you help us out by gathering some further data as described on the wiki page linked below:

https://wiki.asterisk.org/wiki/display/AST/Memory+Leak+Debugging


By: Leandro Dardini (ldardini) 2016-10-31 12:43:23.513-0500

I am sorry, but there is no memory leak, it is just an abnormal memory usage when res_odbc tries to read a BLOB field. I don't know if this can be categorized as bug or not
By: Leandro Dardini (ldardini) 2016-10-31 14:56:39.698-0500

I was able to run some additional tests. Here is asterisk memory usage with just one channel but an access to a table with a BLOB column using ODBC: 6 GB

root     19487 25.8 75.2 6266628 3025996 pts/1 Sl   20:28   1:23 /usr/sbin/asterisk -f -vvvg -c

I think the res_odbc.c is reserving memory for the whole 4 GB BLOB column.



By: Rusty Newton (rnewton) 2016-10-31 15:39:15.328-0500

Actually the MALLOC_DEBUG output should still help us out. Can you still gather it along with the log (as described in the link above) at the same time?
By: Leandro Dardini (ldardini) 2016-11-01 05:17:12.669-0500

I don't think the MALLOC_DEBUG will be of any help, but here are all the files. Due to the fact I don't see any useful information in the log file, I run a strace on asterisk where at least I can see the mmap for 4 GB memory.

I run this command, just to avoid any misunderstanding on the output generated:

strace -f -F -p 12741 -e trace=memory &>/tmp/issue26505-strace-memory.txt &  ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-predial.txt ; asterisk -rx 'logger add channel debug_log_26505 notice,warning,error,debug,verbose,dtmf'; asterisk -rx 'core set verbose 5'; asterisk -rx 'core set debug 5'; asterisk -rx 'module reload logger'; asterisk -rx 'logger rotate' ; asterisk -rx 'memory backtrace on' ; echo "Dial 9999" ; read  ; ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-postdial.txt ; asterisk -rx 'memory show summary' | tee /tmp/issue26505-memory_show_summary.txt ; asterisk -rx 'memory show allocations' | tee /tmp/issue26505-memory_show_allocations

The actual extension code run was:

{noformat}
       9999 => {
            Answer();
            Set(HASH(extension)=${ODBC_READ_EXTENSIONS(5)});
            NoOp(I have loaded ${HASH(extension,ex_name)});
            Wait(60);
            Hangup();
       }
{noformat}

And the func_odbc.conf used was:

{noformat}
[READ_EXTENSIONS]
dsn=asterisk1,asterisk2
synopsis=Retrieve the extension data
readsql=SELECT * FROM ex_extensions where ex_id='${ARG1}'
{noformat}

The table definition is

{noformat}
CREATE TABLE IF NOT EXISTS `ex_extensions` (
 `ex_id` int(11) NOT NULL AUTO_INCREMENT,
 `ex_te_id` int(11) NOT NULL,
 `ex_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_tech` enum('SIP','IAX2','VIRTUAL','PJSIP') COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_number` varchar(16) COLLATE utf8_unicode_ci NOT NULL,
 `ex_trunk` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_tech_id` int(11) NOT NULL,
 `ex_cidnum` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
 `ex_cidname` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_cidusage` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
 `ex_blockcid` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
 `ex_prefix` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
 `ex_mindigitprefix` int(11) NOT NULL,
 `ex_maxdigitprefix` int(11) NOT NULL,
 `ex_fmfmstatus` char(2) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_fmfmnumber` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
 `ex_fmfmdialtimeout` int(11) NOT NULL,
 `ex_fmfmdialmethod` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
 `ex_fmfmcallerid` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_fmfmnumprefix` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_fmfmnameprefix` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_recording` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'no',
 `ex_dialtimeout` int(11) NOT NULL,
 `ex_callgroup` int(11) NOT NULL,
 `ex_pickupgroup` int(11) NOT NULL,
 `ex_unconditionalstatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_onbusystatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_onofflinestatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_onnoanswerstatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_callallowed` varchar(40) COLLATE utf8_unicode_ci DEFAULT 'all',
 `ex_destallowregex` varchar(255) COLLATE utf8_unicode_ci DEFAULT '',
 `ex_rp_id` int(11) NOT NULL,
 `ex_dnd` varchar(16) COLLATE utf8_unicode_ci NOT NULL,
 `ex_emergencycidnum` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
 `ex_webpassword` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_up_id` int(11) NOT NULL,
 `ex_userpanel` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
 `ex_datecreation` datetime DEFAULT NULL,
 `ex_emailrecording` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_fmfmconfirm` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_fmfmconfirmmessage_id` int(11) NOT NULL,
 `ex_minemailrecording` int(11) NOT NULL,
 `ex_branch` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_department` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_autocallerid` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'on',
 `ex_trunkcidoverride` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_trunkemergencycidoverride` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_cr_id` int(11) NOT NULL,
 `ex_trunkcidsource` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
 `ex_description` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_fmfmholdmessage_id` int(11) NOT NULL,
 `ex_notifymissingemail` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 `ex_webuseldap` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
 `ex_webuser` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_txvolume` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_rxvolume` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_mu_id` int(11) DEFAULT '0',
 `ex_faxgateway` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_faxalert` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_trunkdid` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_fmfmdelay` int(11) NOT NULL DEFAULT '0',
 `ex_includeindbn` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'yes',
 `ex_abusedetection` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_referenceid` varchar(40) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `ex_notes` longblob,
 UNIQUE KEY `ex_id` (`ex_id`),
 KEY `ex_tech_id` (`ex_tech_id`),
 KEY `ex_te_id` (`ex_te_id`,`ex_number`)
)
{noformat}




By: Leandro Dardini (ldardini) 2016-11-01 05:21:54.864-0500

Data created using

strace -f -F -p 12741 -e trace=memory &>/tmp/issue26505-strace-memory.txt &  ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-predial.txt ; asterisk -rx 'logger add channel debug_log_26505 notice,warning,error,debug,verbose,dtmf'; asterisk -rx 'core set verbose 5'; asterisk -rx 'core set debug 5'; asterisk -rx 'module reload logger'; asterisk -rx 'logger rotate' ; asterisk -rx 'memory backtrace on' ; echo "Dial 9999" ; read  ; ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-postdial.txt ; asterisk -rx 'memory show summary' | tee /tmp/issue26505-memory_show_summary.txt ; asterisk -rx 'memory show allocations' | tee /tmp/issue26505-memory_show_allocations
By: Rusty Newton (rnewton) 2016-11-01 09:20:12.299-0500

Thanks Leandro!