Summary: | ASTERISK-26505: Using res_odbc module for querying a longBLOB column causes a big memory usage | ||
Reporter: | Leandro Dardini (ldardini) | Labels: | |
Date Opened: | 2016-10-26 11:24:36 | Date Closed: | |
Priority: | Minor | Regression? | |
Status: | Open/New | Components: | Resources/res_odbc |
Versions: | 13.2.0 13.4.0 13.11.2 | Frequency of Occurrence | Constant |
Related Issues: | |||
Environment: | CentOS 6.x 64 bit, Mysql 5.1 | Attachments: | ( 0) dtmf.issue26505.txt ( 1) full.issue26505.txt ( 2) issue26505-memory_show_allocations.gz ( 3) issue26505-memory_show_summary.txt ( 4) issue26505-ps-postdial.txt ( 5) issue26505-ps-predial.txt ( 6) issue26505-strace-memory.txt ( 7) messages.issue26505.txt ( 8) mmlog |
Description: | Please note I have listed the versions where I tried this. Not listed version were not tested.
If I have a table with a longBLOB field (but probably it happens for any type of BLOB column) and I define in func_odbc.conf an entry like {noformat} [READ_EXTENSIONS] dsn=asterisk1,asterisk2 synopsis=Retrieve the extension data readsql=SELECT * FROM ex_extensions where id='${ARG1}' {noformat} Using a function like {noformat} Set(HASH(phone)=${ODBC_READ_EXTENSIONS(56)}); {noformat} The same command, using any other type of column, gives no problems at all. Determines an increase of virtual memory usage in Asterisk... I have seen it getting up to 30 GB on a medium sized server. When this abnormal memory usage happens, asterisk continue to work, but it cannot spawn external processes, like from res_agi.c: {noformat} [2016-10-26 10:04:18] WARNING[26880][C-000002bb] res_agi.c: Failed to fork(): Cannot allocate memory {noformat} | ||
Comments: | By: Asterisk Team (asteriskteam) 2016-10-26 11:24:36.931-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Rusty Newton (rnewton) 2016-10-26 16:55:17.438-0500 Hey Leandro! I hope you are doing well. Can you help us out by gathering some further data as described on the wiki page linked below: https://wiki.asterisk.org/wiki/display/AST/Memory+Leak+Debugging By: Leandro Dardini (ldardini) 2016-10-31 12:43:23.513-0500 I am sorry, but there is no memory leak, it is just an abnormal memory usage when res_odbc tries to read a BLOB field. I don't know if this can be categorized as bug or not By: Leandro Dardini (ldardini) 2016-10-31 14:56:39.698-0500 I was able to run some additional tests. Here is asterisk memory usage with just one channel but an access to a table with a BLOB column using ODBC: 6 GB root 19487 25.8 75.2 6266628 3025996 pts/1 Sl 20:28 1:23 /usr/sbin/asterisk -f -vvvg -c I think the res_odbc.c is reserving memory for the whole 4 GB BLOB column. By: Rusty Newton (rnewton) 2016-10-31 15:39:15.328-0500 Actually the MALLOC_DEBUG output should still help us out. Can you still gather it along with the log (as described in the link above) at the same time? By: Leandro Dardini (ldardini) 2016-11-01 05:17:12.669-0500 I don't think the MALLOC_DEBUG will be of any help, but here are all the files. Due to the fact I don't see any useful information in the log file, I run a strace on asterisk where at least I can see the mmap for 4 GB memory. I run this command, just to avoid any misunderstanding on the output generated: strace -f -F -p 12741 -e trace=memory &>/tmp/issue26505-strace-memory.txt & ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-predial.txt ; asterisk -rx 'logger add channel debug_log_26505 notice,warning,error,debug,verbose,dtmf'; asterisk -rx 'core set verbose 5'; asterisk -rx 'core set debug 5'; asterisk -rx 'module reload logger'; asterisk -rx 'logger rotate' ; asterisk -rx 'memory backtrace on' ; echo "Dial 9999" ; read ; ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-postdial.txt ; asterisk -rx 'memory show summary' | tee /tmp/issue26505-memory_show_summary.txt ; asterisk -rx 'memory show allocations' | tee /tmp/issue26505-memory_show_allocations The actual extension code run was: {noformat} 9999 => { Answer(); Set(HASH(extension)=${ODBC_READ_EXTENSIONS(5)}); NoOp(I have loaded ${HASH(extension,ex_name)}); Wait(60); Hangup(); } {noformat} And the func_odbc.conf used was: {noformat} [READ_EXTENSIONS] dsn=asterisk1,asterisk2 synopsis=Retrieve the extension data readsql=SELECT * FROM ex_extensions where ex_id='${ARG1}' {noformat} The table definition is {noformat} CREATE TABLE IF NOT EXISTS `ex_extensions` ( `ex_id` int(11) NOT NULL AUTO_INCREMENT, `ex_te_id` int(11) NOT NULL, `ex_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_tech` enum('SIP','IAX2','VIRTUAL','PJSIP') COLLATE utf8_unicode_ci DEFAULT NULL, `ex_number` varchar(16) COLLATE utf8_unicode_ci NOT NULL, `ex_trunk` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_tech_id` int(11) NOT NULL, `ex_cidnum` varchar(20) COLLATE utf8_unicode_ci NOT NULL, `ex_cidname` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_cidusage` varchar(20) COLLATE utf8_unicode_ci NOT NULL, `ex_blockcid` varchar(10) COLLATE utf8_unicode_ci NOT NULL, `ex_prefix` varchar(10) COLLATE utf8_unicode_ci NOT NULL, `ex_mindigitprefix` int(11) NOT NULL, `ex_maxdigitprefix` int(11) NOT NULL, `ex_fmfmstatus` char(2) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_fmfmnumber` varchar(50) COLLATE utf8_unicode_ci NOT NULL, `ex_fmfmdialtimeout` int(11) NOT NULL, `ex_fmfmdialmethod` varchar(50) COLLATE utf8_unicode_ci NOT NULL, `ex_fmfmcallerid` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_fmfmnumprefix` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_fmfmnameprefix` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_recording` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'no', `ex_dialtimeout` int(11) NOT NULL, `ex_callgroup` int(11) NOT NULL, `ex_pickupgroup` int(11) NOT NULL, `ex_unconditionalstatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_onbusystatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_onofflinestatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_onnoanswerstatus` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_callallowed` varchar(40) COLLATE utf8_unicode_ci DEFAULT 'all', `ex_destallowregex` varchar(255) COLLATE utf8_unicode_ci DEFAULT '', `ex_rp_id` int(11) NOT NULL, `ex_dnd` varchar(16) COLLATE utf8_unicode_ci NOT NULL, `ex_emergencycidnum` varchar(20) COLLATE utf8_unicode_ci NOT NULL, `ex_webpassword` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_up_id` int(11) NOT NULL, `ex_userpanel` varchar(10) COLLATE utf8_unicode_ci NOT NULL, `ex_datecreation` datetime DEFAULT NULL, `ex_emailrecording` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_fmfmconfirm` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_fmfmconfirmmessage_id` int(11) NOT NULL, `ex_minemailrecording` int(11) NOT NULL, `ex_branch` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_department` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_autocallerid` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'on', `ex_trunkcidoverride` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_trunkemergencycidoverride` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_cr_id` int(11) NOT NULL, `ex_trunkcidsource` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL, `ex_description` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_fmfmholdmessage_id` int(11) NOT NULL, `ex_notifymissingemail` varchar(255) COLLATE utf8_unicode_ci NOT NULL, `ex_webuseldap` varchar(10) COLLATE utf8_unicode_ci NOT NULL, `ex_webuser` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_txvolume` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_rxvolume` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_mu_id` int(11) DEFAULT '0', `ex_faxgateway` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_faxalert` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_trunkdid` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_fmfmdelay` int(11) NOT NULL DEFAULT '0', `ex_includeindbn` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'yes', `ex_abusedetection` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_referenceid` varchar(40) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `ex_notes` longblob, UNIQUE KEY `ex_id` (`ex_id`), KEY `ex_tech_id` (`ex_tech_id`), KEY `ex_te_id` (`ex_te_id`,`ex_number`) ) {noformat} By: Leandro Dardini (ldardini) 2016-11-01 05:21:54.864-0500 Data created using strace -f -F -p 12741 -e trace=memory &>/tmp/issue26505-strace-memory.txt & ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-predial.txt ; asterisk -rx 'logger add channel debug_log_26505 notice,warning,error,debug,verbose,dtmf'; asterisk -rx 'core set verbose 5'; asterisk -rx 'core set debug 5'; asterisk -rx 'module reload logger'; asterisk -rx 'logger rotate' ; asterisk -rx 'memory backtrace on' ; echo "Dial 9999" ; read ; ps aux | grep "/usr/sbin/asterisk" | tee /tmp/issue26505-ps-postdial.txt ; asterisk -rx 'memory show summary' | tee /tmp/issue26505-memory_show_summary.txt ; asterisk -rx 'memory show allocations' | tee /tmp/issue26505-memory_show_allocations By: Rusty Newton (rnewton) 2016-11-01 09:20:12.299-0500 Thanks Leandro! |