| Summary: | ASTERISK-27261: Memory Leak when using ARI in json.c and stasis_channels.c | ||||||
| Reporter: | Ali Shahbour (shahbour) | Labels: | |||||
| Date Opened: | 2017-09-07 12:02:05 | Date Closed: | 2017-09-11 06:09:45 | ||||
| Priority: | Major | Regression? | |||||
| Status: | Closed/Complete | Components: | Applications/app_stasis | ||||
| Versions: | 14.6.0 | Frequency of Occurrence | Constant | ||||
| Related Issues: |
| ||||||
| Environment: | Attachments: | ( 0) json.log.zip | |||||
| Description: | I am running a program that do use ARI to originate calls and also receive calls through SIP into a stasis application , I noticed that almost every 3 days i got my 6 GB server memory full . I first thought it is from moh but then after checking the memory i noticed that stasis_channels.c is getting increased all the time , i did a restart and tested with only one call and i noticed that on each call the number of allocation increased by 2 (while the call is active it increase by 8 then drop 6 and keep 2)
{noformat} astercore*CLI> memory show summary 277909 bytes in 426 allocations in file /usr/local/src/asterisk-14.6.0/include/asterisk/strings.h 709335 bytes in 2456 allocations in file /usr/local/src/asterisk-14.6.0/include/asterisk/threadstorage.h 1263 bytes in 67 allocations in file ael/pval.c 2552 bytes in 6 allocations in file app.c 256 bytes in 3 allocations in file app_agent_pool.c 736 bytes in 1 allocations in file app_bridgewait.c 1168 bytes in 3 allocations in file app_confbridge.c 25520 bytes in 2 allocations in file app_followme.c 36011 bytes in 10 allocations in file app_minivm.c 6784 bytes in 2 allocations in file app_queue.c 14998 bytes in 10 allocations in file app_voicemail.c 104 bytes in 1 allocations in file ari/ari_websockets.c 1109 bytes in 6 allocations in file ari/config.c 34650 bytes in 210 allocations in file ari/resource_channels.c 861 bytes in 3 allocations in file ari/resource_events.c 24454 bytes in 979 allocations in file asterisk.c 745040 bytes in 9313 allocations in file astobj2_hash.c 33352 bytes in 379 allocations in file astobj2_rbtree.c 304 bytes in 2 allocations in file bridge.c 1016 bytes in 1 allocations in file bucket.c 896 bytes in 6 allocations in file ccss.c 618009 bytes in 2091 allocations in file cdr.c 7023 bytes in 7 allocations in file cel.c 176 bytes in 2 allocations in file chan_bridge_media.c 1146800 bytes in 29 allocations in file chan_iax2.c 176 bytes in 2 allocations in file chan_mgcp.c 1916 bytes in 3 allocations in file chan_oss.c 264 bytes in 3 allocations in file chan_phone.c 176 bytes in 2 allocations in file chan_rtp.c 6629795 bytes in 9007 allocations in file chan_sip.c 176 bytes in 2 allocations in file chan_skinny.c 4272 bytes in 3 allocations in file chan_unistim.c 166288 bytes in 2104 allocations in file channel.c 540414 bytes in 1044 allocations in file channel_internal_api.c 16 bytes in 1 allocations in file chanvars.c 13320 bytes in 670 allocations in file cli.c 6176 bytes in 44 allocations in file codec.c 25920 bytes in 1 allocations in file codec_resample.c 150980 bytes in 71 allocations in file confbridge/conf_config_parser.c 89209 bytes in 642 allocations in file config.c 61592 bytes in 288 allocations in file config_options.c 248 bytes in 2 allocations in file core_local.c 20531 bytes in 33 allocations in file data.c 478 bytes in 8 allocations in file devicestate.c 79097 bytes in 840 allocations in file dial.c 9800 bytes in 14 allocations in file endpoints.c 421528 bytes in 2497 allocations in file features_config.c 8184 bytes in 31 allocations in file file.c 30060 bytes in 458 allocations in file format.c 1016 bytes in 1 allocations in file format_cache.c 5624824 bytes in 41936 allocations in file format_cap.c 355731 bytes ( 355731 cache) in 1171 allocations in file frame.c 736 bytes in 1 allocations in file func_dialgroup.c 1472 bytes in 4 allocations in file hashtab.c 1997 bytes in 14 allocations in file http.c 256 bytes in 1 allocations in file iax2/provision.c 67119 bytes in 1200 allocations in file indications.c 41144 bytes in 21 allocations in file io.c 102198920 bytes in 785064 allocations in file json.c 2296 bytes in 1 allocations in file libasteriskssl.c 42144 bytes in 614 allocations in file loader.c 8390 bytes in 6 allocations in file logger.c 190311 bytes in 286 allocations in file manager.c 1120 bytes in 1 allocations in file media_cache.c 3479064 bytes in 16082 allocations in file media_index.c 96 bytes in 2 allocations in file message.c 832 bytes in 2 allocations in file named_acl.c 1760 bytes in 1 allocations in file named_locks.c 464 bytes in 3 allocations in file netsock.c 144 bytes in 1 allocations in file parking.c 162879 bytes in 1889 allocations in file pbx.c 144 bytes in 1 allocations in file pbx_ael.c 494168 bytes in 441 allocations in file pbx_app.c 396384 bytes in 254 allocations in file pbx_functions.c 60 bytes in 6 allocations in file pbx_ignorepat.c 10318 bytes in 37 allocations in file pbx_include.c 9312 bytes in 1 allocations in file pbx_realtime.c 102 bytes in 2 allocations in file pbx_sw.c 79183 bytes in 1637 allocations in file pbx_variables.c 56 bytes in 1 allocations in file presencestate.c 592 bytes in 3 allocations in file res_ari.c 83 bytes in 3 allocations in file res_ari_events.c 448 bytes in 1 allocations in file res_calendar.c 3215 bytes in 7 allocations in file res_clialiases.c 256 bytes in 1 allocations in file res_config_sqlite3.c 304 bytes in 1 allocations in file res_fax.c 48 bytes in 1 allocations in file res_format_attr_opus.c 728 bytes in 4 allocations in file res_hep.c 1443 bytes in 11 allocations in file res_http_websocket.c 2406 bytes in 8 allocations in file res_musiconhold.c 2172 bytes in 9 allocations in file res_parking.c 77645 bytes in 206 allocations in file res_phoneprov.c 5636436 bytes in 1722 allocations in file res_rtp_asterisk.c 160 bytes in 1 allocations in file res_smdi.c 406 bytes in 3 allocations in file res_sorcery_config.c 992 bytes in 1 allocations in file res_sorcery_memory_cache.c 23120 bytes in 383 allocations in file res_stasis.c 736 bytes in 1 allocations in file res_stasis_device_state.c 2176 bytes in 1 allocations in file res_stasis_playback.c 2176 bytes in 1 allocations in file res_stasis_recording.c 376 bytes in 2 allocations in file res_statsd.c 9152 bytes in 1 allocations in file res_timing_pthread.c 56848 bytes in 418 allocations in file res_timing_timerfd.c 1848355 bytes in 6886 allocations in file rtp_engine.c 1147864 bytes in 3430 allocations in file sched.c 21400 bytes in 529 allocations in file sip/route.c 16797 bytes in 73 allocations in file sorcery.c 454143 bytes in 5527 allocations in file stasis.c 52767 bytes in 382 allocations in file stasis/app.c 187488 bytes in 756 allocations in file stasis/control.c 2232 bytes in 2 allocations in file stasis/messaging.c 188878 bytes in 2119 allocations in file stasis_cache.c 50880 bytes in 424 allocations in file stasis_cache_pattern.c 338322816 bytes in 153504 allocations in file stasis_channels.c 71345 bytes in 636 allocations in file stasis_message.c 3672 bytes in 23 allocations in file stasis_message_router.c 37104 bytes in 2 allocations in file stdtime/localtime.c 14556648 bytes in 85741 allocations in file stringfields.c 32335 bytes in 429 allocations in file strings.c 12375 bytes in 159 allocations in file taskprocessor.c 512 bytes in 3 allocations in file tcptls.c 10808 bytes in 16 allocations in file threadpool.c 6872 bytes in 422 allocations in file timing.c 1790400 bytes in 248 allocations in file translate.c 208 bytes in 2 allocations in file udptl.c 40114 bytes in 485 allocations in file utils.c 1126513 bytes in 5371 allocations in file xmldoc.c 490927828 bytes allocated (355731 in caches) in 1154424 selected allocations 490927828 bytes in all allocations 292611 bytes in deferred free large allocations 41378 bytes in deferred free small allocations 333989 bytes in deferred free allocations 491261817 bytes in all allocations and deferred free allocations {noformat} doing memory show allocation stasis_channels.c shows {noformat} 4048 bytes allocated by ast_channel_snapshot_create() line 215 of stasis_channels.c 360 bytes allocated by ast_channel_snapshot_create() line 214 of stasis_channels.c 4048 bytes allocated by ast_channel_snapshot_create() line 215 of stasis_channels.c 360 bytes allocated by ast_channel_snapshot_create() line 214 of stasis_channels.c 360 bytes allocated by ast_channel_snapshot_create() line 214 of stasis_channels.c 360 bytes allocated by ast_channel_snapshot_create() line 214 of stasis_channels.c 360 bytes allocated by ast_channel_snapshot_create() line 214 of stasis_channels.c 4048 bytes allocated by ast_channel_snapshot_create() line 215 of stasis_channels.c 4048 bytes allocated by ast_channel_snapshot_create() line 215 of stasis_channels.c 4048 bytes allocated by ast_channel_snapshot_create() line 215 of stasis_channels.c 4048 bytes allocated by ast_channel_snapshot_create() line 215 of stasis_channels.c . . . . . . . {noformat} | ||||||
| Comments: | By: Asterisk Team (asteriskteam) 2017-09-07 12:02:08.350-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Rusty Newton (rnewton) 2017-09-07 19:00:54.279-0500 Are you able to share your ARI program (or provide another) so that we can reproduce the issue? Ideally, provide any dialplan or other configuration that needs to be used with the program as well. By: Ali Shahbour (shahbour) 2017-09-08 05:50:12.116-0500 I did create a sample project that have the same memory leak [asterisk-demo|https://github.com/shahbour/asterisk-demo/] Basically what it does is generate a call through ARI then receive it for the sake of simulation and answer it . By: Ali Shahbour (shahbour) 2017-09-08 10:51:46.512-0500 i did some more investigation and this is my finding * I remove the stasis on the receiving end i did not see any memory leak , after the call is hang up no more stasis_channels . * I removed the originating call from the picture and did a call from SIP server directly to asterisk and received the call on stasis , memory leak did exist * I tried here different scenario like answer from dial peer or from stasis , continue stasis or hangup from stasis all kept the same memory leak that do increase by 2 every call . To i think to reproduce it again in your lab all you need to do is receive a sip call on asterisk and send it to ARI stasis application with the following dialplan {noformat} exten => 11409613050252,1,NoOp() same => n,Answer() same => n,Stasis(test-dump) same => n,Wait(5) same => n,Hangup() {noformat} In the stasis I just put moh {noformat} ActionChannels actionChannels = ariService.getAri().getActionImpl(ActionChannels.class); //actionChannels.answer(channelId); actionChannels.continueInDialplan(channelId,"","",0,""); actionChannels.startMoh(channelId,""); ariService.getAri().closeAction(actionChannels); {noformat} By: Ali Shahbour (shahbour) 2017-09-08 12:52:14.634-0500 More tests: Now i dumped my java code and just used `wscat` and `curl` my dialplan is sill as {noformat} exten => 11409613050252,1,NoOp() same => n,Answer() same => n,Stasis(test-dump) same => n,Wait(5) same => n,Hangup() {noformat} {noformat} wscat -c "ws://192.168.70.76:8088/ari/events?api_key=risk:risk&app=test-dump" {noformat} Doing a sip call to and then waiting for like 30 second and then hanging up did not incease the stasis_channels.c Doing a sip call and while is stasis_channel sending either {noformat} curl -v -u risk:risk -X POST "http://192.168.70.76:8088/ari/channels/1504892475.6/continue" {noformat} or {noformat} curl -v -u risk:risk -X POST "http://192.168.70.76:8088/ari/channels/1504892867.8/play?media=sound:hello-world" {noformat} and letting the dialplan hangup the call did increase the stasis_channels.c by 2 By: Joshua C. Colp (jcolp) 2017-09-11 06:09:45.875-0500 This does appear to be a duplicate of ASTERISK-27067. They've now been linked so I'm closing this one out as the other one has already been accepted. By: Ali Shahbour (shahbour) 2017-09-11 06:17:13.495-0500 I checked ASTERISK-27067 and i think it does not mention any thing related to json.c memory leak . In my case both are leaking. I tried downgrading to Asterisk 13 as the other issue said it was not leaking and yes stasis_channels.c stopped leaking but json.c continued By: Asterisk Team (asteriskteam) 2017-09-11 06:17:14.411-0500 This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable. By: Joshua C. Colp (jcolp) 2017-09-11 08:55:27.801-0500 Can you provide the "memory show allocation json.c" for it? By: Ali Shahbour (shahbour) 2017-09-11 09:22:42.221-0500 attached i put both json.c allocation and memory show summary Please note that this log is for asterisk 13.7.1 and not 14 . but in both i had same issue. If you want i can do another test in 14 and update you By: David Hajek (hajekd) 2017-09-11 14:25:57.608-0500 We can see similar json.c leaking on lastest asterisk 13.17.0. The problem seems to be with every request which contains content-type: application/json. We even tried latest 2.10 jansson library with the same results. This is how 'memory show allocations' looks after several API requests (eg. call originate). 104 bytes allocated by ast_json_malloc() line 140 of json.c 75 bytes allocated by ast_json_malloc() line 140 of json.c 77 bytes allocated by ast_json_malloc() line 140 of json.c 81 bytes allocated by ast_json_malloc() line 140 of json.c 104 bytes allocated by ast_json_malloc() line 140 of json.c 81 bytes allocated by ast_json_malloc() line 140 of json.c 104 bytes allocated by ast_json_malloc() line 140 of json.c 128 bytes allocated by ast_json_malloc() line 140 of json.c 200 bytes allocated by ast_json_malloc() line 140 of json.c 200 bytes allocated by ast_json_malloc() line 140 of json.c 104 bytes allocated by ast_json_malloc() line 140 of json.c 144 bytes allocated by ast_json_malloc() line 140 of json.c By: Hajek Michal (hajekmi) 2017-09-11 15:06:18.775-0500 Steps to reproduce memory leak json.c 1. Starting asterisk 13.17.0 {quote} {noformat} asterisk -rx 'memory show summary' | grep json 672 bytes in 5 allocations in file json.c {noformat} {quote} 2. Create channel {quote}{noformat} curl -v -u asterisk:asterisk -X POST -H "Content-Type: application/json" "http://localhost:8088/ari/channels" -d '{"endpoint":"SIP/101", "extension":"s", "context":"test-ari", "priority":"1"}' {"id":"1505159369.0","name":"SIP/101-00000000","state":"Down","caller":{"name":"michal","number":"101"},"connected":{"name":"","number":""},"accountcode":"","dialplan":{"context":"internal-device","exten":"","priority":1},"creationtime":"2017-09-11T21:49:29.758+0200","language":"cz"} {noformat}{quote} 3. After call {quote}{noformat} asterisk -rx 'memory show summary' | grep json 2273 bytes in 19 allocations in file json.c {noformat}{quote} 4. Create next channel {quote}{noformat} curl -v -u asterisk:asterisk -X POST -H "Content-Type: application/json" "http://localhost:8088/ari/channels" -d '{"endpoint":"SIP/101", "extension":"s", "context":"test-ari", "priority":"1"}' {"id":"1505159581.1","name":"SIP/101-00000001","state":"Down","caller":{"name":"michal","number":"101"},"connected":{"name":"","number":""},"accountcode":"","dialplan":{"context":"internal-device","exten":"","priority":1},"creationtime":"2017-09-11T21:53:01.197+0200","language":"cz"} {noformat}{quote} 5. After call {quote}{noformat} asterisk -rx 'memory show summary' | grep json 3874 bytes in 33 allocations in file json.c {noformat}{quote} By: LEI FU (lei.fu) 2017-10-19 03:36:26.591-0500 I have same issue with ARI on Asterisk 14.6.0, these two are running for about 3 weeks, took about 8G memory [tech@as4 ~]$ cat /proc/6874/status | grep VmRSS VmRSS: 8 076 456 kB [tech@as5 ~]$ cat /proc/27268/status | grep VmRSS VmRSS: 8 890 336 kB on another server, after restart asterisk, it went down to 221M [root@as3 log]# cat /proc/20203/status | grep VmRSS VmRSS: 221 772 kB By: Richard Mudgett (rmudgett) 2017-11-04 13:30:11.248-0500 The two memory leaks reported by this issue are fixed individually by ASTERISK-27067 (stasis_channels.c in v14+) and ASTERISK-27305 (json.c v13+). | ||||||