[Home]

Summary:ASTERISK-29071: app_confbridge: Memory rises when jitterbuffer enabled and muting over AMI occurs
Reporter:Stefan Ruf (stefan20)Labels:fax
Date Opened:2020-09-08 06:52:33Date Closed:2021-03-03 10:07:59.000-0600
Priority:MinorRegression?
Status:Closed/CompleteComponents:Applications/app_confbridge Core/Bridging
Versions:16.13.0 Frequency of
Occurrence
Constant
Related
Issues:
Environment:Linux svast20 4.12.14-94.41-default #1 SMP Wed Oct 31 12:25:04 UTC 2018 (3090901) x86_64 x86_64 x86_64 GNU/LinuxAttachments:( 0) ami_commands.txt
( 1) confbridge.txt
( 2) debug_log_20200909_1100.txt
( 3) debug_log_20200914_1026_with_asterisk_restart.txt.gz
( 4) mmlog_20200914_1026__with_asterisk_restart.txt
( 5) mmlog.txt
Description:Using ConfbridgeMute / ConfbridgeUnMute over AMI and additionally MuteAudio on/off results in increasing values of frame.c - shown by "memory show summary". The values rises constant over time and channel. So a system with 24 GB RAM using 5 confbridges permanent with about 20 channels each ist filled up within a day and asterisk restarts.

The problem ist reprorduceable and starts at a certain point. This point is action-id 1705904476_335#berta-ddab9ea6-dcb4-4e83-b9bf-ff5e44325110.
At action-id 1705904476_336#berta-67e059f6-fa67-4d4a-a73a-d5fbd4cf6475 the value frame.c increases with double speed per time. Please see attached file <ami_commands.txt>.
Asterisk had been compiled with MALLOC_DEBUG and “cache_media_frames = no” in asterisk.conf
For one channel the value increases about 32 per second; for two channels the value increases about 64 per second.

Tested versions of asterisk are 16.13.0, 16.8-cert3, 13.36.0, 13.21-cert6, all of them with the described behaviour.
Comments:By: Asterisk Team (asteriskteam) 2020-09-08 06:52:35.150-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

By: Stefan Ruf (stefan20) 2020-09-08 06:56:32.690-0500

sent AMI-Commands with comments

By: Stefan Ruf (stefan20) 2020-09-08 06:58:47.330-0500

Configuration of confbridge

By: Stefan Ruf (stefan20) 2020-09-08 07:00:07.490-0500

Dialplan in extensions.conf:
{noformat}
[funk_berta]
exten => _berta,1,Answer()
exten => _berta,2,Set(JITTERBUFFER(fixed)=20,500)
same => n, NoOp(${CALLERID})
same => n, Set(VOLUME(RX)=2)
same => n, Set(VOLUME(TX)=2)
same => n, ConfBridge(berta)
{noformat}

By: Benjamin Keith Ford (bford) 2020-09-08 09:46:39.278-0500

We're going to need logs[1] that include the issue taking place, as well as memory output[2] that shows the changes in memory values.

[1]: https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information
[2]: https://wiki.asterisk.org/wiki/display/AST/Memory+Leak+Debugging

By: Stefan Ruf (stefan20) 2020-09-14 09:27:32.408-0500

Log with restart of asterisk, after lack of memory

By: Stefan Ruf (stefan20) 2020-09-14 09:36:09.859-0500

attached files ...20200914... where asterisk restarts, after lack of memory.

=> memory short time after start of asterisk and 30 channels in use:
Mon Sep 14 10:59:39 CEST 2020
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
asterisk 21366 13.4  3.5 2796292 290024 ?      Ssl  10:26   4:25 /usr/sbin/asterisk -f -C /etc/asteris

=> memory before self-restart of asterisk:
Mon Sep 14 13:26:09 CEST 2020
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
asterisk 21366 18.0 86.9 8731732 7099416 ?     Ssl  10:26  32:20 /usr/sbin/asterisk -f -C /etc/asteris

=> asterisk restart, after "lack of memory"
[Sep 14 13:37:24] Asterisk 16.13.0 built by root @ svast20 on a x86_64 running Linux on 2020-09-08 10:04:07 UTC
...

After playing with the dialplan it seems that "Set(JITTERBUFFER...)" causes the problem.

Please tell me, is this description helpful for you ?

By: Benjamin Keith Ford (bford) 2020-09-15 10:21:09.726-0500

Thanks for the additional information. Were you able to compile with MALLOC_DEBUG enabled? If so, what is the outputs of {{memory show summary}} when Asterisk is using that large amount of memory?

By: Stefan Ruf (stefan20) 2020-09-15 11:51:08.213-0500

compiled with MALLOC_DEBUG, doing a "hangup request all", after asterisk used 20 % of memory; no memory is freed by the hangup request.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
asterisk  3189 22.0 20.2 3382528 1651868 ?     Ssl  18:07   5:06 /usr/sbin/asterisk -f -C /etc/asterisk/asterisk.conf


 27484064 bytes in        610 allocations in file ../src/pj/pool_policy_malloc.c
   302661 bytes in        454 allocations in file /opt/swm_install/asterisk-16.13.0/include/asterisk/strings.h
     4483 bytes in         21 allocations in file /opt/swm_install/asterisk-16.13.0/include/asterisk/threadstorag
      288 bytes in          1 allocations in file acl.c
     1263 bytes in         67 allocations in file ael/pval.c
      216 bytes in          2 allocations in file app.c
      400 bytes in          3 allocations in file app_agent_pool.c
     1024 bytes in          1 allocations in file app_bridgewait.c
     1584 bytes in          3 allocations in file app_confbridge.c
    29616 bytes in          2 allocations in file app_followme.c
    36011 bytes in         10 allocations in file app_minivm.c
    10016 bytes in          2 allocations in file app_queue.c
    45502 bytes in         13 allocations in file app_voicemail.c
      856 bytes in          4 allocations in file ari/config.c
      688 bytes in          1 allocations in file ari/resource_events.c
     4188 bytes in        137 allocations in file asterisk.c
     1008 bytes in         42 allocations in file astobj2.c
     1258 bytes in         16 allocations in file astobj2_container.c
   121360 bytes in       1517 allocations in file astobj2_hash.c
      352 bytes in          2 allocations in file bridge.c
     1528 bytes in          2 allocations in file bucket.c
     1152 bytes in          6 allocations in file ccss.c
    37888 bytes in          9 allocations in file cdr.c
      608 bytes in         16 allocations in file cdr_sqlite3_custom.c
     9831 bytes in          7 allocations in file cel.c
      176 bytes in          2 allocations in file chan_bridge_media.c
  1692072 bytes in         27 allocations in file chan_iax2.c
     1916 bytes in          3 allocations in file chan_oss.c
      264 bytes in          3 allocations in file chan_phone.c
     1136 bytes in          2 allocations in file chan_pjsip.c
      176 bytes in          2 allocations in file chan_rtp.c
   114831 bytes in         54 allocations in file chan_sip.c
    37952 bytes in         14 allocations in file channel.c
       16 bytes in          1 allocations in file chanvars.c
    15608 bytes in        754 allocations in file cli.c
     7072 bytes in         48 allocations in file codec.c
    26496 bytes in          1 allocations in file codec_resample.c
   157644 bytes in         71 allocations in file confbridge/conf_config_parser.c
    91615 bytes in        665 allocations in file config.c
   129272 bytes in        666 allocations in file config_options.c
      248 bytes in          2 allocations in file core_local.c
     1244 bytes in         18 allocations in file devicestate.c
     7928 bytes in         16 allocations in file endpoints.c
     2257 bytes in         13 allocations in file features_config.c
    10320 bytes in         30 allocations in file file.c
    15396 bytes in        199 allocations in file format.c
     1432 bytes in          1 allocations in file format_cache.c
    55136 bytes in        517 allocations in file format_cap.c
1089193619 bytes (1089193619 cache) in    2992535 allocations in file frame.c
     1024 bytes in          1 allocations in file func_dialgroup.c
      608 bytes in          2 allocations in file hashtab.c
      730 bytes in          4 allocations in file http.c
      464 bytes in          3 allocations in file iax2/netsock.c
      256 bytes in          1 allocations in file iax2/provision.c
    46007 bytes in       1200 allocations in file indications.c
    24688 bytes in         13 allocations in file io.c
      304 bytes in          5 allocations in file json.c
     2296 bytes in          1 allocations in file libasteriskssl.c
    69674 bytes in        388 allocations in file loader.c
    12604 bytes in          8 allocations in file logger.c
   215073 bytes in        333 allocations in file manager.c
     1600 bytes in          1 allocations in file media_cache.c
       96 bytes in          2 allocations in file message.c
     3384 bytes in          6 allocations in file mwi.c
     1104 bytes in          2 allocations in file named_acl.c
     2560 bytes in          1 allocations in file named_locks.c
    19408 bytes in        468 allocations in file optional_api.c
      128 bytes in          1 allocations in file parking.c
   120368 bytes in        849 allocations in file pbx.c
      144 bytes in          1 allocations in file pbx_ael.c
   524875 bytes in        443 allocations in file pbx_app.c
   417728 bytes in        268 allocations in file pbx_functions.c
       30 bytes in          3 allocations in file pbx_ignorepat.c
     5390 bytes in         19 allocations in file pbx_include.c
    13888 bytes in          1 allocations in file pbx_realtime.c
       51 bytes in          1 allocations in file pbx_sw.c
      276 bytes in          8 allocations in file pbx_variables.c
      256 bytes in          2 allocations in file pjsip/cli_commands.c
       56 bytes in          1 allocations in file presencestate.c
      232 bytes in          1 allocations in file res_ari.c
      592 bytes in          1 allocations in file res_calendar.c
     3535 bytes in          7 allocations in file res_clialiases.c
      304 bytes in          1 allocations in file res_config_sqlite3.c
      376 bytes in          1 allocations in file res_fax.c
     1093 bytes in         10 allocations in file res_http_websocket.c
     1464 bytes in          2 allocations in file res_musiconhold.c
     2348 bytes in          9 allocations in file res_parking.c
    84597 bytes in        208 allocations in file res_phoneprov.c
     2096 bytes in         35 allocations in file res_pjproject.c
      192 bytes in          6 allocations in file res_pjsip.c
      592 bytes in          2 allocations in file res_pjsip/config_auth.c
      465 bytes in          2 allocations in file res_pjsip/config_global.c
       96 bytes in          1 allocations in file res_pjsip/config_system.c
     1536 bytes in          2 allocations in file res_pjsip/config_transport.c
      256 bytes in          2 allocations in file res_pjsip/location.c
      504 bytes in          1 allocations in file res_pjsip/pjsip_cli.c
     3449 bytes in         10 allocations in file res_pjsip/pjsip_configuration.c
     7728 bytes in          5 allocations in file res_pjsip/pjsip_distributor.c
     1392 bytes in          3 allocations in file res_pjsip/pjsip_global_headers.c
    75984 bytes in          3 allocations in file res_pjsip/pjsip_options.c
     1432 bytes in          1 allocations in file res_pjsip/pjsip_scheduler.c
     2184 bytes in         21 allocations in file res_pjsip/pjsip_session.c
     3184 bytes in          1 allocations in file res_pjsip/pjsip_transport_events.c
     3184 bytes in          1 allocations in file res_pjsip/pjsip_transport_management.c
      109 bytes in          1 allocations in file res_pjsip_authenticator_digest.c
      338 bytes in          7 allocations in file res_pjsip_config_wizard.c
      128 bytes in          1 allocations in file res_pjsip_endpoint_identifier_ip.c
      880 bytes in          1 allocations in file res_pjsip_exten_state.c
     2048 bytes in          1 allocations in file res_pjsip_history.c
     4216 bytes in          1 allocations in file res_pjsip_logger.c
     2816 bytes in          2 allocations in file res_pjsip_mwi.c
     5139 bytes in         43 allocations in file res_pjsip_notify.c
     1536 bytes in          2 allocations in file res_pjsip_outbound_registration.c
     2048 bytes in          2 allocations in file res_pjsip_pubsub.c
      703 bytes in          4 allocations in file res_pjsip_session.c
      160 bytes in          1 allocations in file res_smdi.c
       35 bytes in          2 allocations in file res_sorcery_astdb.c
     6978 bytes in         50 allocations in file res_sorcery_config.c
     9856 bytes in          7 allocations in file res_sorcery_memory.c
     1408 bytes in          1 allocations in file res_sorcery_memory_cache.c
    11600 bytes in          5 allocations in file res_stasis.c
     1024 bytes in          1 allocations in file res_stasis_device_state.c
     3184 bytes in          1 allocations in file res_stasis_playback.c
     3184 bytes in          1 allocations in file res_stasis_recording.c
      344 bytes in          2 allocations in file res_statsd.c
    13648 bytes in          1 allocations in file res_timing_pthread.c
      120 bytes in          1 allocations in file res_timing_timerfd.c
     3327 bytes in         61 allocations in file rtp_engine.c
    52544 bytes in        359 allocations in file sched.c
      376 bytes in          4 allocations in file serializer.c
   126527 bytes in        554 allocations in file sorcery.c
    56392 bytes in        273 allocations in file stasis.c
     3240 bytes in          2 allocations in file stasis/messaging.c
    98274 bytes in         50 allocations in file stasis_cache.c
      728 bytes in          7 allocations in file stasis_cache_pattern.c
    11310 bytes in        238 allocations in file stasis_message.c
     3432 bytes in         20 allocations in file stasis_message_router.c
    37104 bytes in          2 allocations in file stdtime/localtime.c
      216 bytes in          4 allocations in file stream.c
    44288 bytes in       1487 allocations in file stringfields.c
    20288 bytes in        314 allocations in file strings.c
    69643 bytes in        234 allocations in file taskprocessor.c
      338 bytes in          9 allocations in file tcptls.c
    25016 bytes in         90 allocations in file threadpool.c
      200 bytes in          5 allocations in file timing.c
    16768 bytes in         34 allocations in file translate.c
      176 bytes in          2 allocations in file udptl.c
     5650 bytes in         70 allocations in file utils.c
  1114537 bytes in       5609 allocations in file xmldoc.c
1123107432 bytes allocated (1089193619 in caches) in 3012530 selected allocations

1123107432 bytes in all allocations
   487259 bytes in deferred free large allocations
    37097 bytes in deferred free small allocations
   524356 bytes in deferred free allocations
1123631788 bytes in all allocations and deferred free allocations


By: Benjamin Keith Ford (bford) 2020-09-17 11:55:25.095-0500

Are you able to isolate the scenario down to a simple use case? I tried spinning up multiple channels that went through the dialplan, calling {{Set(JITTERBUFFER(fixed)=200}} just to see if I could get the memory to rise, but could not get the same result. Is there a difference in memory consumption for you between {{fixed}} and {{adaptive}}? If you take out the {{JITTERBUFFER}} line in the dialplan, does memory stay at a sane level?

By: Stefan Ruf (stefan20) 2020-09-30 08:56:53.397-0500

I have reduced the dialplan to
 1. Set(JITTERBUFFER(adaptive)=default)
 2. ConfBridge(berta)
Is there a difference in memory consumption for you between fixed and adaptive? => I tried "(FIXED)=DEFAULT", "(ADAPTIVE)=DEFAULT" and "(FIXED)=200"  every scenario rises up the memory
If you take out the JITTERBUFFER line in the dialplan, does memory stay at a sane level? => Yes, without JITTERBUFFER  memory stays sane

The rising memory happens only in conjunction with muting/unmuting the channels. Plesae refer to attached file ami_commands.txt. Before passing actionid: 1705904476_335#berta-ddab9ea6-dcb4-4e83-b9bf-ff5e44325110 all works in a sane state. After handling this command, the memory begins to rise.

By: Friendly Automation (friendly-automation) 2021-03-03 10:07:59.627-0600

Change 15538 merged by Friendly Automation:
channel: Fix memory leak in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15538|https://gerrit.asterisk.org/c/asterisk/+/15538]

By: Friendly Automation (friendly-automation) 2021-03-03 10:14:57.140-0600

Change 15537 merged by Joshua Colp:
channel: Fix memory leak in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15537|https://gerrit.asterisk.org/c/asterisk/+/15537]

By: Friendly Automation (friendly-automation) 2021-03-03 10:15:05.497-0600

Change 15536 merged by Joshua Colp:
channel: Fix memory leak in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15536|https://gerrit.asterisk.org/c/asterisk/+/15536]

By: Friendly Automation (friendly-automation) 2021-03-03 10:15:13.095-0600

Change 15556 merged by Joshua Colp:
channel: Fix memory leak in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15556|https://gerrit.asterisk.org/c/asterisk/+/15556]

By: Friendly Automation (friendly-automation) 2021-03-10 11:07:46.229-0600

Change 15605 merged by George Joseph:
channel: Fix crash in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15605|https://gerrit.asterisk.org/c/asterisk/+/15605]

By: Friendly Automation (friendly-automation) 2021-03-10 11:07:59.551-0600

Change 15606 merged by George Joseph:
channel: Fix crash in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15606|https://gerrit.asterisk.org/c/asterisk/+/15606]

By: Friendly Automation (friendly-automation) 2021-03-10 11:08:12.206-0600

Change 15604 merged by George Joseph:
channel: Fix crash in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15604|https://gerrit.asterisk.org/c/asterisk/+/15604]

By: Friendly Automation (friendly-automation) 2021-03-10 11:08:35.724-0600

Change 15621 merged by George Joseph:
channel: Fix crash in suppress API.

[https://gerrit.asterisk.org/c/asterisk/+/15621|https://gerrit.asterisk.org/c/asterisk/+/15621]