[Home]

Summary:ASTERISK-28943: Asterisk can't start with the errors media_cache_item_populate_from_astdb: Unable to obtain information for file /tmp/...
Reporter:Volodymyr Dotsenko (volodymyrdotsenko@gmail.com)Labels:
Date Opened:2020-06-10 10:12:03Date Closed:
Priority:MajorRegression?
Status:Open/NewComponents:Core/General
Versions:16.5.0 Frequency of
Occurrence		Constant
Related
Issues:
Environment:Attachments:( 0) cache.log
Description:We heavily used media files that asterisk played via ARI by HTTP. As I understand asterisk caches the file in /tmp directory, and because we have a lot of such files we periodically clean the /tmp folder. The problem arises after restart the server and asterisk restart takes a very long time with error: "media_cache_item_populate_from_astdb: Unable to obtain information for file /tmp/bucket-tdh5CZ.wav for URI http://.."
Today our asterisk is still starting already 5 hours.
Comments:By: Asterisk Team (asteriskteam) 2020-06-10 10:12:05.177-0500

The severity of this issue has been automatically downgraded from "Blocker" to "Major". The "Blocker" severity is reserved for issues which have been determined to block the next release of Asterisk. This severity can only be set by privileged users. If this issue is deemed to block the next release it will be updated accordingly during the triage process.
By: Asterisk Team (asteriskteam) 2020-06-10 10:12:05.857-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.
By: George Joseph (gjoseph) 2020-06-10 13:41:39.896-0500

Is asterisk continually printing those "Unable to obtain information" messages while its starting up?   How many files are in the cache in astdb (you can check with "media show cache all")? How many are in the tmp directory?

While asterisk is stuck starting up, can you get us a coredump?
See https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-Runningast_coredumperfordeadlocks,taskprocessorbackups,etc.
for information on how to use ast_coredumper to get the coredump, then attach the txt files to this issue.

By: Volodymyr Dotsenko (volodymyrdotsenko@gmail.com) 2020-06-11 11:01:00.518-0500

Asterisk log
By: Volodymyr Dotsenko (volodymyrdotsenko@gmail.com) 2020-06-11 11:05:19.653-0500

Yes, asterisk continually printing those errors, we have about  2000 media files in the cache and remove them every 1 hour, this time we didn't restart our server almost 2 weeks so I believe the cache contained a huge amount of data but the /tmp dir almost empty.
By: Sean Bright (seanbright) 2020-06-11 16:03:33.934-0500

I just tested this with 5000 entries that did not exist on disk and it took Asterisk about 1 minute to start, so I am not sure what is taking 5 hours for you. Is your /tmp directory tmpfs or is it actually stored on disk?
By: Volodymyr Dotsenko (volodymyrdotsenko@gmail.com) 2020-06-11 16:24:25.589-0500

I'm not exactly sure about the number of files. It could be more than 50k, all files asterisk read from an external server using HTTP, played a file, and then the file was deleted. Are you able to reproduce just the same case as I've described? Directory stored on a disk.
By: Sean Bright (seanbright) 2020-06-12 15:06:35.187-0500

Tested with 50k and it took about 40 minutes for Asterisk to start. It's an astdb performance issue.  I think the performance would generally be the same regardless of if the cache files exist or not.
By: Volodymyr Dotsenko (volodymyrdotsenko@gmail.com) 2020-06-14 08:00:58.972-0500

Is there any workaround for this? I think Asterisk should forget any cache after restart or should be a way how to make Asterisk clean it up.
By: Kevin Harwell (kharwell) 2020-06-15 12:15:01.700-0500

The code just checks to see if the file exist locally. If it doesn't then the entry is removed from the astdb. I'm not sure if it would be any faster, but as a workaround you could try removing items from the database manually using CLI commands [1]. Maybe in smaller chunks over time from a separate thread/process.

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Internal+Database
By: Volodymyr Dotsenko (volodymyrdotsenko@gmail.com) 2020-06-15 13:41:40.020-0500

Thanks a lot for your advice. Do you have any plan to fix it somehow, for example, to perform these checks in a separate thread?
By: Kevin Harwell (kharwell) 2020-06-16 15:13:39.548-0500

I've opened this, and your issue is now in the queue. There are currently no plans as to a fix at this time. However, a developer may work the issue when time and resources become available.

Asterisk is an open source project and community members work the issues on a voluntary basis. You are welcome to develop your own patches and submit them to the project [1], and/or discuss development approaches on IRC or the asterisk-dev mailing list [2].

If you are not a programmer and you are in a hurry to see a patch provided then you might try rallying support on the Asterisk users mailing list or forums.[2] Another alternative is offering a bug bounty on the asterisk-dev mailing list.[3] Often a little incentive can go a long way.

[1]: https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process
[2]: http://www.asterisk.org/community/discuss
[3]: https://wiki.asterisk.org/wiki/display/AST/Asterisk+Bug+Bounties
By: Kevin Harwell (kharwell) 2020-06-16 15:17:40.104-0500

Also I noticed there is an Asterisk CLI command for removing an item from the cache:
{noformat}
*CLI> media cache delete
      Usage: media cache delete <uri>
      Delete an item from the media cache.

      Note that this will also remove any local storage of the media associated
      with the URI, and will inform the backend supporting the URI scheme that
      it should remove the item.
{noformat}
Instead of deleting the file directly from the file system, and having Asterisk "clear" the cache of orphaned records on startup you could delete the item instead directly from the cache while Asterisk is running, thus alleviating the need to restart and clear the cache.
By: Volodymyr Dotsenko (volodymyrdotsenko@gmail.com) 2020-06-16 16:15:14.922-0500

Thanks a lot for the information. We mainly use Asterisk ARI so it would be great to have such command in the API but I understand it is a feature request.
Also, I'd like to contribute but I'm not a C developer I'm a Java developer, so I'd really appreciate if you point me the doc where I can find information on how to build a development environment on a local machine.
By: Kevin Harwell (kharwell) 2020-06-16 17:58:26.410-0500

Sure thing! Check out this page to get your started:

https://wiki.asterisk.org/wiki/display/AST/Gerrit+Usage

Also, the [CLI command|https://github.com/asterisk/asterisk/blob/master/main/media_cache.c#L579] I mentioned above is just a wrapper around the [ast_media_cache_delete|https://github.com/asterisk/asterisk/blob/master/include/asterisk/media_cache.h#L159] API call. So if removing the item from the cache via ARI is all that you need you can probably find how a similar call is handled and do similar.

Be sure to query the IRC channel and/or the asterisk-dev mailing list if you need any help:

http://www.asterisk.org/community/discuss