[Home]

Summary:ASTERISK-24627: Can't take more than 11 voicemails simultaneously when using realtime
Reporter:Ryan Rittgarn (rrittgarn)Labels:
Date Opened:2014-12-16 18:28:25.000-0600Date Closed:2015-04-07 20:23:10
Priority:CriticalRegression?
Status:Closed/CompleteComponents:Applications/app_voicemail/ODBC
Versions:11.15.0 Frequency of
Occurrence		Constant
Related
Issues:
Environment:Debian Wheezy 8 cores, 16GB RAM, 500GB HDAttachments:( 0) context.txt
( 1) generate.sh
( 2) myDebugLog
Description:When using Realtime (via Res_ODBC) for voicemail storage, the system will not save more than 11 voicemails concurrently. Messages 1-10 and whatever the last message that comes in at that time is will save, any between are lost with no error. Reproduced on self contained environments as well as with an external DB Server (Dell R420, 64GB RAM, 500GB HD, 24 Cores - MySQL Tuned to utilize the system)

Issue is also present in the Asterisk11.6-certified8 build
Comments:By: Rusty Newton (rnewton) 2014-12-16 18:35:10.167-0600

We require a complete debug log to help triage the issue. This document will provide instructions on how to collect debugging logs from an Asterisk machine for the purpose of helping bug marshals troubleshoot an issue: https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information


By: Rusty Newton (rnewton) 2014-12-16 18:36:01.817-0600

Please attach the log mentioned above and provide instructional steps for reproduction of the issue.
By: Rusty Newton (rnewton) 2014-12-16 18:36:22.405-0600

Press Send Back or Enter Feedback when you have the requested information attached. Thanks!
By: Ryan Rittgarn (rrittgarn) 2014-12-17 12:33:11.403-0600

Attached is the output from leaving 12 simultaneous voicemails via the generate.sh script that i am attaching as well.
By: Ryan Rittgarn (rrittgarn) 2014-12-17 12:33:59.942-0600

This is the script I used to generate call files to leave voicemails. Takes 1 argument, the number of call files you want to leave.
By: Ryan Rittgarn (rrittgarn) 2014-12-17 12:35:48.525-0600

Relevant Dialplan pieces that are called via generate.sh
By: Ryan Rittgarn (rrittgarn) 2014-12-17 12:38:11.047-0600

Steps to reproduce, on an 11 system, run the generate.sh with any number greater than 11 as the only argument. The script will generate call files that will leverage the attached dialplan X number of times. Once you are past 10, anything between 10 and your last VM will not save, however the last VM will properly save. I have tested this with 12,15,20 simultaneous voicemails. Each time you will get voicemails 1-10, and the last VM, so 12, 15, and 20

Also, thank you for looking into this issue.
By: Ryan Rittgarn (rrittgarn) 2014-12-17 13:34:17.208-0600

Uploaded requested items and descriptions
By: Matt Jordan (mjordan) 2014-12-24 14:36:33.674-0600

From this snippet in your log file:

{noformat}
[2014-12-17 11:25:39] DEBUG[25583][C-0000001c] res_odbc.c: odbc_release_obj2(0x7f5d0c010698) called (obj->txf = (nil))
[2014-12-17 11:31:23] Asterisk 11.6-cert8 built by root @ deb-MerchantsSTT-Bench on a x86_64 running Linux on 2014-12-16 20:34:08 UTC
[Dec 17 11:31:23] VERBOSE[25811] config.c:   == Parsing '/etc/asterisk/logger.conf': Found
{noformat}

You are almost certainly running into a crash.
By: Matt Jordan (mjordan) 2014-12-24 14:36:45.773-0600

Thank you for your bug report. In order to move your issue forward, we require a backtrace[1] from the core file produced after the crash. Also, be sure you have DONT_OPTIMIZE enabled in menuselect within the Compiler Flags section, then:

make install

After enabling, reproduce the crash, and then execute the backtrace[1] instructions. When complete, attach that file to this issue report.

[1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace


By: Ryan Rittgarn (rrittgarn) 2014-12-26 09:30:40.182-0600

I have recompiled with DONT_OPTIMIZE and BETTER_BACKTRACES, I then manually launched asterisk with asterisk -vvvvvg -c in /tmp/ast, re-ran my test, received the same results - got messages 1-10, and final message, but none between 10 and final message.

I then looked in the directory i called asterisk from (/tmp/ast) and did not find  any files. So I believe that either I am doing something wrong (likely), or it is not generating a crash file. Is there something else I am missing?
By: Rusty Newton (rnewton) 2015-01-09 19:41:53.354-0600

Well, lets make sure it is crashing. Can you run "core show uptime" on the Asterisk CLI before you start your test, and then run it after your test finishes?


By: Ryan Rittgarn (rrittgarn) 2015-01-14 12:56:30.072-0600

I ran my same test and got the following:

deb-MerchantsSTT-Bench*CLI> core show uptime
System uptime: 17 hours, 41 minutes, 1 second
Last reload: 17 hours, 41 minutes, 1 second

[[Lots of call output and tt-monkeys]]

deb-MerchantsSTT-Bench*CLI> core show channels verbose
Channel              Context              Extension        Prio State   Application  Data                      CallerID        Duration Accountcode PeerAccount BridgedTo          
0 active channels
0 active calls
26 calls processed
deb-MerchantsSTT-Bench*CLI> core show uptime
System uptime: 17 hours, 47 minutes, 37 seconds
Last reload: 17 hours, 47 minutes, 37 seconds

So it does not look like its crashing.

By: Ryan Rittgarn (rrittgarn) 2015-02-12 13:00:12.487-0600

Still wondering what needs to be done to track this issue.

Thanks
By: Rusty Newton (rnewton) 2015-02-26 17:57:28.182-0600

Ryan you didn't use "Send Back" or "Enter Feedback" so the issue state never changed. It won't pop up in the filters the bug marshals use without you doing this.

I'm thinking we'll have to reproduce the issue to investigate it. Can you provide your voicemail and any other relevant configuration files? Perhaps the odbc related files?  Of course, sanitize before posting.
By: Rusty Newton (rnewton) 2015-04-07 20:21:51.253-0500

Despite not hearing back from you - the scenario looked buggy and since you provided a script and dialplan I went ahead and attempted to reproduce the issue. I had to produce my own voicemail.conf and add one additional context in dialplan to get your script working. I installed all the mysql and odbc packages on Ubuntu 12.04 and setup the voicemail database table using Alembic and the alembic config included with Asterisk.

I could not reproduce the issue.

I was able to record 15 voicemails simultaneously through res_ODBC into a MySQL backend in Asterisk 11 r434260 and Asterisk 13 r434282. I verified the presence and integrity of each voicemail, especially between 10 and 15. All were recorded and sounded fine.

If you can reproduce the issue at will, then there must be a difference between our configurations somewhere. Since I haven't heard back from you I'm going to go ahead and close this out. You can contact me or another bug marshal in #asterisk-bugs or #asterisk-dev on irc.freenode.net to have us re-open the issue if you can provide further information on how to reproduce the issue.