Summary:	ASTERISK-16578: Record audio channels gets out of sync
Reporter:	Marco Aurelio (aureliors123)	Labels:
Date Opened:	2010-08-16 22:16:33	Date Closed:	2011-09-22 07:32:33
Priority:	Minor	Regression?	No
Status:	Closed/Complete	Components:	Applications/app_mixmonitor
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) 620.WAV ( 1) configs.txt ( 2) scenario.jpg
Description:	When recording with mixmonitor, some record files channels gets out of sync. The scenario was: - Medium call traffic: More than 60 and less than 120 concurrent calls. - All calls are recorded - Codec: Alaw - Record File format: WAV GSM - Processor load: Never more than 50% - Asterisk Version: 1.6.2.10 (also happened in older versions, since 1.4.19) - SO Debian 5.0.3 - 2.6.26-2-686 ****** ADDITIONAL INFORMATION ****** - The call channels gets out of sync. The customer answers before the agent´s question. - The same issue can be reproduced using chanspy. After sometime monitoring one agent, the channels gets ou of sync. - Not tryed the same scenario recording WAV format. - Not tryed recording WAV GSM format with no call trafic. - Also realized the same problem at another customers, with different hardware and Asterisk version (older than 1.4.19)
Comments:	By: Marco Aurelio (aureliors123) 2010-08-16 22:22:00 Added audio file that reproduces the issue. By: Marco Aurelio (aureliors123) 2010-08-16 22:30:10 I´ve read at another issue report about the same problem, and saw that someone told that the problem was with de client´s Sound Card. I can´t imagine know how the sound card can be the source of this problem, when both customer and agent does not realize any problem (the issue occurs only with the record). At the scenario of this report we´re using USB Head Sets. By: Leif Madsen (lmadsen) 2010-08-17 14:02:23 As with the last issue please provide the dialplan, sip.conf, and scenario so we can reproduce the issue. I'd also be curious to know what kind of system you're using. By: Marco Aurelio (aureliors123) 2010-08-17 18:03:59 Just added the config files and an scenario png file. Please tell me what exactly informations you want to know about our system. By: Marco Aurelio (aureliors123) 2010-08-30 23:08:17 Please, any news about this issue? It is happening with the same frequency of the reported issue ASTERISK-1740655 By: Leif Madsen (lmadsen) 2010-08-31 14:48:50 We mean what CPU, platform, etc... is the system built on. By: Marco Aurelio (aureliors123) 2010-09-16 16:44:36 ########################################## It is not runnig under virtualization CPU Intel Xeon E3110 1 Gb Ram 250Gb Sata Disk (10% used disk space) SO Debian 5.0.6 - Lenny Kernel 2.6.26-2-686 Asterisk 1.6.2.10 built by root @ ISRAEL-GW3 on a i686 running Linux on 2010-09-09 18:42:14 UTC ########################################## Please tell me if you need more information. By: Ronald Chan (loloski) 2011-01-29 22:04:55.000-0600 any progress on this? i'm experience it also on a very low hardware Intel Atom, but i think it's irrelevant since others is also experiencing on a high hardware specs. By: iasgoscouk (iasgoscouk) 2011-01-30 12:06:52.000-0600 I have been experiencing mixmonitor out-of-sync issues on/off over a long period (12 months) but never been able to track why, as was only using mixmonitor when needed. In december 2010, I started using mixmonitor for all my calls.  When I realised that I had ths problem again, last week, I checked all recorded calls and found that the problem re-occured on January 11th 2011, and only 'incoming calls using dahdi' are affected.  I am currently using SVN, and take a new checkout every few days. For me, the audio from the incoming caller includes the 'ringing' they would hear whilst I answer,  but my audio starts from me answering, so their conversation is always behind mine, and makes to recording no use to listen to. So i reverted back to an SVN release ( I keep a dated history of svn builds) to before that date, expecting calls to be OK, but they weren't.   So wasn't sure whether it's related to fedora updates etc ( on HP intel hardware running Linux 2.6.35.10-74.fc14.i686.PAE). Looked back at previous mixmonitor issues, and there has been issues in the past for this and not sure if this is the same type of issue.   I am in the UK, and only use Asterisk for personal use, and not been able to rely on mixmonitor for incoming calls make the option to keep to record of incoming dahdi calls a problem for me.     Currently, on 'early-retirement' as a 'web developer/IT manager' with plenty of time on my hands to try and help to sort this out. If this is not the same issue, please could you let me know and will raise a separate 'bug/issue'. thanks By: Ronald Chan (loloski) 2011-01-30 20:39:41.000-0600 indeed, this was a problem. my setup also involves PRI/FXO -> Queue -> FXS  no SIP involved on my setup. the problem is pretty much the same as describe by the original reporter. I'm just wondering if Monitor will fixed the issue on my end. By the way i haven't tried to store the recording temporarily on memory /dev/shm then later on move to a real disk. i'll be back to this if this setup can alleviate the situation. if the issue's has something to do with I/O then i'm pretty sure this will fix it. Regards, Ronald By: Marco Aurelio (aureliors123) 2011-01-31 07:38:10.000-0600 Ronald, we´ve tried saving the conversation to a RAM Disk and them moving the record to the phisical disk, but the problem still happening. Also, we tried to use Khomp channels (Excelent Brazilian manufacturer), and again we had problems with audio sync at the record file. The scenario is inbound and outbound, and the problem happens at both kind of calls, but much more frequently at inbound calls. I´ve not been able to test it, but i know that the record audio sync problems started at V1.4.17 or V1.4.18. I realized it when upgrading to newer versions solve SIP stuck chans issue. I hope this can be a clue. Rgds Marco Aurelio By: Ronald Chan (loloski) 2011-01-31 11:04:33.000-0600 Marco, Thank you for jumping on the gun ahead :D, i hope this problem will be fixed soon, since this feature is very important for many of us, I hope digium and other developers can pick this up soon. Regards By: iasgoscouk (iasgoscouk) 2011-02-01 13:08:34.000-0600 am relieved to see the notes that have been posted against this issue after me, and not the only one who has been suffering with similar problems. I saw a post on asterisk-users from a guy asking how to record calls for a call centre, as worries me to think that in all cases it's not working, and turning into a critical requirement in this instance. This issue has not been updated by digium/asterisk, and hope that they provide some feedback soon to this.   Am sure that all 'reporters' who have added notes to this issue are waiting for this to be sorted. thanks IAN By: Marco Aurelio (aureliors123) 2011-02-01 13:53:30.000-0600 As a matter of fact this issue turned very serious for me, as the implementations we have made here grew fast. At this moment we have more than 1500 ports running Asterisk with recording feature. Thank you guys for the contribution, and sorry about the poor english :) Marco By: iasgoscouk (iasgoscouk) 2011-02-01 14:13:22.000-0600 Marco,  i am with you 100% - and your english is OK. We all trying to sort this out - as all our environments/installations are different, but recording calls is a major part of communications now, and can, in most cases, be needed, and when you do, if the recording is not usable, it makes this facilty useless. As I said, just hoping that Digium are aware of this. sorry (to digium) for posting 'personal/non techincal' comments on a 'fault/issues' log, but this is getting a problem which needs to be fixed. IAN By: iasgoscouk (iasgoscouk) 2011-02-06 14:50:09.000-0600 has anyone had any update on this ? By: iasgoscouk (iasgoscouk) 2011-02-08 11:25:22.000-0600 Today, my problem appears to have resolved itself. Nothing has changed in my asterisk installation, but there was a 'YUM' updates.   the updates were : libblkid                      i686      2.18-4.8.fc14         updates    116 k libmount                      i686      2.18-4.8.fc14         updates     81 k libuuid                       i686      2.18-4.8.fc14         updates     63 k libuuid-devel                 i686      2.18-4.8.fc14         updates     72 k perl                          i686      4:5.12.3-141.fc14     updates     11 M perl-CPAN                     noarch    1.9402-141.fc14       updates    249 k perl-Digest-SHA               i686      1:5.47-141.fc14       updates     61 k perl-ExtUtils-MakeMaker       noarch    6.56-141.fc14         updates    292 k perl-ExtUtils-ParseXS         noarch    1:2.2100-141.fc14     updates     43 k perl-Module-Pluggable         noarch    1:3.90-141.fc14       updates     38 k perl-Pod-Escapes              noarch    1:1.04-141.fc14       updates     31 k perl-Pod-Simple               noarch    1:3.13-141.fc14       updates    210 k perl-Test-Harness             noarch    3.17-141.fc14         updates    242 k perl-devel                    i686      4:5.12.3-141.fc14     updates    477 k perl-libs                     i686      4:5.12.3-141.fc14     updates    612 k perl-threads-shared           i686      1.32-141.fc14         updates     51 k perl-version                  noarch    3:0.82-141.fc14       updates     51 k postgresql-libs               i686      8.4.7-1.fc14          updates    196 k sssd                          i686      1.5.1-2.1.fc14        updates    830 k sssd-client                   i686      1.5.1-2.1.fc14        updates     50 k system-config-printer         i686      1.2.6-3.fc14          updates    483 k system-config-printer-libs    i686      1.2.6-3.fc14          updates    742 k util-linux-ng                 i686      2.18-4.8.fc14         updates    1.5 M xen-libs                      i686      4.0.1-7.fc14          updates    386 k xen-licenses                  i686      4.0.1-7.fc14          updates     60 k I looked back at my yum logs from when I think the problem started again, and there was an update to sssd. 'Simple Sound for Small Devices library' - which was updated today, and now my issues have been resolved. I hope that helps someone else to find out what the issue might be thanks IAN By: Ronald Chan (loloski) 2011-02-09 07:48:05.000-0600 Ian, Are you sure this fixed your issue, on centos 5.5 there was no even sssd package available. [root@pbx ~]# rpm -qa | grep sssd [root@pbx ~]# cat /etc/issue CentOS release 5.5 (Final) Kernel \r on an \m [root@pbx ~]# ldd /usr/sbin/asterisk linux-gate.so.1 =>  (0x00ed8000) libssl.so.6 => /lib/libssl.so.6 (0x003b8000) libcrypto.so.6 => /lib/libcrypto.so.6 (0x06864000) libc.so.6 => /lib/libc.so.6 (0x0085f000) libxml2.so.2 => /usr/lib/libxml2.so.2 (0x03d4d000) libz.so.1 => /usr/lib/libz.so.1 (0x00ce1000) libm.so.6 => /lib/libm.so.6 (0x009ba000) libdl.so.2 => /lib/libdl.so.2 (0x00ca9000) libcap.so.1 => /lib/libcap.so.1 (0x00c9a000) libpthread.so.0 => /lib/libpthread.so.0 (0x009ec000) libtermcap.so.2 => /lib/libtermcap.so.2 (0x00ca0000) libresolv.so.2 => /lib/libresolv.so.2 (0x00dd9000) libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00402000) libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00320000) libcom_err.so.2 => /lib/libcom_err.so.2 (0x00cb0000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00d2d000) /lib/ld-linux.so.2 (0x00840000) libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0x0023f000) libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x00def000) libselinux.so.1 => /lib/libselinux.so.1 (0x00203000) libsepol.so.1 => /lib/libsepol.so.1 (0x00d91000) Asterisk doesn't link to a library pertaining to sssd. I might be wrong too, anyway thanks for your your info. will continue to dig into it, worst case scenario i will install also create a development machine based on FC 14 like you do for testing purposes. Regards, Ronald By: iasgoscouk (iasgoscouk) 2011-02-09 07:57:33.000-0600 thanks for the reply, i was just hoping it might be a help for others with similar issues.  The sssd package only appears to be the common link on receiving updates for fedora which created a problem for me in january and then fixed it again yesterday.  It definately fixed my out of sync issue. Am hoping that digium/asterisk might update this issue with their thoughts too. Will keeping mixmonitor on for all my calls and checking everyday.   If my issue appears again, i will be ready/prepared to check everything this time. P.s. just updated this  note !  i monitor my 5060 port attempts to my server/domain via my firewall, and since this mornining they have increased !  not being paranoid but either the RIPE IPs are being attacked again or something else is going on !  lol !  good job i have a firewall thanks Ian By: Ronald Chan (loloski) 2011-02-09 19:56:30.000-0600 @Ian, Luckily you have one, I'm really glad that your issue has been fixed automagically :), this issues has been existed for us on all major * version, with varying hardware config. my Intel Atom pbx at home also affected with this issue :D. @aureliors123 would you be able to fix your issue on this ? Thanks By: Marco Aurelio (aureliors123) 2011-02-10 04:21:03.000-0600 Ronald, we are already planning try this fix, as soon as posible. I hope next week can feed back you all about this. Ian, thanks for your updates. Marco Aurelio By: iasgoscouk (iasgoscouk) 2011-02-11 12:08:39.000-0600 @loloski i'm not sure why this fixed it for me.  I tend to update OS and Apps etc as soon as things are available and therefore seen changes in asterisk and fedora.   I was using asterisk SVN, but recently found a number of issues which are obviously causing more important issues with digium/asterisk, that i have reverted back to 1.8.3-rc2 2 days before I had the Fedora patches which appeared to fix my out of sync problem.  So in a more stable environment now.   I only use asterisk for personal use, as now taken early IT retirement, but am a little astonished that digium/asterisk has still not helped any more. But the way this issue exists across all platforms and versions, there must be something somewhere which can explain this and to me appears to be therefore an OS issue which asterisk is susceptible too. So far as of today, and taken a new fedora 14 kernel today, too, my mixmonitor recordings are still ok . I will continue to monitor this issue and help, if I can ---- thanks to all IAN By: Ronald Chan (loloski) 2011-02-11 14:46:26.000-0600 @ian, Thanks for your feedback, today i'm gonna try to replicate this issue on my personal pc, using centos 5.5 side by side with more bleeding edge distro fedora with the same dialplan and config and asterisk version. If there's an improvement i will hack my way through to compile my own vanilla kernel the same kernel version as shipped with FC 14 to centos 5.5 and will see how it goes. This is really troublesome because this feature was one of the main reason why we got a few dozen of asterisk installation. Regards, Ronald By: iasgoscouk (iasgoscouk) 2011-02-15 13:10:50.000-0600 @loloski Thanks Ronald - i hope we get somewhere with this as digium still won't update the problem. So far, for me, all my mixmonitor calls are ok, and had no further problems. But from your last comments, i do now remember, that when I had fedora 13, i had issues with mixmonitor, and when i moved to fedora 14, the issue was fixed.   So, I've put sssd and also alsa into my 'yum' excludes. For me - i have prob this this issue come and go as I try to keep up-to-date with the latest versions etc, but now, after I now got it working again, am monitoring all updates for fedora very carefully. Ian By: Ronald Chan (loloski) 2011-02-15 20:47:37.000-0600 Updates: ubuntu lucid 10.04 half fix the issue for me, there were some situation that the audio is a little bit behind around 3 seconds, contrast to centos 5.x it's reproducible everytime, my venture on FC 14 has also improve the situation. the only permutation that i haven't test or tried is to backport some reasonable recent kernel to Centos 5.x base. will explore this in a matter of days. System Specs personal PBX Intel Atom D945GCLF-2 2 gb ram. Regards, Ronald By: Marco Aurelio (aureliors123) 2011-02-17 11:44:02.000-0600 @Ian and Ronald We are using Debian at our servers, and this library does not seems to exist at Debian´s repository. Besides this, as far we know, this library is not necessary to run Asterisk and mixmonitor feature, so it is weird for us that this library can be the solution for the problem. Look this: DUBAI-GW2:/usr/src/libsssd/src# ldd /usr/sbin/asterisk        linux-gate.so.1 =>  (0xb7792000)        libssl.so.0.9.8 => /usr/lib/i686/cmov/libssl.so.0.9.8 (0xb7743000)        libcrypto.so.0.9.8 => /usr/lib/i686/cmov/libcrypto.so.0.9.8 (0xb75f0000)        libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb7494000)        libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb7490000)        libpthread.so.0 => /lib/i686/cmov/libpthread.so.0 (0xb7477000)        libncurses.so.5 => /lib/libncurses.so.5 (0xb7445000)        libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb741f000)        libresolv.so.2 => /lib/i686/cmov/libresolv.so.2 (0xb740b000)        libz.so.1 => /usr/lib/libz.so.1 (0xb73f5000)        /lib/ld-linux.so.2 (0xb7793000) Even this way we have tried to install the libsssd package. We´ve downloaded from http://sourceforge.net/projects/libsssd/ and got an error while trying to install. We think that can be something related with sound card (our servers does not have an sound card) DUBAI-GW2:/usr/src/libsssd/src# make && make install cc -O2 -Wall -D_REENTRANT -D__LINUX__ -D__NOPRINTING__ -Iresample/ -D_INCLUDE_ESD   -c -o startAudio.o startAudio.c startAudio.c:18:19: error: esd.h: No such file or directory startAudio.c: In function âsssd_startAudioâ: startAudio.c:99: error: âESD_BITS16â undeclared (first use in this function) startAudio.c:99: error: (Each undeclared identifier is reported only once startAudio.c:99: error: for each function it appears in.) startAudio.c:100: error: âESD_BITS8â undeclared (first use in this function) startAudio.c:102: error: âESD_STEREOâ undeclared (first use in this function) startAudio.c:103: error: âESD_MONOâ undeclared (first use in this function) startAudio.c:105: error: âESD_PLAYâ undeclared (first use in this function) startAudio.c:105: error: âESD_STREAMâ undeclared (first use in this function) startAudio.c:107: warning: implicit declaration of function âesd_play_streamâ make: *** [startAudio.o] Error 1 Regards, Marco Aurelio By: iasgoscouk (iasgoscouk) 2011-03-05 02:40:32.000-0600 To update anyone who is 'monitoring' this issue. Over the last few weeks, I've had no problems since my out of sync issues were resolved following linux updates, and taking a very pragmatic approach to fedora and asterisk updates.   Following any updates, a full shutdown and cold startup performed followed by a test call to check on the mixmonitor files. Until today, everything has been ok.   On 4th March at 7:47 i installed the latest DAHDI 2.4.1 (both Linux and tools), and shutdown/restart/test call and all was ok. I've been using the SVN versions of asterisk for the last week, and everything has been ok;   at 08:26 i took the SVN 309676, and followed the usual process and everything was ok. At 17:20 i took fedora updates : augeas-libs         i686         0.8.0-1.fc14            updates         301 k graphviz            i686         2.26.3-2.fc14           updates         970 k libpurple           i686         2.7.10-1.fc14           updates         6.2 M and again, everything was ok. I made/received a number of calls after then, and all mixmonitor files are ok. Then, between a space of 10 mins, withing nothing occuring, an incoming call now contains 'out of sync' in/out channels. So am completely dumbfounded now ! Ian By: iasgoscouk (iasgoscouk) 2011-03-06 02:35:29.000-0600 Discovered that between the calls been ok and not ok my server performed a ntpd remote time update;  been using ntpd on my server for a while, and sync all my server/pc/Voip device clocks to the server.   Am wondering whether altering the system time via the sec drifts, will affect timings with asterisk. Seen issues raised before with dates etc changing whilst asterisk is running. Are of you that are affected by this also use server NTPD remote syncing/time updates ? I have stopped using ntpd for now, and started to experiment in trying each of the asterisk timing methods to see if this has any affect. Starting with res_timing_dahdi, as have dahdi installed, but asterisk appears to choose res_timing_timderfd Ian By: Ronald Chan (loloski) 2011-03-06 04:07:22.000-0600 I was under an impression that this NTPD thing has nothing to do with it why?, simply because most of our server doesn't use ntpupdate. we don't sync time from the internet just local clock. anyway thank for the update. By: iasgoscouk (iasgoscouk) 2011-03-06 04:17:31.000-0600 @loloski OK - was just trying to put down my findings - as getting fed up with this issue now, and specially as asterisk/digium are offering no support. Going to stop monitoring this issue now as doesn't look like it will ever get fixed. Ian By: Ronald Chan (loloski) 2011-03-06 11:55:16.000-0600 @Ian, To be honest i'm with you i never saw this coming, that this important feature hasn't got any attention. a lot of deployment is depending on this feature because even in our country it's a prerequisite once you are in a project, you can't just simply say hay we have a very good feature, can we come in now? Sorry for Off topic comment, it's really spook me we are engaging for a major bank here and this is one of their highest requirement, we are willing to help to the best that we can, even to pay for someone who can have this fixed and put this to end. By: iasgoscouk (iasgoscouk) 2011-03-06 12:35:10.000-0600 @loloski I've been looking through the issues, after saying I was giving up monitoring this issue, that my issue appears to be related to 13745 specific to ZAP, now Dahdi, and AudioHook, which has 2 other related issues related to it.   All of these issues are either 'closed' or still open without resolution. Interesting though, that I did stop monitoring this issue, but still got the email on your update !  so that means there is also an issue with the 'issues application' lol Ian By: ornix (ornix) 2011-05-03 21:00:19 I have the same issue with 1.8.4rc3 on two different systems. 1) Debian 5.0 / 2.6.36, Athlon 64 X2 3800+ 2) Debian 6.0 / 2.6.32, Intel Celeron 2.53GHz. When i've installed there 1.6.2.18, MixMonitor became OK. By: David Vossel (dvossel) 2011-05-04 09:34:53 Has anyone been able to directly link this syncing problem with what timing module they are using.  For example, with dahdi timing this issue does not exist but with res_timerfd it fails? By: iasgoscouk (iasgoscouk) 2011-05-04 09:51:30 i've had a problem only with dahdi incoming calls record with mixmonitor where the incoming audio includes the ringing from the time it takes me to answer the call. I've tried varies timing modules, and not been able to associate it with any specific one as the problem starts and stops every few weeks, and each time cannot associate it with anything specific.   Been taken a current trunk SVN of asterisk every few days to see if anything has changed but it hasn't so far. i've just seen 0019227 issue just appeared and not sure how close this appears to be the symptom of my problem sorry cannot be anymore specific or help ian By: ornix (ornix) 2011-05-04 21:33:50 I used res_timing_timerfd everywhere. But the problem has been detected only on asterisk 1.8. By: David Vossel (dvossel) 2011-05-09 10:00:26 I would be interested in investigating this issue if I felt like I had a definite lead on how to reproduce it and where to start, but the comments on this issue are all over the place. Some of you are saying it is in 1.8 only, but then there are other reports that say it is in 1.4, 1.6.2, and 1.8.  Some of you are saying it isn't related to Asterisk at all, and that some package updates resolved it. Here are some things that would help an Asterisk developer to be able to investigate this. 1. Precise steps to reproduce the issue using a recent release.  Example.  Using this dialplan segment and these steps my audio is out of sync.   2. Determining any factor that reliably triggers the sync issue.  Example: In this scenario it works 100% of the time, but in this scenario we have problems... or, It work 100% of the time for a single call, but once we reach a certain load the audio begins to drift. Ultimately we need to find out what is the common factor among all of you who are reporting this. By: iasgoscouk (iasgoscouk) 2011-05-09 10:30:50 thanks @dvossel ( NOTE UPDATED SINCE ORIGINAL POST) I will explain my issue, but as you say there may be a number of different issues around mixmonitor and 'sync' issues. I currently have the problem on all incoming Dahdi calls, at present, which lasts for a few weeks and then goes away;   as currently on the latest svn trunk ( every few days ), this issue is still present.  I reverted back to the latest 1.8 release and the issue is still there.  I have all timing modules loaded, and during periods of having the issue, i've tried just using each of them to see if the problem goes away - and it doesn't.   All my incoming/outgoing calls via Dahdi, have been recorded using mixmonitor for the past 18 months, and every time the 'sync' issue appears to appear, I track back to see what i've changed, and each time I've done this never been able to work out what triggers it.   The issue I have, which relates to the incoming call including the ringing tones for my SIP extension ringing, is included in the file, has been a number of issues about this in the past relating to audiohooks. As I said in my previous note - I set mixmonitor from a macro which is called by all incoming and outgoing contexts. In the past once the issue 'resolves itself', i've not tried trying to 'make it happen' and don't want to loose it working again. Sorry that I've been able to give you 'the precise steps to reproduce this' as, for me and for those others who have this issue, and only happy to see it working. I reboot my server every day, and only use for personal calls. The dates when I see it going from working->not-working->working are as follows: Pre 12th Jan 2011 - mixmonitor was working. 12th Jan 2011 16:01 - mixmonitor started with issue 8th Feb 2011 09:32 - mixmonitor started to work again 4th Mar 2011 17:20 - mixmonitor started with issue 29th Mar 2011 08:05 - mixmonitor started to work again 23rd Apr 2011 07:20 - mixmonitor started with issue *** UPDATE:   have put this dates through excel and just realised that there's exactly 24.xx no of days between each of these dates I go back in time like this, and roughly every month or so, it changes from working/not-working or vice versa. Although I've not been able to give you what you asked for in 'Part 1 of your requirements' as I won't know what triggered it. I kept records on when it started/stopped working in relation my time-stamp logs of yum updates, and also against each time-stamped SVN extract of Asterisk.  I don't keep more than a few 'svn extracts' due to the large directory size as I felt that it may not be realted to asterisk, at the time, and more to OS or Time issues, but am know keeping the SVN at the time around it previous started with the issue. I am willing to offer any help I can.   Ian By: ornix (ornix) 2011-05-09 11:14:06 Now i use 1.8.3.3 with res_timing_dahdi - the problem hasn't been detected yet. I'm not a developer, but may be the problem is in res_timing_timerfd? By: iasgoscouk (iasgoscouk) 2011-05-09 11:23:48 I am working with SVN 318229 and have the problem and also tried 1.8.3.3 with dahdi_timing, and still have the issue. As an update to my UPDATE on my previous note - i use ntp time updates with internet. After only just putting the dates in to excel - which I'd never done, for me this issue changes every 24 days. By: Matthew Nicholson (mnicholson) 2011-05-09 15:05:23 Are any of you who are seeing this issue using the jitterbuffer (on sip channels)? By: iasgoscouk (iasgoscouk) 2011-05-09 15:16:48 I have checked all my configs and no reference to jitterbuffer or 'jb', so therefore must assume the answer must be 'no'. By: Matthew Nicholson (mnicholson) 2011-05-09 15:29:11 I have observed poor network conditions to cause sync problems in recordings and enabling the jitterbuffer fixes that problem.  That may not be the only thing contributing to out of sync recordings (especially since some of you are using dahdi channels). By: iasgoscouk (iasgoscouk) 2011-05-09 15:40:44 i have a 8mb connection most of the time and run router network analysis which doesn't show any bandwith issues - periodic testing shows i have full bandwith and 700Kbps.  My ISP does not 'shape' 5060 traffic.   But as you say it's dahdi traffic that's the issue via PSTN = i use a TDM400 card.   Strangely that's in incoming calls only.   I did see an issue on the log about audiohook issues with dahdi and audiohook for incoming calls =  for an outgoing call am assuming there's no bridge =   but for an incoming call, the dialplan takes the call and then transfers/bridges the call via a 'dial' command to the sip channel/extension/phone for that context. There's been alot a issues/changes recently over audiohooks.   Until I wrote my previous update to dvossel's request for info, i've never looked at the dates around when 'my' issue starts/stop mixmonitor working, and was very surprised and cannot more than a co-incidence that it's been every 24 days. I have the 19th in my diary ! Ian By: Marco Aurelio (aureliors123) 2011-05-17 14:35:00 Mnicholson, here at our implementation this problem happens sporadically, and till now we did not reach to reproduce it or find the trigger to make record channels out of sync. About your comment above, i can tell you that we are facing this problem at two implementation of ours, and in both of them the network conditions are not the best. Also, at both implementations, the Jitterbuffer is enabled. What i done now was disabling Jitterbuffer to see what happens. Will feed back you soon about disabling Jitterbuffer. Rgds Marco Aurelio By: iasgoscouk (iasgoscouk) 2011-05-18 03:05:19 @mnicholson Following on from my last update. Since providing you with the rough dates when mixmonitor would 'be ok' or 'be out of sync with garbage at the start of the callers channel', I've not made any changes to my environment or asterisk upto the next time I tought this might happen. I have taken periodic svn asterisk trunk (currently at 319421), after following the install, performed and shutdown/cold start and a test call to see the state of the mixmonitor file.   Over the last few days the issue has still be there. At 17/05/2011 18:26 i had in incoming call which resulted in having the issue;  this date/time of call is about 24.4 days since the problem started to occur again. At 18/04/2011 08:35, at my first chance this morning I performed a test incoming call, and ' mixmonitor is now ok with no sync issues'. So yet again - somewhere between 24-25 days, the issue, for me, either starts or stops. Ian By: iasgoscouk (iasgoscouk) 2011-05-18 14:29:09 @mnicholson just a further update at the 'near end' of the day (20:26), that all my incoming calls today have been 'insync' so will have to leave this with you - i cannot understand what is causing this - but my only thought is down to 'timestamping' etc and the audiohooks.   Ian By: iasgoscouk (iasgoscouk) 2011-06-12 02:13:34.772-0500 just abother update to say, that as above, somewhere between 19:00 on the 11th and 08:00 on the 12th June, mixmonitor starting to cause out of sync issues on incoming dahdi calls. Yet it again, this is bettwen 24-25 days as previously identified, of it starting/stopping have the problem. What is a little strange is that i upgraded my fedora 14 to fedora 15, and this did not affect mixmonitor issues.   But as there appears to be issues with gcc4.6 and building of chan_dahdi within asterisk, which I've not been able to work out why and report as an issue, I had to build a new disk with fedora 14 from scratch, with a fresh OS and build of dahdi/asterisk.   This did not affect the mixmonitor issue at all and was not having out of sync issues for 3 days since the new build; I was waiting to test this morning to see that, if after 24 days whether mixmonitor would begin to have out of sync recording 24 days since it started to work, on another disk/build. thanks Ian By: Marco Aurelio (aureliors123) 2011-06-14 18:52:05.315-0500 Mnicholson, about my comment at 17/May/11 2:35 PM, both enabling a disabling jitterbuffer not solved the problem at all. thanks, Marco By: Marco Aurelio (aureliors123) 2011-06-14 19:01:28.213-0500 Also created an bounty for this issue. http://www.voip-info.org/wiki/view/Asterisk+bounty+-+Asterisk+Issue+-+Record+audio+channels+gets+out+of+sync+- Marco Aurelio By: David Hajek (hajekd) 2011-06-28 18:30:11.032-0500 We have out of sync issues with latest 1.8 - we tested 1.8.4.X and 1.8.3.2 and both versions are afected. Steps to reproduce the problem: - only outbound calls using DAHDI are affected. we were not able to reproduce using SIP. We have TE420 card. - when user dial the outbound call manually, Mixmonitor recording is OK - when user using API (which just create simple .call file) then Mixmonitor recording is out of sync by the number of seconds given by the ringing tone - downgrade to 1.6.0.28 fixed the problem and Mixmonitor recording is fine even when initiate call using .call file Let me know if you need any more info. By: Matthew Wiesemann (mdw19873) 2011-06-30 09:37:48.978-0500 I can confirm issues on asterisk 1.8.3.3, 1.8.4, 1.8.4.3, and 1.8.4.4. This is occurring on ALL calls. I have no DAHDI devices installed and am using SIP only. By: khoanv (khoanv) 2011-07-05 01:52:21.356-0500 Hi all, This issue appeares on my system too. IBM Intel X3650, CentOS 5.3, * v 1.4.26.1 via source build. Call Center for ~120 agents, using SIP Trunks with an VoIP ISP, and no DAHDI or Zap device. The recording issue is only affect the outbound calls. Incomming SIP Trunks calls and Local calls between agents are fine. When dialing out, we can talk just fine, but when replaying the recorded file, I can hear the callee's voice before the caller (agent). This issue has happened for a long time ago (since built time, maybe). I tried to downgrade to 1.4.17 (the stable version as somebody suggested), upgrade to 1.4.42, 1.6.2.19, 1.8.4.4 (latest), but do not solve the problem. My system hasn't got NTPD running. Has anyone have any other idea? By: iasgoscouk (iasgoscouk) 2011-07-06 13:20:55.422-0500 Do i Have to say anymore ! 24-25 days later since the last time I reported on this issue, mixmonitor is nolonger having sync problems in the last hour, when they started 24-25 days previously.  My findings are still going uncommented on from digium, and response has been made to this problem. After my recent 'help' with another open issue, and an unappreciated response from Tilghman Lesher, I am nolonger offering anymore further assistance with digium/asterisk. It apperas to me that there's no support on the asterisk/digium front and nothing is getting progressed unless you are a 'known contributor'/customer to digium/asterisk. I offer my time for nothing, and so does most of the people who are watching this open issue, and we are all hoping that issues will get resolved. Am very disappointed after using asterisk for over 5 years that the project has fallen apart leaving lots of open issues that need to be resolved. By: khoanv (khoanv) 2011-07-07 22:03:13.481-0500 Thks for your reply, Ian More detail about my system: There is no 24-25 days timing period issue. There are about 400-500 outbound calls per day, and almost of them are out of sync, at 97-98%. It happens every days, every months, no period. But it's the only one system which has got that problem. My others are not affected by this issue. Especially, the affected one is an upgraded system from a stable one. The Stable system (called CC 36) is based on CentOS 5.2, Asterisk 1.4.17.1 The Affected system (called CC 37) is based on CentOS 5.3, Asterisk 1.4.26.1 I also have several systems based on various versions, all of them have the same structure and IO connection base. And the issue only appeares on CC 37. Tomorrow, I plan to restore the CC 37 back to CC 36. By: khoanv (khoanv) 2011-07-11 22:18:45.266-0500 Hi all, After reinstalling a new fresh CentOS 5.5 on CC 37 and restoring Asterisk system, the out of sync issue remains no more :D I will continue to monitor the recorded files for period issue which is Ian's problem. By: David Hajek (hajekd) 2011-07-12 16:35:00.881-0500 Do you have outofsync issues on 1.8.5? By: Matthew Wiesemann (mdw19873) 2011-07-13 11:05:48.940-0500 Out of sync issues appear to be fixed in 1.8.5 Will continue to monitor and test. By: Matthew Wiesemann (mdw19873) 2011-07-21 13:28:44.196-0500 Since my last update, I have had zero out of sync issues since updating to 1.8.5 By: Leif Madsen (lmadsen) 2011-09-22 07:32:34.020-0500 I'm closing this issue out. It appears most people have had this problem resolved for them with either an update to Asterisk, or in most cases, and update of the operating system. If you continue to have problems with this, please make sure you're using the latest version of Asterisk, and open a new issue. By: Marco Aurelio (aureliors123) 2011-10-25 14:56:42.831-0500 Sorry for closing before any concrete solution. This is becoming an recurrent behavior of Digium developers... Aparently we solved this issue by configuring jitterbuffer on Dahdi cfg. ###################################### ; Configure jitter buffers in DAHDI (each one is 20ms, default is 4) ; This is set globally, rather than per-channel. jitterbuffers=4 ; ;------------------------------ JITTER BUFFER CONFIGURATION -------------------------- jbenable = yes              ; Enables the use of a jitterbuffer on the receiving side of a                              ; DAHDI channel. Defaults to "no". An enabled jitterbuffer will                              ; be used only if the sending side can create and the receiving                              ; side can not accept jitter. The DAHDI channel can't accept jitter,                              ; thus an enabled jitterbuffer on the receive DAHDI side will always                              ; be used if the sending side can create jitter. jbmaxsize = 200             ; Max length of the jitterbuffer in milliseconds. jbresyncthreshold = 1000    ; Jump in the frame timestamps over which the jitterbuffer is                              ; resynchronized. Useful to improve the quality of the voice, with                              ; big jumps in/broken timestamps, usually sent from exotic devices                              ; and programs. Defaults to 1000. jbimpl = fixed              ; Jitterbuffer implementation, used on the receiving side of a DAHDI                              ; channel. Two implementations are currently available - "fixed"                              ; (with size always equals to jbmax-size) and "adaptive" (with                              ; variable size, actually the new jb of IAX2). Defaults to fixed. jblog = no                  ; Enables jitterbuffer frame logging. Defaults to "no". ################################# By: Peter Draganov (pdraganov) 2012-10-02 02:17:20.929-0500 I have the same problem with Asterisk 1.8.10.1, so nothing is resolved with 1.8.5.0. I found the problem with one outbound call through Sangoma A101D with hardware echo canceller. We don't have jitterbuffers enabled in chan_dahdi.conf. I will try with them now. By: xxot-alex (xxot) 2013-02-09 09:21:08.667-0600 We have this issue on asterisk 1.8.7.1. It is only SIP, no dahdi. What I noticed, that if I use option "r" in Dial() the unsynchronization is much more worse than without this option. All calls are recorded to wav. Can it somehow be connected with codecs? By: Mehmet Tolga Avcioglu (mta59066) 2013-08-02 11:52:08.083-0500 Running 1.8.19 and started seeing this issue on SIP channels. By: xxot-alex (xxot) 2013-08-02 15:24:36.721-0500 Last time we fixed it by the asterisk upgrade.