[Home]

Summary:ASTERISK-05868: [patch] picking up multiple DTMF using RFC2833
Reporter:delvar (delvar)Labels:
Date Opened:2005-12-19 07:50:40.000-0600Date Closed:2006-09-27 16:32:22
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Core/RTP
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) ast1.cap
( 1) ast2.cap
( 2) delvar.20012006-1.svn.dtmf.patch
( 3) fool4kate.rtp.dtmf.path
( 4) rtp.diff
( 5) rtpTraf.txt
Description:Hi, we are experiencing a problem where we are receiving DTMF RTP packets out of order and asterisk seems to be interpreting these as multiple (separate) DTMF signals when it shouldn&ASTERISK-7996;t.

Our setup is as follows,
[PSTN Provider] > SIP > [Ast1] > SIP > [Ast2] + Voicemail APP


****** ADDITIONAL INFORMATION ******

We call from a BT land line to our incoming number, this is then passed to us using SIP from our incoming number provider to Ast1 in our datacenter then from Ast1 to Ast2 which is on our local SDSL circuit which runs vocemailmain app and we use a username of  &ASTERISK-7995;1234&ASTERISK-7996; and pin of &ASTERISK-7996;5678&ASTERISK-7996;.
In all cases, we use dtmfmode=rfc2833.

I have attached a couple traces, one from Ast1 and one from Ast2 of the same call.

In Ast1.cap the DTMF packet numbers are,

&ASTERISK-7995;1&ASTERISK-7996; = 301-306
&ASTERISK-7995;2&ASTERISK-7996; = 320-325
&ASTERISK-7995;3&ASTERISK-7996; = 342-347
&ASTERISK-7995;4&ASTERISK-7996; = 385-390
&ASTERISK-7995;#&ASTERISK-7996; = 410-415

&ASTERISK-7995;5&ASTERISK-7996; = 557-562
&ASTERISK-7995;6&ASTERISK-7996; = 574-579
&ASTERISK-7995;7&ASTERISK-7996; = 591-596
&ASTERISK-7995;8&ASTERISK-7996; = 604-609
&ASTERISK-7995;#&ASTERISK-7996; = 621-626

In Ast2.cap the packet numbers are,

&ASTERISK-7995;1&ASTERISK-7996; = 309-314
&ASTERISK-7995;2&ASTERISK-7996; = 328-333
&ASTERISK-7995;3&ASTERISK-7996; = 350-355
&ASTERISK-7995;4&ASTERISK-7996; = 393-398
&ASTERISK-7995;#&ASTERISK-7996; = 418-424

&ASTERISK-7995;5&ASTERISK-7996; = 565-571
&ASTERISK-7995;6&ASTERISK-7996; = 582-587
&ASTERISK-7995;7&ASTERISK-7996; = 599-604
&ASTERISK-7995;8&ASTERISK-7996; = 612-617
&ASTERISK-7995;#&ASTERISK-7996; = 629-634

looking at Ast1.cap all the RTP packets are in order as expected but on the receiving side on Ast2 we see the DTMF packets out of order.
Look at the &ASTERISK-7995;5&ASTERISK-7996; (565-571 in Ast2.cap) we can see

565 start TS: 58904, Sequence: 17016
566 end TS: 58904, Sequence: 17019
567 start TS: 58904, Sequence: 17017
568 start TS: 58904, Sequence: 17018
569 end TS: 58904, Sequence: 17021
570 end TS: 58904, Sequence: 17020

now for this asterisk took it to mean &ASTERISK-7995;55&ASTERISK-7996; not just &ASTERISK-7995;5&ASTERISK-7996; I assume because it takes the 567 start as the start of a new DTMF tone and not a continuation of 565 as it should.
Comments:By: ennuyeux72 (ennuyeux72) 2006-01-05 07:11:01.000-0600

any news on this one?

By: delvar (delvar) 2006-01-09 04:04:29.000-0600

anyone got a clue whats going on? anyone willing to test it?

By: delvar (delvar) 2006-01-16 04:26:29.000-0600

i have been looking into this and i can't see any easy way to fix it.
i tried just dropping the out of order packets but some of the DTMF would just be lost (most packets out of order).

the only thing i can think of is to build an array of char,seqence number and timestamp and keep track of all DTMF recived then reorder them before sending on.

the RTP packets are all recived within about 60ms so the delay of the buffering shouldnt be too big.

but im no expert :)

By: delvar (delvar) 2006-01-16 07:56:36.000-0600

well i did it...

i keep an array in 'struct ast_rtp' to keep track of
digit, remote timestamp, local timestamp, seqno, status
for each dtmf 'string'.

I just about rewrote, 'static struct ast_frame *process_rfc2833' to do the following,

when i receive a DTMF i check if its 'remote timestamp' and 'digit' are in the array.

if it isn't i add it, i look though the array for any row where the status is 0 (free) or the timestamp is over 1 seconds old then set the status to 1 (ready), set the 'seqno', 'remote timestamp' and the 'local timestamp'.

Check the status, if the 'status' is 1 (ready) i let it through, check its not a duplicate (rtp->lasteventendseqn != seqno ) then call send_dtmf and set the 'status' to 2 (done)
 
in this way it will only accept the DTMF once no matter how many packets it receives or what order they are in.

the only drawback is if a user hits the same button twice very quickly, they will arrive on the asterisk server with the same 'remote timestamp' and 'digit' so it will be ignored.
Not much I can do about that without logging all packets and tracking the sequence numbers.

This is a complete hack and im sure breaks something else but in my situation it works.

I setup a simple system to enter a pin number then do some funky stuff, which worked well, then when this problem started and we 'received' multiple DTMF it totally broke it out of 10 tries I would only get one that worked.
Now with this fix I can get just about all attempts to work, except where I type too fast and it ignores it.


If you want the code let me know and ill do a diff and upload it.



By: Serge Vecher (serge-v) 2006-01-16 09:28:44.000-0600

Delvar: please upload your patch and fax in the disclaimer to digium. Thanks!

By: Leif Madsen (lmadsen) 2006-01-16 12:10:20.000-0600

This bug appears to be related to 5970 which I have been working with. Corydon has uploaded a patch which I will be testing this week so we can hopefully get this and the other bug resolved.

By: Chris Field (cfieldmtm) 2006-01-16 14:27:49.000-0600

I have a much simpler patch.  We were seeing the same problem when packets were comming in badly out of order (SEQ# 3,4e,4e,4e,2,1) which caused a duplicate dtmf due to the statement in process_rfc2833 line 305:
 if (!(event_end & 0x80))
   rtp->resp = resp;
this will set rtp->resp = resp and then will resend the digit when it gets the first packet in the next series.

I changed it to the following:
if ((!(event_end & 0x80)) && rtp->lasteventendseqn<seqno)

which will keep it from changing the value of rtp->resp (which it uses as a condition to enter the process_rfc2833 function) when it gets a previous packet in the sequence.

==Patch & Debug are attached



By: Leif Madsen (lmadsen) 2006-01-16 15:23:02.000-0600

cfieldmtm:

-- patches should be attached as a text file to the bug using diff -u
-- patches require a disclaimer to be on file with Digium before it can be looked at
-- you should attach debugging info, network traces, etc... as an attachment to the bug and not in a bugnote
-- ideally you should read the bug guidelines

Thanks!

By: delvar (delvar) 2006-01-17 02:06:37.000-0600

ok iv uploaded my patch against svn trunk, i faxed over my disclamer last night, can anyone confirm that its been recived?

this patch is just a dirty hack and will probably break when used with non asterisk devices... but it does solve my problem.
there are some improvements to help with compatability but the main problem is asterisk sending the DTMF packets too quickly and getting mixed up in the first place.

i have made the array in 'struct ast_rtp' to hold 5 rows, in testing i never had more than 2 but you never know.

cfieldmtm, i tried something like that, the problem i am having is that ALL of the packets can be out of order and even totaly backwards so dropping them means you lose the whole DTMF signal.


ill take a look at bug 00005970 and see whats happening,

By: delvar (delvar) 2006-01-17 02:43:37.000-0600

hmm i supse i need to update my patch to only send teh DTMF on an end event so the duration is correct.
currently it will take the duration from any packet it recives which is 0 for the first packet.

By: Leif Madsen (lmadsen) 2006-01-17 07:24:07.000-0600

Delvar: Please try the patch from Corydon in bug 5970 and see if it fixes your problem. If it fixes the problem, its probably a cleaner way of going about it. I'll be simulating it this week in the lab and then trying it on a production server, but if it works for more than one person, all the better.

You're looking for the 20060116__bug5970.diff.txt patch.

Thanks!

By: Chris Field (cfieldmtm) 2006-01-17 07:41:43.000-0600

The patch in 5970 looks like it is for the sending of RTP digits not for the interepreting of RTP digits that are being received.

By: delvar (delvar) 2006-01-17 08:14:35.000-0600

sorry i can't put this patch (20060116__bug5970.diff.txt) on our incoming gateways but it sounds like it would fix my problem as it will space out the DTMF packets better and should stop them going out of sequence so much.
but if they do arive out of sequence due to jitter there will still be multiple DTMF.

if i get the chance to try it i will but for now there isn't much i can do.

By: Leif Madsen (lmadsen) 2006-01-17 08:59:15.000-0600

Well, ideally all systems would actually check the sequence number and not even care if the packets arrived out of order -- but we're not that lucky :)

By: delvar (delvar) 2006-01-17 09:30:27.000-0600

yep...
so my new patch tries to account for that by keeping the 'current' state of the DTMF tones in the array, then only issueing a send_dtmf when it receives an end.
if it then receives out of order starts or stops it will just ignore them.

i used an array just incase we receive 2 or more tones very quickly and the first tone still hasn't 'timed out'.
You could probably do away with the array and just use a couple variables instead but i like playing safe :)

i was thinking about tracking the seqno to try to maintain compatibility with other devices which could send a DTMF tone over several seconds (ie hold the button down) it would then ignore any out of order packets but still maintain the current status and update the timer so it wouldn't time out.
i would have to set the timeout value to 3 times the average delay between packets as per the RFC.

the problem then is what to do? as we haven't yet issued a send_dtmf so the tone will essentially be lost...
unless the 5970 patch helps with that with a start/stop in which case we could just have a bit of code to check up on the array it and issue the stop as needed.



=======
alternatively, the receiver can
          start a tone and play it until it receives a packet with the
          "E" bit set, the next tone, distinguished by a different
          timestamp value or a given time period elapses. This is more
          robust against packet loss, but may extend the tone if all
          retransmissions of the last packet in an event are lost.
          Limiting the time period of extending the tone is necessary
          to avoid that a tone "gets stuck". Regardless of the
          algorithm used, the tone SHOULD NOT be extended by more than
          three packet interarrival times. A slight extension of tone
          durations and shortening of pauses is generally harmless.
=======



By: delvar (delvar) 2006-01-18 08:14:42.000-0600

ok i just updated the patch,  delvar.18012006-1.svn.dtmf.patch
not it takes into account tones that last for long time, ie phone direct to asterisk etc..

before it would generate a boatload of digits if you held the key down.

i think i can remove the array totaly and with only one extra value in 'struct ast_rtp' (timer) gain the same behavior.

still to fix:
at the moment if we lose all 3 of the 'end' packets we lose the DTMF tone totaly...
we should send the tone shorlty after but thts not posiable.. so ill do what it did before and if i recive a DTMF packet thats not the one we had before, just send the last one.

By: Leif Madsen (lmadsen) 2006-01-19 16:50:51.000-0600

This patch by Delvar also did not help in our scenario -- more info to follow.

Edit: More info in 5970



By: delvar (delvar) 2006-01-20 02:39:35.000-0600

hi, blitzrage can you post the traces so i can check them over.
here the jitter is generated by our ADSL connection so i have no controle ont eh amount of jitter.
my patch assuems that the packets will not have a jistter mote thna 180ms if its more than that it will assume its another DTMF tone.
was the jitter that high in your tests?
you can change the timeout by modifying this line in rtp.c,
#define MSG_RFC2833_CACHE_TIMEOUT 180
(number of miliseconds before tone is deemd old)

the problem with setting this too high is that if you press the same key twice rapidly it only takes it as one DTMF event and ignores teh others (as they all have the same timestamp and resp).

the only way to help this would to be to look a the seqno and if the lower seqno was a end and this one isn't then treat as a new dtmf, ill try that when i get soem time.

By: delvar (delvar) 2006-01-20 03:57:07.000-0600

iv up loaded patch, delvar.20012006-1.svn.dtmf.patch
iv set the aray to 10 signals (instead of 5), and increased the timeout to 1 second.
also outputs more info to console so you can see wahts going on a bit better.
hopefully will help debug.

let me knwo if it helps.

By: Leif Madsen (lmadsen) 2006-01-20 13:08:46.000-0600

Delvar: yah, the amount of jitter we were getting was MUCH too high to be realistic. I'll let you know of the results in a more real work scenario sometime this weekend. I have a test box that I'm going to place calls through to the provider(s) I'm having the DTMF problems with.

Will keep you updated.

By: Olle Johansson (oej) 2006-01-30 04:43:59.000-0600

Any updates to this issue?

By: delvar (delvar) 2006-01-31 04:26:35.000-0600

my patch works for me, i realy can't test it in anyone elses situation so not much more i can do.
if someone else can test it it will help a lot.

hopefully we won't need this or maybe a cut down version of this patch once the send dtmf is sorted in issue 5970.

By: element (element) 2006-02-09 21:24:08.000-0600

Hi, I am new to patching and am having trouble applying the delvar.20012006-1.svn.dtmf.patch file. First the uploaded file attachments open up to a web page (not a downloadable file) so I copied the output and ran: "cat > delvar.20012006-1.svn.dtmf.patch" and pasted in the text. I then downloaded asterisk: "svn checkout http://svn.digium.com/svn/asterisk/branches/1.2 asterisk-1.2" and applied the patch with the following error:

# patch -p0 < delvar.20012006-1.svn.dtmf.patch
patching file /usr/src/asterisk/rtp.c
Hunk #1 succeeded at 85 with fuzz 2 (offset 4 lines).
Hunk #2 FAILED at 137.
Hunk #3 FAILED at 303.
2 out of 3 hunks FAILED -- saving rejects to file /usr/src/asterisk/rtp.c.rej

I attached the .rej file for reference.

I was hoping this would help resolve my situation where my inbound 800 provider is duplicating DTMF digits any ideas?

thank you.

By: A Brown (fool4kate) 2006-02-27 10:23:23.000-0600

I noticed a simlar problem and generated the patch fool4kate.rtp.dtmf.path (jeez, how did I type that wrong, oh well).  Anyway, it uses a different approach than Delvar's and seems to have been working on three servers for the last month. I'm not sure how robust it is, and after reading some of the comments I think it might have issues with rapidly pressing DTMF keys and possibly long key presses. It is tough to test, because the problem only occurs if there is a lot of route flap between our servers and our provider. -Best Regards

By: delvar (delvar) 2006-02-27 10:49:00.000-0600

element: the patch is for SVN trunk on the date in the file name (21th of jan 2006) not stable.

has anyone else tested this?
any bugs/good news/bad news?

hmm does this qualify for karma? :)

By: Serge Vecher (serge-v) 2006-03-14 16:23:09.000-0600

Delvat: a fix to a related bug ASTERISK-6491 has been committed to 1.2 (will be in 1.2.6 when that's released) and trunk (http://lists.digium.com/pipermail/svn-commits/2006-March/011321.html). [^] Please test with the latest version of either and report if that fix works or not for you. Thanks for testing!

By: Arsen Chaloyan (arsen) 2006-03-15 01:24:07.000-0600

This issue isn't related to ASTERISK-6491 at all, but partially related to ASTERISK-5815.
So patch on ASTERISK-6491 cannot resolve this problem.

By: Arsen Chaloyan (arsen) 2006-03-15 01:39:06.000-0600

This problem is fully reproducible to me.
I've sent rtp stream (ast2.cap) to my asterisk (1.2-r12792) and got the same results as mentioned in bug description.

To be honest, current implementation of process_rfc2833 in rtp.c seems to be buggy.
Delvar's patch seems to be too complex, much easier solutions can be applied.
I like fool4kate approach, but seems it will not close all cases.
I'll work on another patch and share it with you as soon as possible.

By: delvar (delvar) 2006-03-15 02:43:50.000-0600

i agree my patch is a bit complicated and wastefull, if you can come up with a nicer patch id be happy to test it.
i have noticed some odd behavior with grandsteam GXP phones not picking up DTMF at all,i iwll have to debug this at a later date, probably something silly.

By: Arsen Chaloyan (arsen) 2006-03-15 08:10:07.000-0600

Find attached patch (rtp.diff) against asterisk 1.2 -r12792.
This patch contains improved process_rfc2833 function, which will solve the reported issue.
Hopefully it will fix other potential problems, concerning dtmf detection using rfc2833 in asterisk as well.
This surely will work with rfc2833 compliant devices, but it is also intended to work even against broken implementations as well.
I need feedback.
From my side I'm going to test this patch against huge database of captured streams, collected during past years, which contains streams from Cisco, Snom, Aastra, Sipura, and  others, even broken streams as well.
I'll report the results soon.

By: Marcel Barbulescu (marcelbarbulescu) 2006-04-08 10:46:56

Hi guys,

I just want to report back some testing: I was using Asterisk 1.2.5 and later 1.2.6 with both Delvar's patch (delvar.20012006-1.svn.dtmf.patch) and arsen's patch (rtp.diff) for the last three weeks. I still had double digits problems with arsen's patch (but less than without any patch at all) but the delvar's one worked just fine. I have some connections that cause constant trouble so let me know if you want me to do more testing.

Thanks all for the great work.



By: delvar (delvar) 2006-04-10 10:54:27

hi, thanks for testing.

it is a bit overkill but untill the RFC2833 stuff is fixed on the sending side i cant think of a better way to do it.

have you had any problems with my patch? its hard to improve it if it works :)

By: Marcel Barbulescu (marcelbarbulescu) 2006-04-10 11:04:44

Delvar:

You're patch works fine for now. I'll let you know if I will encounter any problems.

I think this bug should not be tightly coupled with the problems Asterisk has on the sending side because I suspect there are double DTMF receiving problems present with other VoIP equipments on connections with high jitter (not all my providers disclose what they use as switches but I strongly suspect some of them not using Asterisk).

By: delvar (delvar) 2006-04-10 11:19:31

to some degree yes but asterisk makes it worce by shotgunning all the 6 DTMF packets at once so teh chance of it ariving out of order is very high even with a tiny amount of jitter.
other gateways may also do this but they should fix that and this shouldn't be a permiant thing in asterisk.
this is simply a hack (and a bulky one at that).

if the devs want to take this into trunk let me know and ill work on it to slim it down a bit. maybe make it optional at compile time.

By: Marcel Barbulescu (marcelbarbulescu) 2006-04-14 16:27:23

Delvar:

Double digits started to appear even when the system is patched with your code. Let me know if there is anyhting I can do to help you improve the code. If you need debug info, let me know how to collect it.

Thank you.

By: delvar (delvar) 2006-04-24 10:00:57

ok if yopu can send me the output from cli (verbos+debug set high) and if posiable an ethereal trace ill look into it, also what phones you are using and conenction information.

something that might help, near the top or rtp.c iv added a few #defines try playing with them, especialy the timeout 'MSG_RFC2833_CACHE_TIMEOUT'.

By: Marcel Barbulescu (marcelbarbulescu) 2006-04-24 12:23:06

Actually it was the same problem, but not with my server but with the Asterisk I got the calls from. That server was wrongly detecting two digits instead of one and was sending two legitim different digits to my server. That server was patched also and everything is ok now. This bug is very real and your solutions (Delvar, arsen) work. Thanks again for all the work.

By: delvar (delvar) 2006-04-26 12:04:54

cool :)
glad you find it usfull.

By: David S Madole (dsmadole) 2006-06-20 09:44:48

Just wanted to offer some feedback on this issue.

I have a site where the packet reordering through the ISP is especially bad, making it impossible 9 tries out of 10 to even enter a three-digit extension correctly.

I applied Delvar's patch and DTMF entry there is now 100% reliable. No issues with doubled digits at all. Amazing and thanks. Please get something like this into release code!



By: delvar (delvar) 2006-06-20 11:48:25

if its not 100%, can you tell me what's happening? maybe i can help fix it.

By: David S Madole (dsmadole) 2006-06-20 11:55:47

Sorry, Delvar, that was a typing error. I mean to say "now 100% reliable" -- it works perfectly. I fixed my prior comment. Thanks again.

By: delvar (delvar) 2006-06-20 12:06:39

oh cool :) /me does a little dance

By: Adam Long (worldlink) 2006-07-06 14:07:22

I'm having same problems with out of sequence udp packets causing double DTMF digits.  I'm running an AEL IVR application with inbound calls fed from a SIP carrier.  I can test this about 50% of the time due to the quality of the connection.  I'm going to test the various patches here starting with Delvar's (thanks for writing these guys, glad to know i'm not the only one battling with this issue)

I'll post my results...

By: Adam Long (worldlink) 2006-07-06 15:50:18

Delvar's patch didnt seem to solve my issues.  I tried adjusting the CACHE time and some of the other variables... No luck.  And it would completely flake out when calling in direct through SIP from my GXP-2000.

However, the one liner posted above by cfieldmtm seemed to fix my problems...

if (!(event_end & 0x80))
   rtp->resp = resp;

to

if ((!(event_end & 0x80)) && rtp->lasteventendseqn<seqno)
   rtp->resp = resp;

Not sure what this breaks if anything... seems to work 100% for me.

By: Ronald Chan (loloski) 2006-07-06 16:24:19

Guys,

 I'm using also delvar's patch without a problem, i'll post his original work to be compatible with the latest SVN trunk rev 37174 in the hope someone will need it also.

Thanks Delvar

Ronald

By: Adam Long (worldlink) 2006-07-07 11:06:27

Hi Ronald,

Delvars code is working for you?
I tried it yesterday and it did not seem to help very much.

The one line change (see my last post above) definitly did the trick however.

Any ideas?

-Adam



By: Ronald Chan (loloski) 2006-07-08 07:39:24

Adam,
 
 Yes!, Well in my case the congestion is happening on my end due to bandwidth constraints, the other leg of the call was getting a duplicate dtmf signal which cause by network jitters.

My Asterisk is SVN trunk rev 37174

My setup is
                  GSM
UA -> SIP -> *  -> SIP -> Voipjet/Teliax -> PSTN

Avg. latency is 192-256 ms at any given time.


 Before the patch apply, if i call from any Toll number or even US did number i won't get thru even with 3 digit extensions on the called party.

 After the patch has been apply, 70-80% of the call went ok compare to _NOTHING_.


Best regards,

Ronald

By: Eldad Ran (eldadran) 2006-08-11 07:01:57

I had the same problem cfieldmtm patch fixed it 100%

By: Serge Vecher (serge-v) 2006-08-11 09:39:12

anybody testing trunk (the soon 1.4.beta to be) here may want to check out the following branches:

http://svn.digium.com/view/asterisk/team/group/vldtmf/
http://svn.digium.com/view/zaptel/team/group/vldtmf/

AFAIK, functional vldtmf support is what's holding up the 1.4 release, so help make it happen!

By: Joshua C. Colp (jcolp) 2006-09-27 16:32:14

Okay folks here is the scoop - DTMF in 1.4 is vastly different to what it is in 1.2, and it can't be backported without A LOT of trouble... so I'm closing this bug. Please do give 1.4 a try as a beta, or wait for a release and open a new bug if you have issues with the implementation there. That is all and thanks all for holding on!