[Home]

Summary:ASTERISK-19610: dsp.c can no longer detect a quick DTMF sequence
Reporter:Jean-Philippe Lord (jplord)Labels:Regression
Date Opened:2012-03-30 15:40:01Date Closed:2012-10-04 15:30:22
Priority:MajorRegression?Yes
Status:Closed/CompleteComponents:Core/Channels
Versions:1.8.10.1 1.8.11.0 10.2.1 10.3.0 Frequency of
Occurrence		Constant
Related
Issues:
is caused byASTERISK-17493 [patch] dsp.c sends multiple DTMF key events up to applications
is duplicated byASTERISK-20218 Alarmreceiver DTMF strings not recieved correctly from alarm-panel->SPA-PAP2->asterisk
is related toASTERISK-20024 India CallerID DTMF apparent in audio captured from TDM410 may not be reliably detected within Asterisk
is related toASTERISK-20218 Alarmreceiver DTMF strings not recieved correctly from alarm-panel->SPA-PAP2->asterisk
is related toASTERISK-20442 dtmf callerid regression
Environment:Attachments:( 0) 1332984810-SIP-2299-00000018-in.wav
( 1) ast-19610.diff.txt
( 2) ast-19610.diff2.txt
( 3) dsp_fastdtmf.patch
( 4) dsp.c
( 5) review2085.diff3.txt
( 6) sip_info_dtmf.diff.txt
Description:dsp.c stopped being able to decode a quick DTMF sequence starting in 1.8.10rc1. 50ms tone, 50ms silence does not work anymore. This isssue is major since asterisk used to be extremely good at detecting fast DTMF and it no longer is. I have removed the below change and issue got resolved.

From the ChangeLog:
2012-01-05 21:46 +0000 [r349672-349728]  Jonathan Rose <jrose@digium.com>

       * main/dsp.c: Fix an issue where dsp.c would interpret multiple
         dtmf events from a single key press. When receiving calls from a
         mobile phone into a DISA system on a connection with significant
         interference, the reporter's Asterisk system would interpret DTMF
         incorrectly and replicate digits received. This patch resolves
         that by increasing the number of frames a mismatch has to be
         detected before assuming the DTMF is over by 1 frame and adjusts
         dtmf_detect function to reset hits and misses only when an edge
         is detected. (closes issue ASTERISK-17493) Reported by: Alec
         Davis Patches: bug18904-refactor.diff.txt uploaded by Alec Davis
         (license 5546) Review: https://reviewboard.asterisk.org/r/1130/
Comments:By: Jean-Philippe Lord (jplord) 2012-03-30 15:43:08.079-0500

This is a sample WAV file of a DTMF sequence that asterisk can no longer detect. 50ms tone 50ms silence.

This dtmf sequence could be easily decoded by all versions of asterisk before 1.8.10rc1
By: Jean-Philippe Lord (jplord) 2012-03-30 16:12:40.275-0500

Attached is a patch to remove change that broke fast dtmf detection.
By: Matt Jordan (mjordan) 2012-04-02 13:00:22.644-0500

Of course, without this patch someone else's system doesn't work either :-(

We'll revisit this issue and attempt to find some middle ground.
By: Jean-Philippe Lord (jplord) 2012-04-02 22:42:20.352-0500

Thanks Matt. I'm a bit concerned about the original issue and its root cause. Many many users are relying on DTMF recognition for disa, voicemail and all... could it have been a configuration, echo or tuning issue that caused ASTERISK-17493 ? I am surprised that it is not until 1.8.10 that this came up... Many users disa and voicemail over with inband dtmf. Many had no issues to report.
By: Rusty Newton (rnewton) 2012-08-14 17:05:05.970-0500

Have ASTERISK-20218 reporting the issue as well. Closed as duplicate to this issue. Reverting their dsp.c is a workaround. See issue for any details.
By: Alec Davis (alecdavis) 2012-08-14 22:40:40.468-0500

referring to http://www.itu.int/rec/T-REC-Q.24-198811-I/en

'On duration' min should be 40ms.
'Pause duration' min should be 30ms, 40ms or 70ms depending on region.

Without removing the refactoring that was done, the easiest fix may just be to change number of HITS and MISSES to 3.
Specifically MISSES_TO_END being 40ms may be 1 too high.

/* How many successive hits needed to consider begin of a digit */
-#define DTMF_HITS_TO_BEGIN     4
+#define DTMF_HITS_TO_BEGIN     3
/* How many successive misses needed to consider end of a digit */
-#define DTMF_MISSES_TO_END     4
+#define DTMF_MISSES_TO_END     3

Changing logger.conf time format to, we get uSecond resolution
[general]
dateformat=%F %T.%6q

and with 3 extra debug prints.

However with HITS=4 and MISSES=3:
<code>
[2012-08-15 15:29:06.679634] DEBUG[27728] dsp.c: DTMF EDGE
[2012-08-15 15:29:06.719632] DEBUG[27728] dsp.c: DTMF BEGIN
[2012-08-15 15:29:06.719670] DEBUG[27728] chan_dahdi.c: Begin DTMF digit: 0x31 '1' on DAHDI/i1/21XXXXXXX-4
[2012-08-15 15:29:07.039630] DEBUG[27728] dsp.c: DTMF EDGE
[2012-08-15 15:29:07.059635] DEBUG[27728] dsp.c: DTMF END
[2012-08-15 15:29:07.059693] DEBUG[27728] chan_dahdi.c: End DTMF digit: 0x31 '1' on DAHDI/i1/21XXXXXXX-4

[2012-08-15 15:29:07.259627] DEBUG[27728] dsp.c: DTMF EDGE
[2012-08-15 15:29:07.279621] DEBUG[27728] dsp.c: DTMF EDGE   << this noise didn't get sent up to dahdi (refactoring ASTERISK-17493 fixed this )
[2012-08-15 15:29:07.419621] DEBUG[27728] dsp.c: DTMF EDGE
[2012-08-15 15:29:07.439637] DEBUG[27728] dsp.c: DTMF BEGIN
[2012-08-15 15:29:07.439672] DEBUG[27728] chan_dahdi.c: Begin DTMF digit: 0x32 '2' on DAHDI/i1/21XXXXXXX-4
[2012-08-15 15:29:07.739624] DEBUG[27728] dsp.c: DTMF EDGE
[2012-08-15 15:29:07.759636] DEBUG[27728] dsp.c: DTMF END
[2012-08-15 15:29:07.759674] DEBUG[27728] chan_dahdi.c: End DTMF digit: 0x32 '2' on DAHDI/i1/21XXXXXXX-4

</code>

By: Jean-Philippe Lord (jplord) 2012-08-15 15:15:19.128-0500

I have tested the below against file revision 349728 and it did the trick ! Quick DTMF are now recognized properly.

/* How many successive hits needed to consider begin of a digit */
-#define DTMF_HITS_TO_BEGIN 4
+#define DTMF_HITS_TO_BEGIN 3
/* How many successive misses needed to consider end of a digit */
-#define DTMF_MISSES_TO_END 4
+#define DTMF_MISSES_TO_END 3




By: Alec Davis (alecdavis) 2012-08-15 15:23:59.534-0500

Jean-Philippe:
I understand that it's working with both HITS and MISSES changed to 3, but are you able to test with DTMF_HITS_TO_BEGIN 4 and DTMF_MISSES_TO_END 3.

Thanks Alec

By: Jean-Philippe Lord (jplord) 2012-08-16 07:45:47.145-0500

Tested again with:
/* How many successive hits needed to consider begin of a digit */
#define DTMF_HITS_TO_BEGIN      4
/* How many successive misses needed to consider end of a digit */
#define DTMF_MISSES_TO_END      3


Does not work. The attached WAV file is not decoded and digits are missed.
Also tested with the alarm panel and same issue.

I went back to DTMF_HITS_TO_BEGIN 3 and DTMF_MISSES_TO_END 3 and the recognition is accurate again.

By: Richard Mudgett (rmudgett) 2012-08-20 11:00:39.743-0500

Alec

Is ASTERISK-17493 still fixed with just changing the DTMF hits begin/end to 3?
Or is a config option needed to make DTMF more tolerant of noise/dropouts by reducing its reaction to short DTMF bursts?
dtmf_sensitivity=normal ;default
dtmf_sensitivity=noise_tolerant ; For ASTERISK-17493 type applications
By: Alec Davis (alecdavis) 2012-08-21 02:08:56.963-0500

Richard:
 My assumption that each DTMF goertzel period was 10ms, if I'm not mistaken it's 12.75ms,
arrived at '#define DTMF_GSIZE 102', with 102 being the number of sample periods at sample rate of 8000Hz.
Which at dtmf_hits_to_begin @ 4 = 51ms, (not including the fact that the DTMF could start anyhwere through a goertzel period).

I believe DTMF_GSIZE should be 80, to reduce each DTMF goertzel sample to 10ms. then 'begin=4' and 'end=3' should work, for 50ms/50ms.

What was confusing is the debug capture, only 20ms between EDGE and BEGIN, which should have been 51ms, DAHDI packetisation?
[2012-08-15 15:29:07.419621] DEBUG[27728] dsp.c: DTMF EDGE
[2012-08-15 15:29:07.439637] DEBUG[27728] dsp.c: DTMF BEGIN

Re: A config option, not yet, until we've exhausted if there's a better way.

But, I need a reliable DTMF tester to inject 40ms/30ms into a call.

Alec

edit: Workable solution, but unable to test with 50/50ms ON/OFF
#define DTMF_TO_TOTAL_ENERGY 32.9 /* for DTMF_GSIZE 80, or 10ms sample period*/
#define DTMF_HITS_TO_BEGIN 4 /* 40 ms on */
#define DTMF_MISSES_TO_END 3 /* 30 ms off */
#define DTMF_GSIZE 80

By: Alec Davis (alecdavis) 2012-08-25 05:17:53.182-0500

Looking at the latest spandsp code dtmf.c the hits and misses can go.
Just rely on checking hit != current digit && lasthit != current digit, then it's knows something has changed.
 

By: Alec Davis (alecdavis) 2012-08-25 05:21:07.011-0500

But I still wonder why the DTMF_GSIZE is 102.
Changing it to 80, didn't work at all.

Someone else has commented in a similar vein, in mf_detect, where it's commented that 80 is the optimize value, but MF_GSIZE is define as 120.

edit:
#define DTMF_TO_TOTAL_ENERGY 42.0 /* for 102 samples (DTMF_GSIZE) or 12.75ms sample period */
#define DTMF_TO_TOTAL_ENERGY 32.9 /* for 80 samples (DTMF_GSIZE), or 10ms sample period*/

By: Alec Davis (alecdavis) 2012-08-25 06:30:13.031-0500

attached ast-19610.diff.txt

dtmf_detect()
Now ignore hits_to_begin and misses_to_end.

Same methodolgy a spandsp's dtmf.c method.
By: Alec Davis (alecdavis) 2012-08-25 07:23:33.798-0500

attached ast-19610.diff2.txt
replaces ast-19610.diff.txt

don't break squelch, that relied on 'hit' being preserve.
By: Alec Davis (alecdavis) 2012-08-26 04:21:53.234-0500

backtracking:

Removing the hits_to_begin and missed_to_end was the wrng approach.
They were introduced to allowing tuning in case of wobbly GSM signals. refer ASTERISK-11413 (https://issues.asterisk.org/view.php?id=11968#84583)

The values required for (40ms on)/(40ms off) is as follows;
#define DTMF_HITS_TO_BEGIN 2 /* 25.5ms, allowing for DTMF to start mid sample block */
#define DTMF_MISSES_TO_END 2 /* 25.5ms, allowing for DTMF to end mid sample block */

40ms/30ms is not achievable with the current DTMF_GSIZE of 102.
By: Alec Davis (alecdavis) 2012-08-29 15:29:04.381-0500

https://reviewboard.asterisk.org/r/2085/ Will also help when no interdigit pause is present, in that the detect time for each different digit is the same as a digit received with a previous interdigit pause.

By: Alec Davis (alecdavis) 2012-09-20 02:50:31.364-0500

uploaded review2085.diff3.txt as committed lied to 1.8 upwards.
By: Jean-Philippe Lord (jplord) 2012-09-30 09:45:07.154-0500

Hello Alec,

After a number of days in testing, I am still experiencing some issues in fast tone recognition. Local SIP clients go thru no issues with a fast sequence of tones but remote SIP clients takes about 10-20 retries before the sequence goes thru. I am a bit lost in all the versions of dsp.c I have tried over the past few months but I have gone back to a previous version to get all clients to work 100% of the time. Can we re-open this ? I am open to do any test you need me to do.
By: Jean-Philippe Lord (jplord) 2012-09-30 09:46:07.105-0500

This version works for any of my clients. The current SVN version is hit and miss.
By: Alec Davis (alecdavis) 2012-09-30 14:49:06.303-0500

DAHDI calls will use the dtmf detector from dsp.c, SIP doesn't normally.

SIP generally uses RFC2833 for DTMF which is out of band, unless inband DTMF audio is also/only enabled at the SIP client.

The dsp code doesn't come into play when only RFC2833 is being used.

By: Jean-Philippe Lord (jplord) 2012-09-30 21:36:39.660-0500

Hello Alec,

Sorry, I should have specified but I do use in-band. With current CVS, no success, it takes 10 to 20 sends before it catches all digits... With the version I attached, it works 1st time, 100% of the times.
By: Alec Davis (alecdavis) 2012-10-01 01:21:01.580-0500

I did not notice you'd uploaded dsp.c of the version you are using.

As I understand it, you're using this uploaded version with the latest from the SVN?

I think to make the lastest version of dsp.c to be equivalent to the one you've uploaded you need to change the DTMF_HITS_TO_BEGIN to 3.
DTMF_MISSES_TO_END is already 3.
By: Alec Davis (alecdavis) 2012-10-01 02:50:26.015-0500

Just an idea.

I downloaded your version of dsp.c from here and compiled with lastest 1.8 SVN-branch-1.8-r374032M

Need to set in sip.conf dtmfmode=inband, also only allow codecs alaw or ulaw.

I've just tried this with a grandstream GXP2000 set to 'inband' and 'SIP INFO' and get double 'DTMF end' events and are unable to succesfully enter voicemail.

[2012-10-01 20:37:31.025198] DTMF[23216] channel.c: DTMF begin '8' received on SIP/822-00000002
[2012-10-01 20:37:31.025241] DTMF[23216] channel.c: DTMF begin ignored '8' on SIP/822-00000002
[2012-10-01 20:37:31.057398] DTMF[23216] channel.c: DTMF end '8' received on SIP/822-00000002, duration 480 ms
[2012-10-01 20:37:31.057445] DTMF[23216] channel.c: DTMF end passthrough '8' on SIP/822-00000002
[2012-10-01 20:37:31.105292] DTMF[23216] channel.c: DTMF end '8' received on SIP/822-00000002, duration 51 ms
[2012-10-01 20:37:31.105390] DTMF[23216] channel.c: DTMF end passthrough '8' on SIP/822-00000002

When GXP2000 is set to 'inband' only, DTMF detection works as expected.

By: Alec Davis (alecdavis) 2012-10-01 03:55:34.626-0500

uploaded sip_info_dtmf.diff.txt

Note: Used a clean checkout of SVN 1.8 branch.

This fixes my senario, when the cleint is set to send both 'inband' and 'sip info', and in sip.conf dtmfmode=inband.


By: Alec Davis (alecdavis) 2012-10-01 04:34:19.139-0500

reopening, as originally reported (but missed), this report is regarding inband DTMF delivered over SIP.

There are 2 issues
1). The dsp.c code was broken for fast 50/50ms DTMF.
2). Double DTMF End event that happens when both "inband" and "SIP INFO" are enabled.

By: Jean-Philippe Lord (jplord) 2012-10-01 20:47:02.984-0500

Hello Alec,

Thanks for you help. I have done some more testing and understood that the issue lies in app_alarmreceiver's latest revision. Not in dsp.

I have tested your patch but it did not make a difference in my case. I am very sorry for this. I will log this issue in ASTERISK-20484


By: Jean-Philippe Lord (jplord) 2012-10-01 21:44:00.752-0500

Again, I apologize, I believe this can be closed.