ASTERISK-13335: iax2 trunked channels not being cleared

[Home]

Summary: ASTERISK-13335: iax2 trunked channels not being cleared

Reporter: clive18 (clive18) Labels:

Date Opened: 2009-01-10 14:53:44.000-0600 Date Closed: 2009-05-15 16:17:12

Priority: Minor Regression? No

Status: Closed/Complete Components: Channels/chan_iax2

Versions: Frequency of
Occurrence

Related
Issues:

Environment: Attachments:

Description: It appears to me as a memory leak, as many old iax2 channels are appearing in the "iax2 show netstats" as:
(None) 1000 0 40 0 0 0 0 0 0 0 0 0 0 0 0

At present I see 36 out of 57 of these channels reported on "iax2 show netstats", but when I do a "show channels" there are only 21 calls.

In my humble opinion, something on the channel is not being closed properly at the end of a call.
Before I was using version 1.2.6 and had no issue at all. I know its been a long time since then, but I know something is now broken, although I am not sure which is the latest version which has this behavior.

If I leave asterisk to run on its own, after 1 week there are about 60 of these ghost channels displayed, and the number keeps increasing with time.

Comments: By: Leif Madsen (lmadsen) 2009-01-11 10:41:26.000-0600

Can you reproduce this on the 1.4 branch? 1.2 is end of life, and would not receive any additional bug fixes; just security issues.
By: Rudy (rudy) 2009-01-19 10:12:29.000-0600

Same problem with Asterisk 1.4.21.2

It seems to be recent, I see these "zombie" channels for 10 days while the server is running for about 150. I never saw them before.

When the number reaches 100 the sound quality is very poor.

By: clive18 (clive18) 2009-01-20 00:46:52.000-0600

I know its not the best solution, but I "upgraded" to version 1.2.19 and thats working for me. It would be good if this bug in 1.4 was fixed though.
Thanks.
By: GERD GERD (gerd1000) 2009-01-20 05:11:57.000-0600

Same for 1.6.0 release, downgraded to 1.6 beta 9
By: Brett Matthews (brett matthews) 2009-02-04 16:17:59.000-0600

I have the same issue with trunks between our offices. All using 1.4.21.2. All boxes showing similar issues. Below are dud channels to both our VoIP provider, and to another office on a trunked channel.

begapabx*CLI> iax2 show channels
Channel Peer Username ID (Lo/Rem) Seq (Tx/Rx) Lag Jitter JitBuf Format
(None) 59.154.32.66 2707 01085/03839 00002/00002 00000ms 0000ms 0040ms unknow
(None) 59.154.32.66 2707 01459/04580 00002/00002 00000ms 0000ms 0040ms unknow
(None) 59.154.32.66 2707 02447/06946 00002/00002 00000ms -0001ms 0000ms unknow
(None) 59.154.32.66 2707 08649/03051 00002/00002 00000ms 0000ms 0040ms unknow
(None) 59.154.32.66 2707 08875/14002 00002/00002 00000ms -0001ms 0000ms unknow
(None) 59.154.32.66 2707 10545/13002 00002/00002 00000ms 0000ms 0040ms unknow
(None) 59.154.32.66 2707 12379/03152 00002/00002 00000ms 0000ms 0040ms unknow
(None) 59.154.32.66 2707 12557/03747 00002/00002 00000ms -0001ms 0000ms unknow
(None) 59.154.32.66 2707 16374/06228 00002/00002 00000ms 0000ms 0040ms unknow
(None) 192.168.2.60 begambl 16384/00957 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.2.60 begambl 16385/12993 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.2.60 begambl 16386/03190 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.2.60 begambl 16387/06712 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.5.20 begaeden 16388/13416 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.5.20 begaeden 16389/13203 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.5.20 begaeden 16390/08487 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.5.20 begaeden 16391/09938 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.5.20 begaeden 16392/07845 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.5.20 begaeden 16393/10026 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.5.20 begaeden 16394/06473 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.2.60 begambl 16395/11629 00002/00002 00000ms -0001ms 0000ms unknow
(None) 192.168.2.60 begambl 16397/05709 00002/00002 00000ms -0001ms 0000ms unknow
By: Leif Madsen (lmadsen) 2009-02-09 11:02:56.000-0600

I've assigned this issue to dvossel to see if there is something he can do with it.

Thanks!
By: Rudy (rudy) 2009-02-09 11:09:02.000-0600

Thank you !
By: David Vossel (dvossel) 2009-02-09 13:42:03.000-0600

Is it possible to give me any specific steps to reproduce the issue, or does it seem to be completely random?
By: Brett Matthews (brett matthews) 2009-02-09 17:01:45.000-0600

I found one instance where it created a invalid channel when I was testing out codecs on trunks. I had the iax.conf not quite right, and the trunk could not negotiate the codec between boxes, and failed back to an alternate trunk. At that point it created an invalid. I think it generally happens when there is something wrong with the initial channel creation ... perhaps when the other end doesn't respond for some reason ?
By: Rudy (rudy) 2009-02-10 02:39:50.000-0600

We first had these invalid channels after connecting a new iax peer on our server.
So it's possible that they are due (as Brett Matthews said) to problems in codec negotiation.
By: clive18 (clive18) 2009-02-11 04:05:07.000-0600

To reproduce this issue, I would set up 2 asterisk boxes, one to send calls and one to receive calls. Set up with Iax2 trunking enabled. Send a lot of calls from box A to box B, and watch box A's behaviour, using "iax2 show netstats". I am pretty certain you will start seeing these ghost channels after a short amount of time.

thanks for your help.
By: Brett Matthews (brett matthews) 2009-02-16 17:24:03.000-0600

I just restarted our box yesterday for a different issue, and noticed as soon as I restarted an invalid IAX channel was created when it tried to create a trunk to our test box that is not turned on at the moment.

(None) 192.168.3.205 (None) 10028/00000 00001/00000 00000ms -0001ms 0000ms unknow
By: David O Reilly (trendboy) 2009-03-20 16:29:55

I've noticed this too on my box, but it turned out I had in my extensions.ael the following:

s => {
Answer();
Dial(IAX/etcetcetcetc);
Hangup();
}

(you may have done this on the old extensions.conf format, same thing)

Anyway. It would show up phantom calls as whenever I hung up a call it could continue to appear as alive when I did a iax2 show netstats

The fix or work around was to REMOVE the Answer() method and this fixed it. I have no idea why and have logged a bug (0014632).

Give that a try and see if it helps you. If you have never used the AEL format, then just use it as you would in the old conf format and just remove the answer bit, so 1,answer; is removed to and the next number is decremented in the list for (each) record.
Dave :)
By: David O Reilly (trendboy) 2009-03-20 16:31:34

Ps-I have this (none) trunk appearing too - I think it may be a test account I gave a user that I deleted, so perhaps you "had" and account set up and deleted them from your iax.conf settings?
By: Stephan Monette (monettes) 2009-03-31 21:27:24

I had the same problem with Asterisk 1.4.21.2. I changed my config file iax.conf to disable iaxcompat and it fixed this problem.

file: iax.conf

iaxcompat=no ### If I leave it to yes, I get thousands of fantom IAX2 channels.

Hope this information helps.
By: Rudy (rudy) 2009-04-01 10:39:12

The default value of iaxcompat is "no", isn't it ?

I have no "iaxcompat=yes" and thousands of zombie channels (* 1.4.21.2)... :(
By: Stephan Monette (monettes) 2009-04-01 13:33:51

I had to restart Asterisk after changing the iax.conf file to get rid of the zombie channels.

Here's my iax.conf file:

; Inter-Asterisk eXchange driver definition
;
; This configuration is re-read at reload
; or with the CLI command
; reload chan_iax2.so
;
; General settings, like port number to bind to, and
; an option address (the default is to bind to all
; local addresses).
;
[general]
iaxcompat=no
nochecksums=no
delayreject=yes
language=en
bandwidth=high
disallow=all
allow=ulaw
allow=gsm

jitterbuffer=yes
forcejitterbuffer=no
maxjitterbuffer=1000
maxjitterinterps=10
resyncthreshold=1000

minregexpire = 120
maxregexpire = 300

iaxthreadcount = 256
iaxmaxthreadcount = 256
autokill=yes
maxauthreq=10
authdebug=no

=====================================

I dnot get zombie channels that stays forever. With this config in 1.4.21.2, I get a few zombie channels that expires which is normal for IAX2 (I think!).

Cheers.
By: David Vossel (dvossel) 2009-04-01 14:30:45

default for iaxcompat is no. Does anyone else thats having this issue have iaxcompat=yes?
By: Alex Balashov (abalashov) 2009-04-29 07:36:59

I've got this problem too (1.4.21.2), and none of the remedies suggested in the notes on this page (removing Answer(), iaxcompat={yes|no}, autokill=yes, etc.)
By: David Vossel (dvossel) 2009-04-30 15:58:43

It seems that most everyone reporting this issue is using 1.4.21.2. Can anyone give me information on what version they were using before this problem existed? I'm trying to isolate where this issue was introduced.

I still have not been able to reproduce this. I've seen these channels pop up after doing some odd things (like calling then disconnecting the device from the network and purposefully causing codec negotiation to fail), but every time I see them they go away after a short period of time.

Since I am still unable to reproduce this, It would also be nice if someone could verify that this issue exists in the latest 1.4 branch code.
By: David Vossel (dvossel) 2009-05-04 09:39:33

I know what the problem is now. "iax2 show channels" doesn't just show channels, it also shows pending registration and pokes. If Asterisk is attempting to register or poke a peer that is unreachable, then these "Ghost" channels will appear as Asterisk waits for a response. This should not cause any performance problems. I'm developing a patch to clarify the output so you'll be able to see what the "Ghost" channels really are.
By: Alex Balashov (abalashov) 2009-05-04 09:49:05

What you are describing appears to be consistent with what I am seeing in 1.4.24, to which I upgraded from 1.4.21.2 to fix the problem. A few phantom IAX2 channels will show up briefly, nothing ever significantly in excess of actual calls in progress.

In 1.4.21.2 - and, I think, in all scenarios in which people have captured this problem in its full manifestation - what you are describing is false. It most definitely causes performance problems. I have a system that never had more than ~10 concurrent calls going through it, and usually under 4. Over the course of the day, thousands of zombie IAX2 channels had been run up until the process simply became sluggish and unresponsive.
By: David Vossel (dvossel) 2009-05-06 10:29:12

So, you can verify that upgrading to 1.4.24 solves the excess phantom IAX2 channels from occurring?
By: Alex Balashov (abalashov) 2009-05-08 13:56:05

Seemingly, yes.
By: Digium Subversion (svnbot) 2009-05-14 17:59:50

Repository: asterisk
Revision: 194557

U branches/1.4/channels/chan_iax2.c
U branches/1.4/channels/iax2-parser.c
U branches/1.4/channels/iax2-parser.h

------------------------------------------------------------------------
r194557 | dvossel | 2009-05-14 17:59:50 -0500 (Thu, 14 May 2009) | 10 lines

IAX2 "Ghost" Channels

There is a bug tracker issue where people are reporting "Ghost" channels in their 'iax2 show channels' output. The confusion is caused by channels being listed as "(NONE)" with format "unknown". These are not channels of coarse. They are usually just pending registration or poke requests, but it is confusing output. To help make sense of this I have added two columns to 'iax2 show channels'. One shows the first message which started the transaction, and the second shows the last message sent by either side of the call. This helps diagnose why the entry exists and why it may not go away.

(closes issue ASTERISK-13335)
Reported by: clive18

Review: https://reviewboard.asterisk.org/r/246/

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=194557
By: Digium Subversion (svnbot) 2009-05-15 10:40:45

Repository: asterisk
Revision: 194685

U branches/1.4/channels/chan_iax2.c
U branches/1.4/channels/iax2-parser.c

------------------------------------------------------------------------
r194685 | dvossel | 2009-05-15 10:40:45 -0500 (Fri, 15 May 2009) | 6 lines

Update to previous IAX2 "Ghost" Channels patch.

Fixed some comments made on reviewboard for the previous patch.

(issue ASTERISK-13335)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=194685
By: Digium Subversion (svnbot) 2009-05-15 15:52:20

Repository: asterisk
Revision: 194833

_U trunk/
U trunk/channels/chan_iax2.c
U trunk/channels/iax2-parser.c
U trunk/channels/iax2-parser.h
U trunk/channels/iax2.h

------------------------------------------------------------------------
r194833 | dvossel | 2009-05-15 15:52:20 -0500 (Fri, 15 May 2009) | 24 lines

Merged revisions 194557,194685 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r194557 | dvossel | 2009-05-14 17:59:43 -0500 (Thu, 14 May 2009) | 10 lines

IAX2 "Ghost" Channels

There is a bug tracker issue where people are reporting "Ghost" channels in their 'iax2 show channels' output. The confusion is caused by channels being listed as "(NONE)" with format "unknown". These are not channels of coarse. They are usually just pending registration or poke requests, but it is confusing output. To help make sense of this I have added two columns to 'iax2 show channels'. One shows the first message which started the transaction, and the second shows the last message sent by either side of the call. This helps diagnose why the entry exists and why it may not go away.

(closes issue ASTERISK-13335)
Reported by: clive18

Review: https://reviewboard.asterisk.org/r/246/
........
r194685 | dvossel | 2009-05-15 10:40:37 -0500 (Fri, 15 May 2009) | 6 lines

Update to previous IAX2 "Ghost" Channels patch.

Fixed some comments made on reviewboard for the previous patch.

(issue ASTERISK-13335)
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=194833
By: Digium Subversion (svnbot) 2009-05-15 16:08:35

Repository: asterisk
Revision: 194834

_U branches/1.6.0/
U branches/1.6.0/channels/chan_iax2.c
U branches/1.6.0/channels/iax2-parser.c
U branches/1.6.0/channels/iax2-parser.h
U branches/1.6.0/channels/iax2.h

------------------------------------------------------------------------
r194834 | dvossel | 2009-05-15 16:08:34 -0500 (Fri, 15 May 2009) | 31 lines

Merged revisions 194833 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r194833 | dvossel | 2009-05-15 15:52:12 -0500 (Fri, 15 May 2009) | 24 lines

Merged revisions 194557,194685 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r194557 | dvossel | 2009-05-14 17:59:43 -0500 (Thu, 14 May 2009) | 10 lines

IAX2 "Ghost" Channels

There is a bug tracker issue where people are reporting "Ghost" channels in their 'iax2 show channels' output. The confusion is caused by channels being listed as "(NONE)" with format "unknown". These are not channels of coarse. They are usually just pending registration or poke requests, but it is confusing output. To help make sense of this I have added two columns to 'iax2 show channels'. One shows the first message which started the transaction, and the second shows the last message sent by either side of the call. This helps diagnose why the entry exists and why it may not go away.

(closes issue ASTERISK-13335)
Reported by: clive18

Review: https://reviewboard.asterisk.org/r/246/
........
r194685 | dvossel | 2009-05-15 10:40:37 -0500 (Fri, 15 May 2009) | 6 lines

Update to previous IAX2 "Ghost" Channels patch.

Fixed some comments made on reviewboard for the previous patch.

(issue ASTERISK-13335)
........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=194834
By: Digium Subversion (svnbot) 2009-05-15 16:13:47

Repository: asterisk
Revision: 194835

_U branches/1.6.1/
U branches/1.6.1/channels/chan_iax2.c
U branches/1.6.1/channels/iax2-parser.c
U branches/1.6.1/channels/iax2-parser.h
U branches/1.6.1/channels/iax2.h

------------------------------------------------------------------------
r194835 | dvossel | 2009-05-15 16:13:47 -0500 (Fri, 15 May 2009) | 31 lines

Merged revisions 194833 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r194833 | dvossel | 2009-05-15 15:52:12 -0500 (Fri, 15 May 2009) | 24 lines

Merged revisions 194557,194685 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r194557 | dvossel | 2009-05-14 17:59:43 -0500 (Thu, 14 May 2009) | 10 lines

IAX2 "Ghost" Channels

There is a bug tracker issue where people are reporting "Ghost" channels in their 'iax2 show channels' output. The confusion is caused by channels being listed as "(NONE)" with format "unknown". These are not channels of coarse. They are usually just pending registration or poke requests, but it is confusing output. To help make sense of this I have added two columns to 'iax2 show channels'. One shows the first message which started the transaction, and the second shows the last message sent by either side of the call. This helps diagnose why the entry exists and why it may not go away.

(closes issue ASTERISK-13335)
Reported by: clive18

Review: https://reviewboard.asterisk.org/r/246/
........
r194685 | dvossel | 2009-05-15 10:40:37 -0500 (Fri, 15 May 2009) | 6 lines

Update to previous IAX2 "Ghost" Channels patch.

Fixed some comments made on reviewboard for the previous patch.

(issue ASTERISK-13335)
........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=194835
By: Digium Subversion (svnbot) 2009-05-15 16:17:08

Repository: asterisk
Revision: 194836

_U branches/1.6.2/
U branches/1.6.2/channels/chan_iax2.c
U branches/1.6.2/channels/iax2-parser.c
U branches/1.6.2/channels/iax2-parser.h
U branches/1.6.2/channels/iax2.h

------------------------------------------------------------------------
r194836 | dvossel | 2009-05-15 16:17:08 -0500 (Fri, 15 May 2009) | 31 lines

Merged revisions 194833 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r194833 | dvossel | 2009-05-15 15:52:12 -0500 (Fri, 15 May 2009) | 24 lines

Merged revisions 194557,194685 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r194557 | dvossel | 2009-05-14 17:59:43 -0500 (Thu, 14 May 2009) | 10 lines

IAX2 "Ghost" Channels

There is a bug tracker issue where people are reporting "Ghost" channels in their 'iax2 show channels' output. The confusion is caused by channels being listed as "(NONE)" with format "unknown". These are not channels of coarse. They are usually just pending registration or poke requests, but it is confusing output. To help make sense of this I have added two columns to 'iax2 show channels'. One shows the first message which started the transaction, and the second shows the last message sent by either side of the call. This helps diagnose why the entry exists and why it may not go away.

(closes issue ASTERISK-13335)
Reported by: clive18

Review: https://reviewboard.asterisk.org/r/246/
........
r194685 | dvossel | 2009-05-15 10:40:37 -0500 (Fri, 15 May 2009) | 6 lines

Update to previous IAX2 "Ghost" Channels patch.

Fixed some comments made on reviewboard for the previous patch.

(issue ASTERISK-13335)
........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=194836