[Home]

Summary:ASTERISK-00823: Asterisk + chan_sip + chan_capi leaks pipes
Reporter:gcc (gcc)Labels:
Date Opened:2004-01-13 11:27:48.000-0600Date Closed:2011-06-07 14:00:45
Priority:BlockerRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:Under some circumstances that I haven't determined yet, Asterisk will not close a pipe() that it has opened. This leads to the process file descriptor table filling up with pairs of pipes, like this:

asterisk 10964 root  257r  FIFO        0,5           2008558 pipe
asterisk 10964 root  258w  FIFO        0,5           2008558 pipe
asterisk 10964 root  259r  FIFO        0,5           2035715 pipe
asterisk 10964 root  260w  FIFO        0,5           2035715 pipe
asterisk 10964 root  263r  FIFO        0,5           2195281 pipe
asterisk 10964 root  264w  FIFO        0,5           2195281 pipe

****** ADDITIONAL INFORMATION ******

Eventually Asterisk is unable to create any more pipes, and crashes or blocks. I think it blocks, since it's impossible to connect to the console anymore.

SIP is configured with 7 peers: one Cisco 7940, two Cisco ATA-186s with 2 channels each, and two Grandstream Budgetones.

CAPI is configured using KPJ's chan_capi 0.2.5c on three AVM Fritz PCI cards, using the kernel hack to support more than one Fritz.

Perhaps it's worth adding some debugging code, for example to keep a list of open pipes along with the source file and line where they were opened?
Comments:By: ww (ww) 2004-01-13 13:02:48.000-0600

a quick grep through the source shows that
chan_sip.c and chan_capi.c do not use the
pipe(2) system call.

% find . -name '*.[ch]' -exec grep 'pipe(' '{}' /dev/null ';'          
./channel.c:                                    pipe(pvt->alertpipe)) {
./apps/app_agi.c:       if (pipe(toast)) {
./apps/app_agi.c:       if (pipe(fromast)) {
./apps/app_agi.c:               if (pipe(audio)) {
./apps/app_festival.c:        if (pipe(fds)) {
./apps/app_mp3.c:       if (pipe(fds)) {
./channels/chan_alsa.c: res = pipe(cmd);
./channels/chan_alsa.c: res = pipe(sndcmd);
./channels/chan_iax.c:          if (pipe(com)) {
./channels/chan_iax2.c:         if (pipe(com)) {
./channels/chan_oss.c:  res = pipe(sndcmd);
./db1-ast/recno/rec_get.c:__rec_fpipe(t, top)
./db1-ast/recno/rec_get.c:__rec_vpipe(t, top)
./pbx/pbx_gtkconsole.c: if (pipe(clipipe)) {
./res/res_musiconhold.c:        if (pipe(fds)) {        
./res/res_musiconhold.c:        if (pipe(moh->pipe)) {
By: ww (ww) 2004-01-13 13:09:05.000-0600

the pipe(2) call in channel.c is in ast_channel_alloc(),
and the file descriptors it opens are closed correctly
in ast_channel_free(). so unless entire channels are
not being freed, the problem lies elsewhere...

We need to track down the circumstances under which this
happens for you. Can you maybe add some ast_log calls to
your source?

I would put:

ast_log(LOG_DEBUG, "creating pipe");

near the pipe(2) calls that look likely to you, as well
as

ast_log(LOG_DEBUG, "closing pipe");

near the corresponding close(2) calls  on the file descriptors
By: nickkknight (nickkknight) 2004-01-14 13:16:49.000-0600

I haven't gone down to this level as yet, but this is the exact situation I have. I upgraded all the hardware just to ensure it wasn't hardware issues - so now it has brand new hardware!

It is using
chan_capi 0.3.0
asterisk was the latest as of 5 days ago.

I have calls coming in via capi, which first ring the sip clients then if that times out they go into a queue, it could ring the sip clients but not create a channel when the call was picked up. I have also noticed that the memory on the system slowly builds up (512MB) over a period of a day or so.

I have

5 pingtel expressa clients
3 Snom clients

it is also using moh

I will happily do any debugging required if you point me in the right direction.
By: gcc (gcc) 2004-01-22 08:16:06.000-0600

ww: chan_capi uses pipes, but it's not in the standard Asterisk source, so you wouldn't find it with your grep.

I found one way to reproduce this repeatably: place a call from an external ISDN phone, over CAPI into Asterisk, then out to a SIP phone. Hang up before the remote side answers. Two pipes are leaked.

nickkknight: can you confirm that this has the same effect for you?
By: nickkknight (nickkknight) 2004-01-22 11:18:44.000-0600

How do I test if pipes have been leaked, I added the ast_log when I first looked into it and didn't notice anything but will test the above senario when perhaps you can let me knwo about pipes (just getting past my newbie phase!).

Thanks

Nick
By: gcc (gcc) 2004-01-22 11:25:48.000-0600

lsof -p <pid-of-asterisk> will show output like:

asterisk 10964 root 257r FIFO 0,5 2008558 pipe
asterisk 10964 root 258w FIFO 0,5 2008558 pipe

Every line is an open pipe, and the number is the 4th column (257 and 258 in this case). Run lsof before the test, and make a note of the last six pipe numbers. Then run the test, and then run lsof again.

If new numbers appear in the top six, then your asterisk is leaking pipes.
By: nickkknight (nickkknight) 2004-01-22 15:13:45.000-0600

Everything I do seems to leak a pipe - lsof -p :

asterisk 661 root   22w  FIFO        0,5           3239 pipe
asterisk 661 root   23r  FIFO        0,5           4188 pipe
asterisk 661 root   24w  FIFO        0,5           4188 pipe
asterisk 661 root   25r  FIFO        0,5           4441 pipe
asterisk 661 root   26w  FIFO        0,5           4441 pipe
asterisk 661 root   27r  FIFO        0,5           4697 pipe
asterisk 661 root   28w  FIFO        0,5           4697 pipe
asterisk 661 root   29r  FIFO        0,5           5469 pipe
asterisk 661 root   30w  FIFO        0,5           5469 pipe
asterisk 661 root   31r  FIFO        0,5           6006 pipe
asterisk 661 root   32w  FIFO        0,5           6006 pipe
asterisk 661 root   41r  FIFO        0,5           6281 pipe
asterisk 661 root   42w  FIFO        0,5           6281 pipe

wether the call is answered or not - and wether it was origionated or terminated.
By: Mark Spencer (markster) 2004-01-23 10:10:38.000-0600

Can you narrow it down to sip or capi?  What if you terminate each call in an application instead of in a bridge?
By: nickkknight (nickkknight) 2004-01-23 12:13:56.000-0600

Just to sanity check what I am doing I am running

lsof -p 661|grep pipe -c

which returns the number of open pipes.

I call the capi channel it increments by two every time - I have now also tested it going into meetme and to sip. If you use sip to sip or sip to application then it appears to close them correctly.

So it would appear that it is chan_capi which is doing the leaking?
By: gcc (gcc) 2004-01-23 12:21:17.000-0600

Looks like chan_capi is at fault.
By: nickkknight (nickkknight) 2004-01-23 15:30:31.000-0600

I have just added a couple of lines in chan_capi to close some pipes which look as though they are not been closed properly elsewhere. It seems to work (well the pipes are not leaking anymore) and I have forwarded it onto kapejod to have a look at to make sure it is sencible.
By: philipp2 (philipp2) 2004-01-25 10:19:23.000-0600

hm... maybe you would want to test this with chan_capi 0.3.0? Unless kapjod already confirm this, of course.
By: nickkknight (nickkknight) 2004-01-26 03:19:41.000-0600

I already am testing it with 0.3.0
By: farnwomt (farnwomt) 2004-01-27 18:05:31.000-0600

I would like to suggest this fix to chan_capi.c, not properly tested yet, but I only wrote it this evening:

void remove_pipe(int PLCI) {
   struct capi_pipe *p,*ptmp;
               
   ast_mutex_lock(&pipelock);
   p = pipelist;  
   ptmp = NULL;
   while (p) {
       if (p->PLCI == PLCI) {
           <b>/* Start of patch by Maxima Systems */
           if ( p->fd )
               close(p->fd);
           if ( p->i != NULL )
           {
               close(p->i->fd);
               p->i->fd = 0;
           }
           if ( p->c != NULL )
           {
               if ( p->c->fds[0] == p->fd )
               {
                   p->c->fds[0] = 0;
               }
           }
           p->fd = 0;
           /* End of Maxima Systems patch */</b>
           if (ptmp == NULL) {
               // mypipe == head of pipelist
               pipelist = p->next;
               free(p);
               if (option_verbose > 4) {
                   ast_verbose(VERBOSE_PREFIX_3 "removed pipe for PLCI =
%#x\n",PLCI);
               }
               break;
           } else {

I have forwarded a copy of this to the author of chan_capi so hopefully something similar will appear in chan_capi in the not too distant future.

Mtf

edited on: 01-27-04 17:51

edited on: 01-27-04 17:54
By: Private Name (falves11) 2015-02-05 21:10:18.120-0600

I have the exact same issue with PJSIP and the app Transfer.
asterisk 4077 root *450w FIFO 0,8 0t0 110430221 pipe
asterisk 4077 root *451r FIFO 0,8 0t0 110429239 pipe
asterisk 4077 root *452w FIFO 0,8 0t0 110429239 pipe
asterisk 4077 root *453r FIFO 0,8 0t0 110417598 pipe
asterisk 4077 root *454w FIFO 0,8 0t0 110417598 pipe
asterisk 4077 root *455r FIFO 0,8 0t0 110426507 pipe
asterisk 4077 root *456w FIFO 0,8 0t0 110426507 pipe^

After a few hours the command
lsof | grep asterisk
goes to 700.000+
I cannot access Asterisk via asterisk -r.
SVN-branch-13-r431555M
Note: I only use PJSIP as channels, nothing else is loaded.