[Home]

Summary:ASTERISK-03764: [patch] Generic rtp jitterbuffer for Asterisk
Reporter:zoa (zoa)Labels:
Date Opened:2005-03-25 06:03:31.000-0600Date Closed:2006-05-31 12:03:26
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 2006-02-08-trunk-9215.patch
( 1) 2006-02-20-svn-1.2-rev10558.patch
( 2) 2006-02-20-svn-trunk-rev10536.patch
( 3) 62796252-98013356-2006-02-27_13-07-28.ogg
( 4) ast_fixed_jb_SIP#1001499-4364--SIP#pstngw2-d008.log.gz
( 5) ast_fixed_jb_SIP#pstngw2-d008--SIP#1001499-4364.log.gz
( 6) ast_jb-1.2.0.patch2
( 7) ast_jb-1.2.0.patch3
( 8) ast_jb-1.2.0.patch4
( 9) ast_jb-svn.patch
(10) badaudiojblogs.tar.gz
(11) chan_zap.patch
(12) diff-new-jb-2006-02-28.patch
(13) jbdebug.tar.gz
(14) jb-rtp-sip.scx.new.patch
(15) jitterbuf-rtp-sip.patch
(16) oej-jb.2006.03.01.patch
(17) oej-jb.patch
(18) sip_jitter_buffer_update.diff
(19) trunk-2006-01-26.patch
Description:(Bug marshal note: This description is for the first version, that was a purely
RTP/SIP jitter buffer. Read on further down to follow the development into
a more generic solution)

So here they are, the long awaited patches for chan_sip.
I know they do not have the correct formatting, and they have cpp comments in there, we will fix all that in the final versions (Lets first get it to work properly).

I didnt check if this one applies to the latest cvs head if not we will fix it.

It has issues under high load, we are still trying to fix that using different approaches, so far without luck.

Please leave this bug open, even without if we dont reply very fast. Its work in progress.
Comments:By: zoa (zoa) 2005-03-25 06:11:42.000-0600

disclaimer on file... yada yada....

By: Brian West (bkw918) 2005-03-25 08:04:45.000-0600

w00t!

/b

By: m00bh000 (m00bh000) 2005-04-30 18:01:58

Can I just patch the current CVS-HEAD with this? Do I need to set options in my sip.conf?

Cheers,

Tim

By: zoa (zoa) 2005-05-03 23:04:41

you should be able to patch chan_sip with this without altering sip.conf

But i suggest to wait just a little more, a new and much improved patch is coming soon.

By: gourij (gourij) 2005-05-05 11:47:56

I am not sure if your new and improved version will address the some of the issues we have been seeing with the current patch.  So I thought I list them below:
1. Patched the CVS-HEAD from 03-31-05.
2. The CPU utilization is jumping to 50% when we use JB vs 1-2% while not using it (by not defining USE_JB)
3. Seeing a lot of voice quality issues.  Seems almost like a lot of packets get stuck in the JB (could be related to CPU loading).
4. rtp_read_callback does not get called for more than 100ms and then 5 times back to back. We have seen this multiple times during a call. The packets are looking good over ethernet.

We are wondering if we have an issues with the linux kernel. What version is recommended to use? Any patches required? We are running Asterisk on a dedicated machine running RH9 (kernel version 2.4.20-8).
Have you seen similar behavior?

By: zoa (zoa) 2005-05-05 17:59:45

go have a look at astertest.com, i posted some completely different patches there. (Without the formatting needed for mantis.)

These should be a lot more stable. (we did find one person who has issues with it, investigating what exactly is going on now).

By: Pash (opa__) 2005-05-21 01:41:41

zoa. that is about sip/jb patch. i need this for h323. so i am able help you to debug/test. :)

By: Brian West (bkw918) 2005-06-09 14:52:24

Zoa can we get an update?

By: zoa (zoa) 2005-06-15 03:45:35

i just added the two latest patches, live from astricon.

They are supposed to be stable. (your asterisk should NOT crash with them) but, they don't work for all phones.

They should work for: Cisco, snom, grandstream, x-lite, fritzbox
they are known to have problems with: some draytek firmware version, allied telesyn.

Please report what phones seem to work, (and if possible include the firmware).

By: outcast (outcast) 2005-06-20 12:07:30

Has anyone noticed problems with inband dtmf with this patch or is it just me?

By: Joe Antkowiak (antkojm1) 2005-06-20 13:41:51

I've had false positives with inband dtmf with this running, but I haven't had time to look at it more and confirm that this was causing it.

By: zoa (zoa) 2005-06-23 03:24:13

I dont see how this could come from the jitter buffer part, they could only be more correct then before using the jitter buffer.

But, with the jitterbuffer, this also enables the PLC things and the plc might influence it.

Maybe try disabling the PLC (codecs.conf iirc).

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-06-28 05:42:09

compiling fails on current CVS

rtp.c: In function `ast_rtp_jb_enable':
rtp.c:759: error: structure has no member named `max_jitterbuf'
rtp.c:760: error: structure has no member named `resync_threshold'
rtp.c:761: warning: implicit declaration of function `jb_setinfo'
rtp.c: At top level:
rtp.c:95: warning: `rtp_framelog_counter' defined but not used
rtp.c:410: warning: `recalc_rxcore' defined but not used
make: *** [rtp.o] Error 1

By: zoa (zoa) 2005-06-30 01:46:55

hmm, we will check it and post a new one if necessary.

Zoa

By: zoa (zoa) 2005-06-30 03:01:20

Ok, I have removed the old patches and added only the last one. Its from cvs-head 13.06.2005, but successfuly applies against today cvs-head

By: John Mylchreest (jmylchreest) 2005-06-30 07:23:05

is it at all possible to get this patch to work on the stable releases? or is there some other SIP/RTP jitter buffering solution we can use?

By: zoa (zoa) 2005-07-01 06:25:37

We could get the jitter buffer to work on cvs-stable BUT it would be a lot of work and i'm not willing to lose any more time on this.

I tried the dtmf problem, it seems to be just cvs-head that is broken, DTMF inband doesnt work with or without jitter buffer for me.

By: gourij (gourij) 2005-07-01 18:34:31

From my testing using G711u, I found that the inband dtmf works fine with cvs-head (06/13/2005) and the jb-sip-rtp patch.  But rfc2833 DTMF has problems.  I hear a lot of delays (upto 5 secs) as soon as I send DTMF from the SIP client to PSTN.  And JB takes atleast a min to catch up.

When JB is disabled it works fine (i.e., no delays).

I am using X-Pro client.



By: zoa (zoa) 2005-07-06 02:06:42

We will investigate this a little further, last tests we did with xlite, the inband didnt work and the dtmf worked fine.

Zoa

By: Olle Johansson (oej) 2005-07-20 13:31:51

We will get this to work with the release version when 1.2 is out of the door...

Zoa: Any updates? We're closing the shop soon, going into feature freeze...

By: zoa (zoa) 2005-07-31 16:32:56

We are testing a new version with fixes for the dtmf problems, the main programmer is on a vacation atm, so no updates for a while now...

By: Olle Johansson (oej) 2005-08-01 01:38:41

That means that we propably won't have the jitter buffer for SIP/RTP in 1.2.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-08-03 06:12:23

IMHO, SIP jitterbuffer is critical, and 1.2 should by no means be released before this is in. without the jitterbuffer, asterisk can't compare with commercial solutions.

By: philipp2 (philipp2) 2005-08-03 06:21:25

Together with ARA/Realtime this is probably the most important feature of 1.2, I agree. So if this JB can't make it into 1.2 then I'd suggest to permit it into 1.2.1 even though its a new feature and not a fix. That way the published time schedule for 1.2 wouldn't be affected.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-08-03 06:26:11

I'd say put more work into stabilising JB and rather postpone 1.2 until JB is stable. Perhaps put JB into HEAD already to make more people test it. we know JB isn't stable yet, but that's the whole purpose of HEAD...

By: zoa (zoa) 2005-08-05 10:25:50

well, nobody will notice any problem with the jitter buffer now, except when doing it under heavy load. Which would be a pain in the ass to debug (and point to the jitter buffer) if a lot of other changes were also made.

By: Brian West (bkw918) 2005-08-18 09:40:27

Is this updated for CVS-HEAD as of today?

/b

By: Michael Jerris (mikej) 2005-08-24 01:08:40

suspended due to no response.  Please update with up to date code and comment on status and reopen when ready.  Is this ready to go in, with configuration defaults to off so that it can be more thoroghly tested, or should we be waiting until post 1.2?

By: Brian West (bkw918) 2005-08-25 12:30:55

bkw_ <39>:patch -p1 < sip.diff
bkw_ patching file rtp.c
bkw_ Hunk #1 succeeded at 68 (offset 1 line).
bkw_ Hunk #2 succeeded at 161 (offset 1 line).
bkw_ Hunk #3 succeeded at 381 (offset 2 lines).
bkw_ Hunk #4 succeeded at 975 with fuzz 1 (offset -10 lines).
bkw_ Hunk ASTERISK-1 succeeded at 1667 (offset 18 lines).
bkw_ Hunk ASTERISK-2 succeeded at 2429 (offset 1 line).

By: Brian West (bkw918) 2005-08-25 12:37:34

but you get one way audio.

/b

By: Brian West (bkw918) 2005-08-25 12:58:47

fixed and updated.

/b

By: slav (slav) 2005-08-25 13:16:55

Uploaded new patch "jb-rtp-sip.scx.new.patch" against cvs-head from 25.08.2005.
The new SIP jitterbuf no longer uses the stevek's jb algorythm - it uses sipmler but more stable fixed size jitterbuf algorythm - no longer problems with the DTMF or with timestamp jumps. A simple jb config is added to sip.conf.sample. However, it also suffers from a memleak problem on very high load (when the CPU usage is at 99%). Any help will be appreciated.

By: Olle Johansson (oej) 2005-08-25 13:47:12

This code does not follow the coding guidelines. It needs cleaning up when we have something that works.

Do we use assert() anywhere else in Asterisk?

Mailed out to mailing list, asking for tests.

By: slav (slav) 2005-08-25 14:11:33

Its a test code still. I think the formating is its last problem ...
Moreover some compilers generate an error due to a C++ style local variable declaration - this will be fixed

By: Olle Johansson (oej) 2005-08-25 14:27:11

Ok, Slav. If you yourself consider this test code, we do have to move this to [post 1.2]. However much I want to have this in 1.2, I think you agree with me that we don't want untested code in the new release.

I hope you take this right, I really appreciate all of the hours you have put into this and all the work. We will get it right, but not for 1.2.

What do you think?

By: zoa (zoa) 2005-08-25 14:36:12

we agree, this is not for 1.2

It needs more testing (this last patch is completely untested).

Then it needs even more testing on a range of different endpoints...

By: Brian West (bkw918) 2005-08-27 17:03:18

I think it needs to be in 1.2 with an ifdef maybe?  We need it out there for testing.  Otherwise why have sip without a jitter buffer?  RTP with no jitter buffer is totally unacceptable, Its shocking how long it has gone without one.

/b

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-08-27 18:52:49

I can second that
Using asterisk for SIP termination without a jitterbuffer is worthless

By: deti (deti) 2005-08-28 10:06:30

I think a RTP jitterbuffer is neccessary when poorly connected (DSL) users are connecting to/via asterisk. Otherwise some bad constructs with SER and mediaproxy are neccessary.

By: Martin Vit (festr) 2005-08-28 10:26:06

jitterbuffer is not only for "bad" connections. Almost wireless devices sometimes have jitter. LANs due to QoS (shaping) is jittery too. So without jitter it is unusuable for almost end users without perfect QOS (and who this have?). I think, that jitter for sip is mission critical for commercial use.

By: Brian West (bkw918) 2005-08-28 17:45:16

Ok here is something I just thought of.  How hard would It be to just include hooks to make the jitter buffer pluggable?  Then if/when we load a jitter buffer its just a .so file that will allow it to be there or not .. even totally rewritten or changed without dinking with the core code.

/b

By: zoa (zoa) 2005-08-29 15:34:02

there is an ifdef in the code already, so it could be activated on compile time.

The .so thing would be hard i think, especially considering what we went through already.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-09-09 05:54:32

We need this urgently
Is there a way to speed up the debugging?
We will gladly pay for this work

roy

By: Chris Bianchi (voicechat) 2005-09-28 12:15:04

Just some feedback.  I tried the patch with cvs from 8/25.  I had no problems getting it to build and run but when makign a test call from a SIP (xten softphone) to SIP (budgettone 101) there seems to be delay in the call introduced by the JB.  When the call started there was virtually no delay (<.5 sec) but as the call progressed (after about a minute) the delay continued to grow to the point where I could count to 7 (about 7 seconds) before I could hear myself at the other end of the call (having both phones next to each other obviouslly on the same server same LAN, one on mute).  

If ifdef'ed the jb back out and the delay was back to normal and didn't get any worse as time went on.  

Are there any new diff's for a more recent CVS HEAD that just haven't been posted yet?



By: Roy Sigurd Karlsbakk (rkarlsba) 2005-09-28 12:23:43

Perhaps this should go into CVS by now? We need this in HEAD. Asterisk really is BAD BAD BAD on bad DSL lines. I talked to oej about it and we agreed very much about the subject. Get this into CVS HEAD and make compiler defines in the makefile so that people can comment it _out_. That's the only way of getting this tested toroughly.

(methinks)

roy

By: Guenther Starnberger (gst) 2005-09-28 12:55:47

i second that. IMO this is one of the most important SIP related features. of course it's up to digium and the other developers to decide what goes in and what not, but from a users perspective i would prefer having this feature in 1.2 (or 1.2.1).

By: Serge Vecher (serge-v) 2005-09-28 14:35:40

Slav, could you please update the patch to latest HEAD and apply proper formatting/commenting rules? Let's push the testing on this one!



By: zoa (zoa) 2005-09-29 03:14:13

we are first trying to make a channel.c based jitter buffer to see if its any better.

By: M March (cowmix) 2005-10-05 13:21:26

I am sorry to clutter up this 'ticket'.. but I am going to try to get this patch to work against the current *@H Beta.. so I would like to know what this is supposed to do before I put a lot of work into that...

My question is, doesn't both sides of the the connection need to implement jitter control for this concept to work? Meaning, if I 'connect' a SIP device that does not have good jitter compensation (over a WAN) to an Asterisk server that has this patch applied, I will still suffer from 'jitters'... right?

My question is doesn't take two to tango in the fight to end jitters?

thanks!



By: Adam Gundy (adamgundy) 2005-11-04 09:23:57.000-0600

hi. I just upgraded our CVS-HEAD asterisk from the original jitterbuf-rtp-sip patch (which we have been using successfully for months now) to the jb-rtp-sip.scx.new patch. Our (almost) production system crashed all the time in the new code - scx_jb_put() would assert on line 314 all the time. A quick look at the thing under GDB makes me think it was a race condition (ie not enough locking somewhere). Our asterisk box is a twin Xeon 64, so it's probably far more likely to find races than a uniprocessor.

I've had to back out to the old code now, but I guess I could put it in again over the weekend if it would help?

By: Chris Bianchi (voicechat) 2005-11-04 11:32:51.000-0600

This thread has been quiet for a few weeks. Any news on moving/adding the JB to the channel.c or should we continue to test and try to use the current patches from 8/25?  Just curious where this is going.

By: zoa (zoa) 2005-11-30 12:07:34.000-0600

I just posted the first version of a channel independant jitter buffer.

It still needs quality testing on all devices out there, but the code itself should be stable. (No deadlocks or crashes), the patches are made for asterisk 1.2.0

We have a bunch of devices lined up for testing in our lab, results will be posted here.

By: stevekstevek (stevekstevek) 2005-11-30 12:38:18.000-0600

Very interesting!

I just briefly looked over this;  do you have any design notes to go along with it?

Questions I have:
1) Does this perform jitterbuffering only on bridges (i.e. SIP->ZAP), or does it also perform jitterbuffering when going SIP->Application.  Because the latter is pretty important (for proper re-ordering for non-real-time apps like Record/VM, and for real-time apps like meetme/app_conference, etc).

2) How does this interact with Monitor() and friends?

I guess the plan is to use this infrastructure with other channels besides SIP, right?  

These questions are probably answered in the code, but I did only take 2 mins to read over it once..  It seemed like the get/put only happened in the bridge -- and I guess I wasn't sure what the zap changes were for..

By: slav (slav) 2005-12-01 08:32:20.000-0600

1. Yes, the jitterbuffering is performed only in a bridge. It is currently implemented in ast_generic_bridge(). Thus it won't be applied in the chan->app case. However, apps like Record, VM, Meetme are doing stuff, very similar to ast_generic_bridge(), so the same technique may be used in them.

I thought about implemening jb in (generic) ast_read(), but it seemed more complicated to me, and the configuration seems to be a problem too - the need of a jb  actually depends on the receiving leg, not on the sending one. I mean only the receiving side can knows if it can (should) handle jitter and this works fine with two bridged channels. It's very interesting question how this can be generally extended to support the chan->app case. Maybe some dummy channel will help this.

2. I have never tested this with Monitor. Theres nothing done in this patch to interact with Monitor.

The new jb can be easily extended to support each channel tech, capable to transmit timing information (timestamps). Currently it's done for rtp only, but with a few lines of code can be used in IAX2 for example. Also, the jb usage is abstracted and different implemetaions can be chosen at runtime (i.e. fixed, adaptive, some exotic new alg....)

chan_sip and chan_zap are modifed by the patch only because the jb is per technology configurable and it is disabled by default. So they need to read the configuration and to enable jb. Sample config files for sip and zap are included in the patch.

By: stevekstevek (stevekstevek) 2005-12-01 08:43:52.000-0600

Slav wrote:
> I thought about implemening jb in (generic) ast_read(), but it seemed more
> complicated to me, and the configuration seems to be a problem too - the need of
> a jb actually depends on the receiving leg, not on the sending one. I mean only
> the receiving side can knows if it can (should) handle jitter and this works
> fine with two bridged channels. It's very interesting question how this can be
> generally extended to support the chan->app case. Maybe some dummy channel will > help this.

Can't you determine if you're bridged, and who you're bridged to in ast_read just like the IAX2 JB implementation determines this (i.e. with
 ast_bridged_channel(iaxs[fr->callno]->owner) &&    
 (ast_bridged_channel(iaxs[fr->callno]->owner)->tech->properties & AST        _CHAN_TP_WANTSJITTER)

Then, you could just implement things in ast_read, and it would be enabled for all cases except when you're bridged to a channel that wantsjitter?

By: slav (slav) 2005-12-01 09:21:46.000-0600

In principle yes (if I'm getting your idea). I'll need to modify also ast_waitfor() to wake up when a frame should be delivered for a channel. Than ast_read() will see if we have a frame received (to put it into jb) or it should get (or interpolate) one from jb.
It doesn't seems too far from what I have now, however currently I'm not sure how this can be done in ast_read() - I think a lot of modifications should be made. Moreover, I don't know if I will have the time for this now, nor if this will work fine and what additional problems will cause. Maybe its better to start with this and change it at some future point. It should work in SIP->ZAP case now, which IMHO will be the most used scenario.

By: philipp2 (philipp2) 2005-12-01 12:51:05.000-0600

Hm... can the jb be enabled/disabled per user?

By: slav (slav) 2005-12-02 11:10:57.000-0600

No - its configurable per technology. It won't be easy to do this now. Moreover, the new jb needs a lot of testing. After we consider it for stable we can start thinking about implementing per user/peer configuration.

By: philipp2 (philipp2) 2005-12-02 13:41:20.000-0600

Understood. We've been running this for a couple of days on two boxes now, no apparent glitch so far (but granted, we don't use much fancy stuff like Monitor or MeetMe).

Can you provide some instructions how to look into what the jb is doing, e.g. smth along the line of "CLI> IAX2 jb debug"?

By: Ray Van Dolson (rayvd) 2005-12-02 14:01:09.000-0600

I plan to begin testing this in the next week or so.  We're basically doing SIP/RTP bridging between Asterisk and an AudioCodes gateway and previously had problems using the jitter buffer in tandem with the AudioCodes.

Didn't have any problems with SIP bridging between two Asterisk installs.

Anyone out there using the current incarnation of the patch with a similar setup?

By: deti (deti) 2005-12-03 04:19:04.000-0600

I've got two different setups running where asterisk serves SIP-Clients (canreinvite=no):

* asterisk with TE405 card connected to PSTN (zap-sip bridged)
* asterisk with external Cisco SIP to PSTN Gatway (sip-sip bridged)

Now all of the time ast_jb seems to work properly. But with "jb-log = yes" there was an echo on one side and after 4-5 minutes one direction sounded choppy. BTW: the /tmp/ast_fixed_jb_SIP* files are not very helpful. There should be something that works from the asterisk console.

When bridging betweeen iax2 and sip with jitter buffers enabled there is a problem. It's really hard to hear anything. It seems there is a interference of two different jitter buffer mechanisms. Can anyone confirm this?

A reload during ast_jb is active and bridged to iax2 causes an assertion to fail:

0 0x40119db7 in raise () from /lib/tls/libc.so.6
1 0x4011b549 in abort () from /lib/tls/libc.so.6
2 0x4011358a in __assert_fail () from /lib/tls/libc.so.6
3 0x080d5035 in scx_jb_put (jb=0xd2, data=0x820bf70, ms=136369328, ts=135356127, now=15377) at scx_jitterbuf.c:136
4 0x080d660d in jb_put_scx (jb=0x0, fin=0x6, now=0) at abstract_jb.c:744
0000005 0x080d5f17 in ast_jb_put (chan=0x817b8d8, f=0x81c56b8) at abstract_jb.c:373
0000006 0x08068bec in ast_channel_bridge (c0=0x817b8d8, c1=0x817d448, config=0x4106be80, fo=0x4106b388, rc=0x4106b38c) at channel.c:3282
0000007 0x40413579 in ast_bridge_call (chan=0x817b8d8, peer=0x817d448, config=0x4106be80) at res_features.c:1312
0000008 0x40a8fd73 in dial_exec_full (chan=0x817b8d8, data=Variable "data" is not available.
) at app_dial.c:1558
0000009 0x40a92a6a in dial_exec (chan=0x0, data=0x6) at app_dial.c:1600
0000010 0x08093899 in pbx_extension_helper (c=0x817b8d8, con=Variable "con" is not available.
) at pbx.c:544
0000011 0x08095095 in __ast_pbx_run (c=0x817b8d8) at pbx.c:2220
0000012 0x08096b8c in pbx_thread (data=0x0) at pbx.c:2507
0000013 0x4003c693 in start_thread () from /lib/tls/libpthread.so.0
0000014 0x401aab7a in clone () from /lib/tls/libc.so.6



By: stevekstevek (stevekstevek) 2005-12-03 07:24:28.000-0600

"When bridging betweeen iax2 and sip with jitter buffers enabled there is a problem. It's really hard to hear anything. It seems there is a interference of two different jitter buffer mechanisms. Can anyone confirm this?"

If you're bridging between two VoIP channels, the jitterbuffers should both be automatically bypassed, as this is the situation where you don't want them.  (because the other endpoints should be doing the jitterbuffering).

By: deti (deti) 2005-12-03 10:18:36.000-0600

Right normally I wouldn't do that either. IMHO it makes sense testing these scenarios. I thought finding possible glitches, races and crashes might help...

By: slav (slav) 2005-12-06 03:55:05.000-0600

10x deti. That definitely was a bug. Its fixed now. Now you shouldn't see any jbs on the iax2->sip direction. The reason is simple - to perform jb, the sending side should provide timing information in the frames (ast_frame). Currently only rtp.c is modified to add such and you can see jb only on sip->some chan direction.

By: deti (deti) 2005-12-06 16:08:50.000-0600

Thanks for fixing the bug. Now still the same problem occures when bridging between iax2 and zap. This is no freaky setup - it's something you really need. Can anybody confirm this?



By: slav (slav) 2005-12-07 06:15:05.000-0600

Very strange! In iax2<->zap bridge the sip jb shouldn't be enabled. iax2 and zap doesn't provide timestamps in the generic ast frames they produce - jb is created only when the first frame with timing info is received. Moreover, chan_iax2 is not configured to enable jb, nor it registers the tech property AST_CHAN_TP_CREATESJITTER. zap will try to use jb on its receive side only if you set jb-enabled=yes and jb-forced=yes in zapata.conf. But the lack of timing info will prevent it.
deti, can you confirm that a problem occures in iax2<->zap bridge when you compile asterisk with the AST_JB flag set?

By: deti (deti) 2005-12-07 11:10:16.000-0600

I have set jb-enabled=yes and jb-forced=yes in zapata.conf
With jb-forced=no bridging works properly and jitter buffers are only created when coming from sip. Do you think it makes sense having a jb-forced parameter? IMHO a user will always be tempted to enable something :)

By: slav (slav) 2005-12-07 11:44:59.000-0600

When the sending chan cannot create jitter or the receiving one can accept jitter, there's basically no need of a jb. However, there are cases you'll need one, because of a sophtphone with poor jb or (I don't know if this can actually happen?) broken PSTN line with losses. The jb-force param is present to let you handle such kind of cases. This is for SIP. For ZAP you're maybe right, because if we have incomming frames with timestamps included (which is required to create jb), this generally means the frame is comming from a VoIP channel which can create jitter and jb will be used anyway, disregarding the force flag.
Aside from this I'm not sure I understand your last 2 posts. jb-force=yes in zapata.conf isn't enough to create jb on zap. The audio should comes from SIP. If the audio is comming from IAX, there shouldn't be jb in any case. So, is there any problem with the iax2<->zap bridge when jb-force=yes in zapata.conf? No audio? reload problem? Is there jb created in this case?

By: deti (deti) 2005-12-07 12:09:08.000-0600

have a look at this:

Dec  7 20:06:16 DEBUG[7482]: chan_zap.c:8736 pri_dchannel: Queuing frame from PRI_EVENT_PROCEEDING on channel 0/1 span 1
   -- Zap/1-1 is proceeding passing it to IAX2/baycom-16385
Dec  7 20:06:17 DEBUG[7482]: chan_zap.c:1418 zt_enable_ec: Enabled echo cancellation on channel 1
   -- Zap/1-1 is ringing
Dec  7 20:06:21 DEBUG[7482]: chan_zap.c:1398 zt_enable_ec: Echo cancellation already on
   -- Zap/1-1 answered IAX2/baycom-16385
Dec  7 20:06:21 DEBUG[7487]: chan_iax2.c:6653 socket_read: Ooh, voice format changed to 256
   -- ***[JB LOG]*** fixed jitterbuffer created on channel Zap/1-1

How does this look like?

By: zoa (zoa) 2005-12-08 04:47:40.000-0600

new patch added!

By: slav (slav) 2005-12-08 05:07:48.000-0600

Well, it seems like in chan_iax2 some frames are not zero initialized. The new ast_jb-1.2.0.patch3 modifies chan_iax2 to zero the timing info flag when queueing a frame. Unfortunatly I don't have the possibility to test this now, but it should fix the problem.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-08 06:53:05.000-0600

attached file badsound.ogg is me and another. left channel is me talking from a SIP ATA. right chan is a gsm phone. server uses asterisk 1.2.1 with ast_jb-1.2.0.patch3, forcing jitterbuffer

roy

By: deti (deti) 2005-12-08 07:01:13.000-0600

Well despite other experiences now it works properly for me. JBs are created only between sip and zap channels. Great work, thanks!

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-08 07:21:23.000-0600

can someone else please upload a monitor()ed conversation on a bad internet connection?
i keep getting these really bad noise even without the jb
wierd

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-09 05:11:52.000-0600

same error seems to occur with this ATA and 1.2 even without the jb, so please ignore this for the jb case

By: deti (deti) 2005-12-09 05:45:02.000-0600

patch used 24h on a production system and nothing bad happened!

By: Rosario Pingaro (rpingar) 2005-12-09 06:30:51.000-0600

for deti:
did you measured any real improvment using jb?

Thanks
Rosario

By: deti (deti) 2005-12-09 06:45:23.000-0600

yes for sure. Our DSL-Teleworkers sound quite better (no more dropouts) when talking to a PSTN number. This is what I expect from a jitterbuffer. Maybe the latency could have increased a bit. What else should I have noticed?

By: Martin Vit (festr) 2005-12-09 07:16:45.000-0600

there is also some patch (asynchronous generation) <http://bugs.digium.com/view.php?id=5374>. This jitterbuffer patch doesnt solve this am'i right?

Next question to this jitter impl: is it interpolating dropped frames? Is this jitterbuffer based on new iax jitter?

By: slav (slav) 2005-12-09 08:45:00.000-0600

"there is also some patch (asynchronous generation) <http://bugs.digium.com/view.php?id=5374>. [^] This jitterbuffer patch doesnt solve this am'i right?"

This patch modifies asterisk to send frames generally every 20 ms (this of course depengs on codec ...) without any dependency on what and when is received. Of cource if there is no frame received to play now, it will request from the codec to interpolate one and if there is nothing received for a long time, only silence will be sent (interpolated from codec).

"Next question to this jitter impl: is it interpolating dropped frames? Is this jitterbuffer based on new iax jitter?"

Yes, it is somehow based on the new iax jb - it requests interpolation from the coded in the same way as iax jb does. Of coure, like with iax, real interpolation happens only during transcoding. However in the SIP->ZAP case you always have transcoding.
Aside from this - two different jb algorithm implementations can be used with this patch. The first one is a "fixed" jb algorithm, provided with the patch. The second one, called "adaptive", is actually the jb algorithm used for the new iax2 jb. Additional implementations can be added later... You can swith them from sip.conf or zapata.conf and choose what's better for you.

By: Martin Vit (festr) 2005-12-09 08:58:14.000-0600

thank you for very complete answer. Next question :-)

Consider this situation --SIP phone-> {asterisk1} -IAX-> {asterisk2->ZAP}

when SIP phone generates jitter and both SIP and IAX is the same codec, will the jitter pass through IAX to the asterisk2, so dejjitering is on asterisk2? (to zap channel). If no, what should be enabled on asterisk SIP->IAX?

By: slav (slav) 2005-12-09 09:32:50.000-0600

Yes, if iax is not configured to do force dejittering, you won't have any dejittering from sip to iax. And you don't need! Moreover, when both channels are using the same codec, the dejittering is not so effective, because of the inability to interpolate frames (its rather reordering than dejittering in this case). IMHO ZAP on asterisk2 is the right place for a jb here.

By: Martin Vit (festr) 2005-12-09 09:51:39.000-0600

so for fully understanding SIP->IAX jb: When SIP phone sends frames every 20ms and there is 100ms wait, iax channel does not send any packet? After that 100ms every packets arraived from the sip phone and then will go to IAX, there must be timestamps from SIP not from IAX?

By: slav (slav) 2005-12-09 10:04:21.000-0600

No, we don't have asterisk1 SIP timestamps in asterisk2 IAX. On asterisk2, IAX will provide jb (the new iax2 jb), not SIP. Maybe I wasn't clear enough: when I say "ZAP on asterisk2 is the right place for a jb" I mean (IAX-jb)->ZAP. It's a little bit confising to have two independent jb's :)

By: Martin Vit (festr) 2005-12-09 10:19:57.000-0600

in my example consider jittery SIP (wireless) but stable IAX (ethernet). So if i have asterisk1 which is doing SIP connections and ast1 is connected to ast2 via IAX (operator for example) how to perform dejjitering?

If i understand that iax have no idea about sip timestamps (it will not propagate from sip to iax), there is no way (yet) dejjiter SIP connection (on ast2 IAX->zap) which comes from ast1 but over iax to ast2. (sorry for my english, i hope that now is it clear what i'm thinking :-)

By: zoa (zoa) 2005-12-09 11:23:31.000-0600

i think the best reason to implement the generic jitterbuffer for iax2 (and remove the current one) would be this situation:

[jittery iax2 link] -> * -> [not so jittery sip link] -> * [zaptel]

if the iax2 would provide timestamp information, then dejittering will only happen on right before going to zaptel, but dejittering will happen for both the jitter on the iax link and on the sip link, as the timestamps will be reused.

(and the same in the other direction).

As 2 times jitter could kind of auto compensate, 2 jitter buffers would be overkill and will cause a larger delay.

By: alvi (alvi) 2005-12-16 01:55:30.000-0600

I am probably missing something, but I can't make the patch working. I patched Asterisk 1.2.1 sucessfully, enabled the compilation flag in Makefile and compiled succesfully. I edited sip.config to force jitter buffer and turned on the debug. Anyway I can't see any log file in /tmp, nor there are any warning message like "cannot create log file".
Is there any check I can do to see if the Jitter Buffer is actually working?
Thanks.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-18 07:59:07.000-0600

Crash:

(gdb) bt
#0  0x401491d9 in _int_free () from /lib/tls/libc.so.6
#1  0x401495fb in free () from /lib/tls/libc.so.6
#2  0x08064a1d in ast_channel_free (chan=0x401f9580) at channel.c:954
#3  0x0806a2f8 in ast_hangup (chan=0x42f03740) at channel.c:1346
#4  0x080950f7 in __ast_pbx_run (c=0x42f03740) at pbx.c:2457
ASTERISK-1  0x080967dc in pbx_thread (data=0x87d08f0) at pbx.c:2507
ASTERISK-2  0x400329dd in start_thread () from /lib/tls/libpthread.so.0
ASTERISK-3  0x4019effa in clone () from /lib/tls/libc.so.6
(gdb) bt full
#0  0x401491d9 in _int_free () from /lib/tls/libc.so.6
No symbol table info available.
#1  0x401495fb in free () from /lib/tls/libc.so.6
No symbol table info available.
#2  0x08064a1d in ast_channel_free (chan=0x401f9580) at channel.c:954
       cur = Variable "cur" is not available.
(gdb)

By: slav (slav) 2005-12-19 02:21:06.000-0600

This looks like you didn't enabled the jb in Makefile - in channel.c, line 954 there is free() in case you don't have AST_JB defined. Are you sure the jb was enabled (there's an obvious evidence of this - a green line in the cli which says "fixed jitterbuffer created on channel ..." when a call is answered)? Moreover - please disable any compiler optimizations (-O0) when generating coredump, otherwise gdb often get confused and reports wrong line numbers and param values. Please, provide us also with the exact asterisk and patch versions.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-19 04:04:34.000-0600

strange

   -- Executing AGI("SIP/1990035-8010", "nyte.agi|dial") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/nyte.agi
   -- AGI Script Executing Application: (Dial) Options: (IAX2/sipgw1:password@pstngw2/98013356)
   -- Called sipgw1:password@pstngw2/98013356
   -- Call accepted by 213.160.242.21 (format alaw)
   -- Format for call is alaw
   -- IAX2/pstngw2-3 is ringing
   -- IAX2/pstngw2-3 stopped sounds
   -- IAX2/pstngw2-3 answered SIP/1990035-8010
   -- Hungup 'IAX2/pstngw2-3'
   -- AGI Script nyte.agi completed, returning 0

and the JB is enabled in the makefile:

gcc  -pipe  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3  -Iinclude -I../include -D_REENTRANT -D_GNU_SOURCE  -O6 -march=i686 -DZAPTEL_OPTIMIZATIONS   -DAST_JB       -fomit-frame-pointer    -c -o channel.o channel.c

and it's enabled in sip.conf

jb-enable = yes                 ; Enables the use of a jitterbuffer on the receiving side of a
jb-force = yes                  ; Forces the use of a jitterbuffer on the receive side of a SIP

...and I did a 'make valgrind' to stop it optimising, but this seems not to do the job anymore :( 'valgrind' is a pointer to 'dont-optimize' which simply seems to do an 'install'

but then again. i do not get a 'green line'

roy

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-19 04:39:58.000-0600

I just tried on another server with a sangoma A104 card
latest svn 1.2
patch 3 from here
AST_JB uncommented in Makefile
jb-enable = yes
jb-force = yes

and NO JITTERBUFFER OUTPUT:

   -- B-channel 0/31 successfully restarted on span 1
   -- Executing GotoIf("SIP/1990035-26d9", " 0?nocallerid:havecallerid") in new stack
   -- Goto (default,98013356,4)
   -- Executing Set("SIP/1990035-26d9", "TAG=unknown") in new stack
Dec 19 11:38:10 NOTICE[27198]: pbx.c:1478 pbx_substitute_variables_helper_full: Error in extension logic (missing '}')
   -- Executing Set("SIP/1990035-26d9", "FILENAME=unknown-98013356-") in new stack
   -- SIP Seeding peer from astdb: '1990035' at 1990035@80.202.161.55:58546 for 1800
   -- Executing Monitor("SIP/1990035-26d9", "wav:unknown-98013356-") in new stack
   -- Executing Dial("SIP/1990035-26d9", "Zap/g1/98013356") in new stack
   -- Requested transfer capability: 0x00 - SPEECH
   -- Called g1/98013356
   -- Zap/1-1 is proceeding passing it to SIP/1990035-26d9
 == Parsing '/etc/asterisk/manager.conf': Found
 == Manager 'nagios' logged on from 213.160.242.10
 == Manager 'nagios' logged off from 213.160.242.10
   -- Zap/1-1 is ringing
   -- Zap/1-1 answered SIP/1990035-26d9
   -- Registered SIP '21973550' at 80.239.107.100 port 5060 expires 120
   -- Saved useragent "Asterisk PBX" for peer 21973550
   -- Channel 0/1, span 1 got hangup request
   -- Hungup 'Zap/1-1'

By: slav (slav) 2005-12-19 04:55:59.000-0600

Enabling the jb in the config and Makefile doesn't means a jb will be used. This means - create jb if possible, which is the case only in SIP<->ZAP and SIP<->SIP calls currently. In your case SIP<->IAX chan_iax2 should be modified to enable jb (like zap does) if you want to have dejittering on the receiving iax side. Also chan_iax2 can be modified to add timing information in the generic ast frames if you want to have dejittering on the receiving sip side. Currently there's no way to have generic ast_jb in this case at all.

About the optimization level: find "OPTIMIZE+=-O6" in Makefile and replace it with "OPTIMIZE+=-O0".

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-19 04:59:26.000-0600

The above test was SIP<->Zap and nothing happened....

By: slav (slav) 2005-12-19 05:00:27.000-0600

sorry, didn't saw your last post - did you set jb-enable = yes in zapata.conf too? Because the jb should be actually enabled on the ZAP channel, not on SIP.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-19 05:05:52.000-0600

That did it
How can I make the jitterbuffer work in the current setup with SIP<->IAX?
Will this jitterbuffer help with audio quality on apps, such as app_queue?

it seems strange that the jitterbuffer should be in the _sending_ channel (zap) and not in the _receiving_ channel (sip)

roy

By: slav (slav) 2005-12-19 05:17:42.000-0600

Well, I mean jb is applied when frames are sent to the PSTN, e.g. when an analogue phone is receiving sound. In this case it seems logically to me to say receiving side, because a frame from SIP is going to ZAP, where it enters a jb.

By: Roy Sigurd Karlsbakk (rkarlsba) 2005-12-19 05:20:21.000-0600

ok
zoa said you were working on something with sip<->iax2. is this so? we need this urgently for testing.....

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-01-05 09:19:25.000-0600

we've done some testing now, and thus far, the sounds is good, very good.

someone please remove that badsound.ogg file. it's got nothing to do with this :P

roy

By: Martin Vit (festr) 2006-01-05 15:53:57.000-0600

hello, any progress on sip->iax2?

By: zoa (zoa) 2006-01-06 09:17:45.000-0600

The goal is to have no jitter buffer on SIP to IAX2, but if you ask if we made changes for it to have passthrough of the timestamps, no not yet.

We'd prefer to first get the sip to zap part in there.

By: anest (anest) 2006-01-06 10:17:32.000-0600

all world look on you, guys, with hope...

By: zoa (zoa) 2006-01-06 10:23:31.000-0600

try it try it, it should work just fine, no known issues with this version

By: zoa (zoa) 2006-01-11 05:55:09.000-0600

please test this guys, there are so many people monitoring this issue and so little reporting on it.

The only file you need to apply is:

ast_jb-1.2.0.patch3

Zoa

By: deti (deti) 2006-01-11 06:04:55.000-0600

We now have ast_jb-1.2.0.patch3 in production use for more than 4 weeks and still it works fine without any problems. I really would appreciate if this patch could be included in the head release.

By: Andrew Kohlsmith (akohlsmith) 2006-01-11 14:37:41.000-0600

There is a lot of conflicting information in the conversation thread, so let me ask clearly:

Will this dejitter SIP *and* IAX2 calls?  I am not concerned about dejittering voip-voip calls such as SIP-SIP, SIP-IAX2 and IAX2-IAX2, only SIP-Zap and IAX2-Zap.

By: zoa (zoa) 2006-01-11 15:34:27.000-0600

Currently, no. The patch is sip only.

But it should be very trivial to add this to iax2.

I propose to first get this sip part tested / added, and then to focus on all other channel types.

By: abramoff (abramoff) 2006-01-12 02:00:55.000-0600

This patch broke g729 licenses count.

I purchased one g729 license from Digium. Then I tried to setup simple call:

[analog phone]--->[PSTN]--->[Zap fxo,*,SIP/g729]--->[VoIP provider]

I got one way audio and g729 codec flood console with messages:

WARNING[16357] codec_g729.c: Out of G.729 Encoder Licenses!

"show g729" give me:

1/0 encoders/decoders of 1 licensed channels are currently in use

and count of "encoders" don't decrease after I terminate call. Without patch, licenses counting work right.

By: anest (anest) 2006-01-12 07:53:13.000-0600

muhaha! i think you need buy more licenses if you want real TWO WAY sound ;)
read explonations why on digium site (or probably in README with codec, im not remember)
sorry for flame, but i think you dont know about "free" release of 723/729 codecs from intel ;) keep googling... and good lack

By: abramoff (abramoff) 2006-01-12 08:22:16.000-0600

2 anest:

Again: without patch, licenses counting work right - I get 2-way audio. This is jitterbuffer patch issue, not Digium codec.

Yes, I know about "free" G.729 binaries, but they free _only_ for "Study or experiment" - not for _production_!

By: anest (anest) 2006-01-12 09:24:34.000-0600

anyway, buy one more license and see what will happen ;)

By: pj (pj) 2006-01-12 15:31:29.000-0600

patch3 successfully applied agains 1.2.0, edited Makefile (enabled AST_JB, disable optimalization, disable crypto), sip_jb enabled & forced in sip.conf, but NO messages about JB when asterisk starts (green info message), no debug log :-(
(gcc 4.0.2)

By: slav (slav) 2006-01-13 03:36:13.000-0600

You'll see a green line when a call is answered. For the debug log set jb-log=yes in the conf. Also you'll need to enable the jb in zapata.conf if you want dejittering on a SIP->ZAP call.

By: pj (pj) 2006-01-13 04:43:57.000-0600

yes, seems that's working well for now, before I still looked at asterisk startup for JB messages;-)
Is JB usefull for SIP-OH323 calls, i.e. in all cases when asterisk doing media stream termination, or in pure sip-sip or sip-zap calls?
when I use asterisk's echo test application, seems JB isn't activated, is this correct?
PJ

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-01-17 07:25:01.000-0600

i just tried to setup one box with 1.2.svn with your jitterbuffer patch. the box is currently doing zap/iax bridging but i can see no jitterbuffer messages. jitterbuffer is enabled in iax.conf and zapata.conf and compiled in (AST_JB set in makefile)

roy

By: pj (pj) 2006-01-17 07:54:44.000-0600

I thing, you can apply this patch only to 1.2.0 asterisk release to patch successfuly.Also, seems that jitterbuffer can currently help only for sip-sip and sip-zap calls...
PJ

By: Serge Vecher (serge-v) 2006-01-17 07:57:03.000-0600

rkarlsba: please read zoa's comment on 01-11-06 15:34. Also, bug's title provides a hint. If your box did sip/zap bridging, then you would see the effects of this patch.

By: anest (anest) 2006-01-18 06:59:50.000-0600

what about 1.2.2 ? patch will work with-out any changes?

By: deti (deti) 2006-01-26 02:33:27.000-0600

FYI: Added patch for subversion head release - just in case anyone needs it...

By: Olle Johansson (oej) 2006-01-26 03:21:06.000-0600

All new features are *only* applied to svn trunk, so we do need a patch for the current svn trunk, yes.



By: deti (deti) 2006-01-26 03:46:12.000-0600

My patch only contains a diff against the files in the subversion repository (M). The extra files (?) have to be taken from the old ast_jb-1.2.0.patch3. Sorry for this inconvenience...

?      scx_jitterbuf.c
?      scx_jitterbuf.h
?      abstract_jb.c
M      channel.c
M      channels/chan_zap.c
M      channels/chan_sip.c
M      channels/chan_iax2.c
M      Makefile
M      translate.c
?      include/asterisk/abstract_jb.h
M      include/asterisk/channel.h
M      include/asterisk/frame.h
M      rtp.c
M      frame.c
M      configs/sip.conf.sample
M      configs/zapata.conf.sample

By: Olle Johansson (oej) 2006-01-26 06:10:35.000-0600

deti: If you changed any code, I need a disclaimer assurance for you as well. I would like one complete patch so I can create a branch for people to test.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-01-26 06:28:17.000-0600

uploaded patch for rev 8705
and, yes, i have disclaimed the code

By: Olle Johansson (oej) 2006-01-26 07:14:48.000-0600

Opening a new branch for this:

svn checkout http://svn.digium.com/svn/asterisk/team/oej/jitterbuffer

Patches to that branch for changes are happily accepted. Just make a note whether it's a patch towards this development branch or something else to make it more easy to maintain.

Please test this new code and report results here, to make a decision about including this in Asterisk or not a bit easier. We do need to test this thoroughly to give it a good start. If we want to get anything into Asterisk 1.4, we are in a hurry.

By: Olle Johansson (oej) 2006-01-26 07:17:44.000-0600

Royk: That patch was not clean. You have changes to other files in there. (app_morsecode)

By: damin (damin) 2006-02-02 12:33:24.000-0600

Hello all.. Now that Olle is maintaining a seperate branch for this in SVN, I thought I'd throw it on a test box and see what I can break.

I'm running 1.2 Zaptel and Libpri from SVN and when I attempt to build the jitterbuffer branch I get the following error:

gcc -c  -pipe  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3  -Iinclude -I../include -D_REENTRANT -D_GNU_SOURCE  -O6 -march=i586 -DZAPTEL_OPTIMIZATIONS          -fomit-frame-pointer  -DT38_SUPPORT -Wno-missing-prototypes -Wno-missing-declarations -DZAPATA_PRI -DIAX_TRUNKING -DCRYPTO -fPIC  -o chan_zap.o chan_zap.c
chan_zap.c:66:2: #error "You need newer libpri"
chan_zap.c: In function `zap_send_keypad_facility_exec':
chan_zap.c:2281: warning: implicit declaration of function `pri_keypad_facility'

Should I be running trunk for this? Comments?

By: Olle Johansson (oej) 2006-02-02 12:38:04.000-0600

I keep updating the branch so it's compatible with current trunk, so the zaptel that works with current asterisk trunk should work with this, unless slav has dependencies to something else.

Slav?

By: slav (slav) 2006-02-06 10:37:00.000-0600

"That patch was not clean. You have changes to other files in there. (app_morsecode)"

oej, are you sure the jitterbuffer branch contains properly modified files? Please make sure all files, modified and added by ast_jb-1.2.0.patch3 are present.

"I keep updating the branch so it's compatible with current trunk, so the zaptel that works with current asterisk trunk should work with this, unless slav has dependencies to something else."

No. Unless the merging doesn't broke something, because the patch contains modifications in chan_zap too.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-08 09:51:27.000-0600

try this patch for trunk rev 9215

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-20 04:04:38.000-0600

for convenience, try 2006-02-20-svn-1.2-rev10558.patch for the latest svn 1.2 branch

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-20 04:20:25.000-0600

for latest trunk as well

By: Olle Johansson (oej) 2006-02-20 08:44:26.000-0600

Roy, Please don't post patches for trunk, post patches for the branch if needs a fix. The branch should be the most recent version of this all the time, so people test that. Thanks.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-20 10:08:05.000-0600

Olle:
As you said "All new features are *only* applied to svn trunk, so we do need a patch for the current svn trunk, yes."
I did post a patch for the trunk, but you said it wasn't clean, so I thought you'd like another.
Sorry if this was wrong

roy

By: Olle Johansson (oej) 2006-02-20 10:30:59.000-0600

Look back in history, that was *before* I created the branch. Have you downloaded the branch and worked with it? I need assurance that the branch works.

By: anest (anest) 2006-02-21 17:05:52.000-0600

hey, guys, can you make "one-file" patch for 1.2.4 version? (not for svn)
i think many, many people will tested and report back. (i waiting long time for this too)

By: Olle Johansson (oej) 2006-02-22 08:51:15.000-0600

The "jitterbuffer" branch is kept up to date with trunk.

This patch now also included in "test-this-branch" for easy testing.

Thank you for testing all of this, providing feedback, suggestions etc.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-22 08:55:13.000-0600

anest: I beleive the  2006-02-20-svn-1.2-rev10558.patch  patch will apply quite cleanly to 1.2.4

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-22 09:48:53.000-0600

attached patch fixes typo/conflict problems with the svn version

By: anest (anest) 2006-02-22 15:03:57.000-0600

rkarlsba: thanx for answer. but i have one more question - i need only one patch (i see many files here)?

By: Matt O'Gorman (mogorman) 2006-02-22 15:07:23.000-0600

perhaps the most recent one....  02-22-06 09:48    rkarlsba    File Added: oej-jb.patch

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-23 03:07:41.000-0600

2006-02-20-svn-1.2-rev10558.patch will do alone

oej-jb.patch is just a small patch to fix some typos in code from

svn checkout http://svn.digium.com/svn/asterisk/team/oej/jitterbuffer

roy

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-23 03:58:09.000-0600

reproducable with 1.2-rev10761
I didn't manage to reproduce it with trunk, but we only ran that for a short time since there's an error with meetme (it doesn't read the configuration correctly)

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-23 03:59:48.000-0600

return -EWRONGBUG;

please remove last comment "reproducable with 1.2-rev10761..."

roy

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-27 05:30:04.000-0600

there seem to be audio problems when doing SIP/SIP calls with dejittering. The caller (from SIP) can be heard, but he can not hear the one he's calling after a while. the audio simply drops out - see 62796252-98013356-2006-02-27_13-07-28.ogg (viewable/playable with audacity or similar). when turning off the jitterbuffer in sip.conf, audio is fine.

this testing has been done with 1.2.4 and the original patch.

when dialing, asterisk reports jitterbuffer has been created, twice per call:

   -- ***[JB LOG]*** fixed jitterbuffer created on channel SIP/pstngw2-8d6b
   -- ***[JB LOG]*** fixed jitterbuffer created on channel SIP/1001499-51b6

Testing is done with SIP ATA -> * 1.2 with JB -> * 1.0 -> PSTN

roy

By: slav (slav) 2006-02-27 07:23:01.000-0600

This ogg file is not very useful to me. Please set jb-log = yes in sip.conf and upload the generated log files in /tmp both for the caller and the callee.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-28 03:42:49.000-0600

These two were created with one call
Do you need anything more? Asterisk debug logs or so?

By: edo (edo) 2006-02-28 04:23:21.000-0600

2oej: is svn branch working?
i can't compile it:
chan_zap.c:113: warning: `default_jbconf' defined but not used
chan_zap.c:120: warning: `global_jbconf' defined but not used
chan_zap.c:123:1: unterminated #ifndef

ps:  revision 11332

By: slav (slav) 2006-02-28 04:42:32.000-0600

rkarlsba, no I don't need nothing more. 10x for the files - I saw where the problem is. Its normal to have two jbs in SIP->SIP call when jb-force=yes in sip.conf - you just have one jb for each direction. The actual problem in this case is in the core dejittering algorithm and I'll try to fix it ASAP. You wrote: "this testing has been done with 1.2.4 and the original patch". Is this "original patch" the ast_jb-1.2.0.patch3 patch from 12-08-05 04:47? I prefer to update this patch first rather than trunk (seems to be totally broken). At some future point I'll make a new patch for trunk, but I have to make sure ast_jb-1.2.0.patch4 will fix this kind of dropout problems first. Also I plan to make other changes and improvements, but I'll need some extra time for this...

Slav

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-28 04:51:32.000-0600

the testing was done with the 12-08-05 patch, yes, with a few changes to make it compile on 1.2.4  (minor conflicts and a renamed variable)

By: slav (slav) 2006-02-28 06:54:30.000-0600

Uploaded new version for 1.2.0 (ast_jb-1.2.0.patch4). Roy, please test the new version with the same conditions that the jb logs were generated in. Did the dropout problem is reproducible each time with the old version and did the new one fixes this?

Slav

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-28 08:33:10.000-0600

This works far better
With lots of downloads on a 704/128 ADSL link, I get crystal sound, but 2 secs latency, but that's ok

By: Olle Johansson (oej) 2006-02-28 08:34:11.000-0600

Slav, I should give you -2 karma for patching something else than the branch... Please upload a fixed patch for the jitterbuffer branch :-)

Version 1.2 is *not* the base for any development, even though it can be friendly to maintain patches for it. Please focus on the jitterbuffer branch for now.

Thanks
/Olle

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-28 08:41:03.000-0600

oej: Please understand that this patch was ordered and paid for by me to be written for 1.2. Trunk is by no means stable enough to be put into production. Therefore, this patch is written for 1.2.

By: Olle Johansson (oej) 2006-02-28 08:57:58.000-0600

I understand that Roy, but we still need to use the bug tracker for development and always produce patches for trunk as well. Otherwise this will not make it into asterisk and you don't want that to happen, right?

By: anest (anest) 2006-02-28 09:05:04.000-0600

thank you guys for your hard work

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-28 09:22:11.000-0600

AFAICS diff-new-jb-2006-02-28.patch contains slav's fix

By: Olle Johansson (oej) 2006-02-28 09:45:27.000-0600

RoyK: Thanks.

Updated the branches "jitterbuffer" and "test-this-branch" with your patch. Keep testing the branch, please!

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-28 10:13:15.000-0600

hi

it seems that with fixed jitterbuffer the latency now is equal to the given 'jb-max-size'. is this intended? I've also done some testing with the adaptive jitterbuffer, is this stable yet? i got some noise which sounded quite like jitter when using the adaptive variant. log files are attached in tar ball.

roy

By: slav (slav) 2006-02-28 10:29:18.000-0600

10x for updating the branch Roy! I didn't have the time to do this right now.
Tomorrow will checkout the branch and will fix something if wrong.

"it seems that with fixed jitterbuffer the latency now is equal to the given 'jb-max-size'. is this intended?"

yes, the delay the fixed jb is adding is just jb-max-size. 2000 ms too much for a jb. Usually 200 is good enough, but on really bad connections you can use 400. Don't let jb-resynch-threshold to be smaller than jb-max-size - it can result in undefined behaviour.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-02-28 10:32:54.000-0600

thanks

but what about the adaptive one? is this stable? should it be discussed here at all?

roy

By: zoa (zoa) 2006-02-28 10:37:22.000-0600

the adaptive jitterbuffer is 100% steve kann's jitterbuffer, we are not trying to change or improve this one.

Its the same that will be used with iax2, and i personally think it still has some little issues sometimes. (i've seen an ever increasing delay for example).

The ticks are normal when the jitter buffer is shrinking i think.
I also think a bug was fixed in the jitter buffer last week, i dont know if we need to upgrade to this newer version or its done automatically.

By: edo (edo) 2006-02-28 13:36:20.000-0600

tiny patch to chan_zap.c from svn

By: slav (slav) 2006-03-01 09:39:02.000-0600

The brach team/oej/jitterbuffer/ is broken. It doesn't compile nor works correctly. Uploaded oej-jb.2006.03.01.patch, generated against team/oej/jitterbuffer/ from 03-01-06 which fixes the problem. It also includes the preceding chan_zap.patch (from 02-28-06). It seems that the original patch was not merged correctly - one piece of jb code in chan_sip appears in a similar code pattern but in a completly different function!

Slav

By: edo (edo) 2006-03-01 15:54:34.000-0600

other inconsistency in team/oej/jitterbuffer/ - abstract_jb.h must be moved to include/asterisk.

someone besides oej does use svn branch?

By: Rosario Pingaro (rpingar) 2006-03-01 16:03:35.000-0600

I tried to patch the 1.2.4 releas aginst ast_jb-1.2.0.patch4 but it is not patching correctly in fact channel.c has some little modification in variable toms->to and other little change.

Could some one be so kind to let me test jitter baffer on a production box?
I tried but was not able to manually fix the differencies.

Regards
Rosario

By: Rosario Pingaro (rpingar) 2006-03-01 16:10:04.000-0600

I tried to patch the 1.2.4 releas aginst ast_jb-1.2.0.patch4 but it is not patching correctly in fact channel.c has some little modification in variable toms->to and other little change.

Could some one be so kind to let me test jitter baffer on a production box?
I tried but was not able to manually fix the differencies.

Regards
Rosario

By: Rosario Pingaro (rpingar) 2006-03-01 16:10:20.000-0600

I tried to patch the 1.2.4 releas aginst ast_jb-1.2.0.patch4 but it is not patching correctly in fact channel.c has some little modification in variable toms->to and other little change.

Could some one be so kind to let me test jitter baffer on a production box?
I tried but was not able to manually fix the differencies.

Regards
Rosario

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-05 15:44:43.000-0600

I don't know if this is relevant to this bug, but I found a memleak with asterisk patched with this: http://bugs.digium.com/view.php?id=6652

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-06 05:54:08.000-0600

I am afraid this might be the cause for bug ASTERISK-6476
I don't know for sure yet, but after four hours, that leak is not visible, after commenting out AST_JB in the makefile

By: Olle Johansson (oej) 2006-03-06 08:27:11.000-0600

Closed ASTERISK-6476 - let's discuss the memory leak in this issue report.

By: slav (slav) 2006-03-06 09:14:41.000-0600

Roy, is this memory leak present with asterisk/tags/1.2.0 pathced with the original patch (i.e. ast_jb-1.2.0.patch4)? Or it leaks only in the 1.2 branch / oej's jitterbuffer branch?

As I mentioned before, the original patch does not apply correclty to the oej branch and possibly to 1.2.4 and other versions from svn/1.2 branch/trunk. As a result, one piece of code appears in similar but different function and looking at the code - this definitly will result in memory corruption (take a look at the build_peer() function where AST_JB is defined - there shouldn't be AST_JB in this function at all!). The uploaded from me oej-jb.2006.03.01.patch fixes this problem, but as I saw from the today svn commits, someone already applied the preceding chan_zap.patch (instead of the next one, which also addresses the chan_zap compile problem), so now the oej-jb.2006.03.01.patch won't apply cleanly.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-06 09:19:25.000-0600

It leaks with 1.2.5 which I am using now. I am not using trunk. Also, as far as I can see, the only hichups with 1.2.5 as compared to 1.2.0 are absolute minor

roy

By: Olle Johansson (oej) 2006-03-06 09:36:22.000-0600

Slav: Added your fixes to the jitterbuffer branch now. Please check. Thanks.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-06 10:10:28.000-0600

slav,

how can i do more testing to isolate this?

By: slav (slav) 2006-03-07 08:36:29.000-0600

ole: Its OK now

rkarlsba: Use valgrind. Its very heplful in detecting memory leak sources.

Tests I made before with 1.2.0 and ast_jb-1.2.0.patch... didn't show any memory leaks for 48 hours and more. I tested with about 120 SIP->SIP and SIP->IAX concurrent calls. I don't have the possibility to test with many SIP->ZAP calls now.

Can you confirm that its leaking with asterisk 1.2.0 and ast_jb-1.2.0.patch4 (or ast_jb-1.2.0.patch2/3)?

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-07 08:47:05.000-0600

How can I use valgrind on this system? the leak does not seem to be related to concurrent calls, though. but to call volume (started/stopped calls). the leak is only visible with 1.2.x with the patch, so is it really necessary to test it without it? earlier asterisk versions (pre 1.2.4) has known leaks as well

By: slav (slav) 2006-03-07 09:06:05.000-0600

valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --error-limit=no asterisk -vvvcf

"the leak is only visible with 1.2.x with the patch, so is it really necessary to test it without it?"

I don't get something: Is this means 1.2.0 patched with 1.2.0.patch4 doesn't leak? This will be an importent clue for me.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-07 09:16:59.000-0600

1.2.4/1.2.5 does not leak a bit without the patch

i will try to run it through valgrind

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-07 09:20:51.000-0600

what is this?
--num-callers=20
does that limit the number of loggable threads or something?

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-07 09:22:48.000-0600

sorry
i cannot run asterisk from within valgrind
cpu load goes BOOM

By: slav (slav) 2006-03-07 09:56:00.000-0600

--num-callers=20 means show backtrace with max depth 20. Default is something like 5, which offten is not enough. You cannot expect to run many concurrent calls under valgrind - it slows down 25 times or something... Try with fewer concurrent calls, but more frequent start/stop

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-07 10:01:04.000-0600

Please talk to me on IRC, freenode, RoyK or roy@karlsbakk.net on MSN Messenger. I need to discuss a few topics in private

By: Gilberto Mautner (gmautner) 2006-03-07 14:51:54.000-0600

Hello... I've been following this thread and I'm kind of lost among the multitude of patches.

I have a 1.2.5 installation and I'd like to know for short which patch(es) shoud I apply to have the latest jitterbuffer implementation working?

I'm aware that of the trunk and oej branches but for other reasons it's important to us that at the same time the rest of the code is pure 1.2.5 except for the jitterbuffer itself.

By: mike240se (mike240se) 2006-03-08 02:34:21.000-0600

Ok, after some minor modifications to channel.c patching (variable issue) i got ast_jb-1.2.0.patch4 to apply to 1.2.5, i have it running now in a heavy use enviroment which should be a good test.  On some SIP -> SIP test calls it performed quite well, i normally have problems with SIP to SIP calls via teliax my provider, this cleared alot of them up.  I had to enable force-jb to get it going at all, otherwise it wouldnt work on any sip - sip calls, what exactly makes the Jitterbuffer activate?  I couldnt get it to activate at all without the force-jb....  thanks for all the awesome work guys!

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-08 03:13:44.000-0600

Testing now shows 1.2.0 has been running smoothly for more than 16 hours without any indication for a leak, so there must be something between 1.2.0 and 1.2.5 that causes the error. I'll do a diff and see if I can find it.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-08 07:24:37.000-0600

with 1.2.0, there seems to be no 'channel leak', but there is still a significant memory leak somewhere. this may perhaps be the bug fixed in 1.2.4. i don't know

could you please try to write a patch for 1.2.5?

thanks

roy

By: mike240se (mike240se) 2006-03-08 13:06:12.000-0600

After 4 hours of continuous usage on 1.2.5 and after probably about 30 calls, this started showing up, in the CLI and logs every like 1/100 of a second, the logs got to be several megs quite fast.
abstract_jb.c: Recieved frame with invalid timing info: has_timing_info=0, len=0, ts=0

could be the modification i made to channel.c?

By: Rosario Pingaro (rpingar) 2006-03-08 15:16:33.000-0600

I made the modification on channel.c too and got the same messages. I think it happens because we didn't do a good job about channel.c

Hope some one could do e patch against 1.2.5

Thanks
Rosario

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-09 05:11:05.000-0600

Leak conclusion:

- 1.2.5 patched up with patch4 from feb 28 leaks sip channels and memory like a sieve
- 1.2.0 patched up with the same patch does not leak sip channels, but does indeed leak memory, the same sieve being used as the bucked :P

we had some 400/800 megs virt/physmem allocated in about nine hours with 1.2.0, it this should indeed be considered major

roy

By: Serge Vecher (serge-v) 2006-03-09 09:14:21.000-0600

Am I wrong in understanding that development focus of jitterbuffer is on trunk vs. 1.2? I know this feature was wanted by many 1.2 but didn't make it due to instability issues of jitterbuffer code. So, if people are already willing to run "development" jitterbuffer code on top of "stable" branch, why not just run the "jittebuffer" branch that Olle has specifically created for the purpose of ironing out the bugs out of jitterbuffer code?

Think about the benefits ... You run the jitterbuffer branch, help stabilize the code, and it makes it into 1.4 released just several months away. Or, let's all distract the developers with backporting issues so the code grows a beard and doesn't make it into another stable release.

Sorry for off-topic post not related to development and thank you for understanding.

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-09 09:17:05.000-0600

See above
This has been discussed before

The problem is that trunk is by no means suitable for running in production, and this code was ordered for 1.2 (back in the cvs head days).

roy

By: mike240se (mike240se) 2006-03-09 13:00:37.000-0600

First i want to say I am not one of the people who paid for this code, that being said, i am a little confused, because if someone paid for code and they want it for their production system, so they dont want to run a trunk ver or a branch based of trunk, because of stability, then wouldnt they have the right to ask for it to be patched and working on any version they want?  Please, this is not a flame, i am just wondering cause that would be what i assume, but i guess i assume wrong.  Now for the relevant note: the branch that olle made for jitter buffer, is that based of what version??  trunk or 1.2.x ?

By: Olle Johansson (oej) 2006-03-09 13:41:58.000-0600

This is the official asterisk bug tracker. If someone orders features for the release version from a developer that is fine, but there's no reason to use the bug tracker for it since that code will not be accepted for inclusion in Asterisk. If you want the code for a new feature to be included in Asterisk, then  you have to develop for svn trunk. The release version needs to be protected from new features.

We accept that you add additional patches for the release version, but we can not accept using the bug tracker for new features not ported at all to the development branch. This has been discussed with RoyK and we understand each other, so there's no reason to continue this discussion here. If needed, please use the asterisk-dev mailing list for discussions unrelated to this particular code.

The jitterbuffer branch contains the latest version of this code, adopted for svn trunk. Currently Roy have discovered some memory glitch and we are all anxiously waiting for a fix for svn trunk and I guess that Roy will need a patch for 1.2 as well.

By: mike240se (mike240se) 2006-03-09 14:28:00.000-0600

Ok thank you for the clarification, I now fully understand, it makes sense that the developers want this done the right way, which means not working on current 1.2, but of course us production users want it now, now, now :) its the nature of the beast.  

My question is, olle, what would you say is the most stable branch to run with this JB, is test-this-branch fairly bug free and does it not have the memory leak?  Also where is the test-this-branch reported on?  Does it have a wiki or a bug tracker id#, or is it discussed somewhere else, i might load it on a production system cause its been working ok for me for a few days.

By: zoa (zoa) 2006-03-09 14:37:12.000-0600

i just wanted to clarify a little why we internally work with version 1.2.0

We only make the development on this version to make sure that we are not hit newly introduced bugs. (or newly introduced situations).

We do a lot of testing and want to make sure that we didnt introduce some stability issue ourselves while writing the jitter buffer.

(We had some issues in the past where we hit some problem which was caused to an upgrade, this can take days of pointless debugging and rollbacks of our code to then finally realise this was something new).

We do occasionally upgrade to a newer version and port the jitterbuffer to this newer version.

Anyhow, we will make sure that in the end it works for the most recent development version too.

By: mike240se (mike240se) 2006-03-09 21:19:32.000-0600

I tried to compile test-this-branch to help contribute to development rather than trying to fix 1.2.5 but couldnt compile, i get:
I didnt know where to report this cause i dont know which patch is causing the problem.

chan_sip.c: In function `set_device_defaults':
chan_sip.c:12140: error: `peer' undeclared (first use in this function)
chan_sip.c:12140: error: (Each undeclared identifier is reported only once
chan_sip.c:12140: error: for each function it appears in.)
make[1]: *** [chan_sip.o] Error 1
make[1]: Leaving directory `/usr/src/asterisk-test-branch-12455/channels'
make: *** [subdirs] Error 1
asterisk5:/usr/src/asterisk-test-branch-12455#

By: Olle Johansson (oej) 2006-03-10 00:47:54.000-0600

That error was fixed over 24 hours ago in test-this-branch. Something is wrong with the SVN mirrors.

The test branch is a compilation of a lot of patches and branches. Every patch and branch have an open issue in the bug tracker. General problems with the testbranch is discussed on the asterisk-dev list. So the jitterbuffer code in the testbranch and the jitterbuffer branch is exactly the same.

By: slav (slav) 2006-03-10 04:12:03.000-0600

Already two days I'm testing the last jb version with 1.2.0 and the oej/jitterbuffer branch. Bulk SIP call generator - (*) - SIP - (another *), cpu load 90 - 95%. No sign of a leaking for over 12 hours. Maybe the problem is realtime...

By: Olle Johansson (oej) 2006-03-10 04:14:17.000-0600

royk: Which realtime driver are you using?

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-10 04:57:05.000-0600

mysql

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-12 11:27:44.000-0600

Are there works on writing a patch that works with 1.2.5? I need this urgently :P

By: Olle Johansson (oej) 2006-03-12 12:13:12.000-0600

On request, opened "oej/jitterbuffer-1.2" branch to keep the 1.2 backports up to date with svn head for 1.2. Please provide patches for this branch and the svn trunk-based branch simultaneously in the future. Thank you.

By: Olle Johansson (oej) 2006-03-12 13:05:21.000-0600

The update of jitterbuffer-1.2 to head caused problems in channel.c - anyone that can take a look at that? Just check it out and you will see the compile errors.

By: Olle Johansson (oej) 2006-03-12 14:13:53.000-0600

Ok, the jitterbuffer-1.2 branch now compiles again. Thanks RoyK.

By: mike240se (mike240se) 2006-03-12 17:43:07.000-0600

so jitterbuffer-1.2 is 1.2.5 based?  Does it have the channel.c fixed?  Does it have the memory leak?

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-13 05:51:18.000-0600

channel.c is fixed
I beleive the leak is still there

By: slav (slav) 2006-03-13 08:52:47.000-0600

oej/jitterbuffer branch (10.03.2006) took 320 MB memory for a 72 hours continous work under heavy load - 120 SIP-SIP concurrent call with jitterbuffer and transcoding, avg. 4 calls per sec. and 98-99% CPU load.
However, the 1.2.0 version with the original patch didn't show any abnormal memory consumption for 12+ hours. I'll test again this version for a longer time...

One question (oej): In the first versions of the jb patch, I copied each received frame using ast_frisolate(f) (to put a duplicate in the jb). The original frame follows its regular way (just as the jb is disabled) and gets freed from ast_generic_bridge(). Later, testing with AST_JB in chan_iax2, I realized ast_frisolate(f) doesn't duplicate correctly frames read from IAX (there was memory corruption) and I should use ast_frdup(f) instead. Currently I'm using

if(f->mallocd & AST_MALLOCD_HDR)
{
frr = ast_frdup(f);
}
else
{
frr = ast_frisolate(f);
}

I don't believe theres a real problem with this (if the frame data is leaking the leak will be much more than several megabytes per hour...), but is this correct? When I should use ast_frisolate() and when ast_frdup()?

By: Gilberto Mautner (gmautner) 2006-03-13 09:41:59.000-0600

Slav:

"oej/jitterbuffer branch (10.03.2006) took 320 MB memory for a 72 hours continous work under heavy load - 120 SIP-SIP concurrent call with jitterbuffer and transcoding, avg. 4 calls per sec. and 98-99% CPU load."

Just curious... what kind of CPU? Transcoding from/to what?

By: slav (slav) 2006-03-13 10:14:46.000-0600

Intel(R) Pentium(R) 4 CPU 2.40GHz
ulaw - gsm

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-17 06:25:13.000-0600

Memory leak seems to be fixed with http://bugs.digium.com/view.php?id=6732 after a few hours run.

Will return back after a few more days

By: Rosario Pingaro (rpingar) 2006-03-17 09:11:42.000-0600

so witch patch to apply to 1.2.5 to have the jitter buffer workin gfine on sip channel?

Thanks
Rosario

By: mike240se (mike240se) 2006-03-17 15:59:16.000-0600

I am using oej's jitterbuffer-1.2 and dont see a memory leak at this time.  Bu its only been a couple days.  The issue I am having is i get the error: abstract_jb.c: Recieved frame with invalid timing
info: has_timing_info=0, len=0, ts=0

whenever a non-sip channel is created, zap channel gives the error and so does iax.  Is there a conflict maybe? I didnt add anything to zapata.conf since i figured it would default to off and iax2 has jitter-buffer=yes but i figured it would use the iax2 new jitterbuffer instead?  maybe cause i have force jitterbuffer in sip.conf?  Shouldnt it detect that its a zap channel and not use it?

Also it would be nice if there were a netstats type output for sip jb, iax2 will tell you the loss, jitter, etc but there is no way to see what the JB is doing in sip....

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-18 02:26:39.000-0600

hi

I still see the leak. The MRTG logging had hung up, so I didn't see it at first :P

roy

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-18 02:37:56.000-0600

Also please note that the leak exists on all versions from 1.2.0 and forward. The leak does NOT seem to be very closely related to the amount of traffic. I will try to run standard 1.2.5 to see if I can reproduce it there

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-18 04:49:45.000-0600

The memory leak persists, but if I disable sip jitterbuffer in sip.conf and do a sip reload, LOTS of memory is released after an hour or so.
How can we debug this further?

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-03-20 08:34:28.000-0600

Please, slav, take a look at the last uploaded file. I called a SIP client and had no luck with the jitterbuffer. This is getting urgent, so please haste.

thanks

roy

By: Adam Moffett (adammoffett) 2006-03-20 09:44:28.000-0600

I also do not see the memory leak.  My calls through this system are all SIP->IAX, And whenever the jitterbuffer is used I see in the console:

Mar 17 10:57:09 WARNING[22866]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=-1, ts=166273064
Mar 17 10:57:09 WARNING[22866]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=-1, ts=166273064

hundreds of lines of this.  In the source it seems that I would get this message when len < 0.  I haven't yet found what condition gives me a length of -1.

By: John Laur (gork) 2006-03-22 21:39:35.000-0600

I am using the jitterbuffer-1.2 branch.

It does not seem that Asterisk uses a jitterbuffer when it is a SIP endpoint. I only see it creating a jitterbuffer when it is bridging the audio between two SIP endpoints (and it creates two, which is what you'd expect.) Is this the designed behavior, am I misconfigured, or is this a bug? I am having terrible jitter problems.

By: Rosario Pingaro (rpingar) 2006-03-23 00:53:31.000-0600

Please Roy may you do a working patch for 1.2.5 to test with?

We don't have a tdm line on our test box to test the jitter branch......

So to make a trade off between jitter goodness and memory leak I need to test it on a production like machine........

A lot of bugs corrected from 1.2.1 (the last one that patched fine) to 1.2.5 so maybe also the memory leak has been involontary fixed.

Regards
Rosario

By: Olle Johansson (oej) 2006-03-23 00:58:30.000-0600

rpingar: The jitterbuffer-1.2 branch is based on latest 1.2 - there's no need for a separate patch.

By: Rosario Pingaro (rpingar) 2006-03-23 01:05:18.000-0600

thanks, I was not aware about it.

very sorry about stupid post.

Ros

By: Rosario Pingaro (rpingar) 2006-03-23 01:35:30.000-0600

why hundreds of lines of this kind in my log (like others have posted) using jitter branch-1.2:

Mar 23 08:32:30 WARNING[21689]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=0, ts=0

Is this a wrong implementation of jitter buffere for latest 1.2 branch?

regards
Ros

By: Rosario Pingaro (rpingar) 2006-03-28 11:13:04.000-0600

any new step?

regards

By: zoa (zoa) 2006-03-28 11:58:19.000-0600

i just came back from a tiny snowboard trip, currently going through my emails. Will be back here a later.

By: zoa (zoa) 2006-03-29 10:16:27.000-0600

i just came back from a tiny snowboard trip, currently going through my emails. Will be back here a later.

By: Rosario Pingaro (rpingar) 2006-03-30 03:45:09.000-0600

hi Zoa,

thanks about the help. using jitterbuffer-1.2 branch as I mentioned before I get ons of log about:
WARNING[21689]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=0, ts=0

This couppled with no difference in audio quality leed me to think that the jb is not working fine.

Could you please check it and try to figure out what is going on?

Regards
Rosario

By: Hekuran (haks) 2006-04-16 10:48:23

Im trying to run CallinCard with jitterbuffer-1.2 but asterisk segfaults. the following is the backtrace:

#0  0x40d6c8a0 in callingcard_acct_start (cdr=0x38373635, cd=0x40c16ab0) at app_prepaid_call.c:491
#1  0x40d6eba9 in callingcard_exec (chan=0x81cb158, data=0x40c1b090) at app_prepaid_call.c:998
#2  0x08093ed4 in pbx_extension_helper (c=0x81cb158, con=<value optimized out>, context=0x81cb2a8 "default",
   exten=0x81cb39c "0037744387555", priority=202, label=0x0, callerid=0x815f0a8 "100", action=1) at pbx.c:553
#3  0x08095625 in __ast_pbx_run (c=0x81cb158) at pbx.c:2227
#4  0x080971ac in pbx_thread (data=0x0) at pbx.c:2514
ASTERISK-1  0x4002aaa7 in start_thread () from /lib/tls/libpthread.so.0
ASTERISK-2  0x4019cc2e in clone () from /lib/tls/libc.so.6


Note that I have been able to use CallinCard with asterisk versions 1.2.0 - 1.2.7.1 but not with jitterbuffer-1.2

By: Olle Johansson (oej) 2006-04-16 22:41:31

This bug now has so many people monitoring it, so that PHP will time out before sending all the mails. Your message will be saved, even if you get an error message. Do not publish many times, thanks.

By: Olle Johansson (oej) 2006-04-16 22:47:08

haks: We need to make sure we support a "clean" asterisk without add-on modules before we start figuring out errors in external modules.

By: Adam Moffett (adammoffett) 2006-04-17 09:23:04

Several of us have asked about hundreds of lines like this in our log files:
WARNING[21689]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=0, ts=0

I haven't yet seen any comment on it.  What does it mean?

By: Tony Mancill (tmancill) 2006-04-18 19:26:58

I'm experiencing the same problem reported by abramoff on 1-12-06, except on a systems with (24) g729a licenses from Digium.  With the jb patch applied, as soon as any call is received where g729a can be used as a codec, all 24 encoder licenses are consumed and not released until Asterisk is restarted.  I'm also getting one-way audio for the call as well.  As soon as it happens, here's how it looks on CLI, despite there only being one call (two channels) active.

localhost*CLI> show g729
24/0 encoders/decoders of 24 licensed channels are currently in use

The logs scroll the following until the restart:

2006-04-18 12:00:36 WARNING[19621] codec_g729.c: Out of G.729 Encoder Licenses!

---
Some additional information:

When using ast_jb-1.2.0.patch3, the problem existed regardless of which direction the transcoding needed to occur.  (I.e. g711 calls g729 or g729 calls g711 - both would run it out of licenses.)  

After upgrading to ast_jb-1.2.0.patch4, originating g711 and calling to g729 works correctly; the licenses are used during the call and then released.  Originating g729 suffers the same problem as before.



By: mtaht (mtaht) 2006-04-21 01:02:19

I have a pathological test case

test-this-branch on both sides
Turn on internal timing (ztdummy timer)
Turn on the new jitterbuffer
use speex to transcode the iax link (and turn on plc, vad and dtx). This sends a gratifyingly small number of small packets when compared to ulaw.

The audio is outstandingly good so longasyoutalkveryquickly. if you talk at a normal pace, with a small gap between words, the jitterbuffer goes to hell in a handbasket and the voice is very broken up. If you then pause for a long breath, the first word often sounds fine. Another way to fix it is to sing for a long moment then talkveryfastwhenitclearsup. With the the jitterbuffer disabled on both sides, a speex transcoding session sounds pretty good.

with the jitterbuffer enabled, also, dtmf is way messed up. (see: http://bugs.digium.com/view.php?id=6993 and others)

ON THE RECEIVING SIDE I see:

I pause speaking for a while

Apr 21 01:22:01 WARNING[15568]: chan_iax2.c:768 jb_warning_output: Resyncing the jb. last_delay -32, this delay 14655, threshold 1066, new offset -22605
Apr 21 01:22:03 WARNING[15570]: chan_iax2.c:768 jb_warning_output: Resyncing the jb. last_delay 10, this delay 1155, threshold 1066, new offset -23760
Apr 21 01:22:16 NOTICE[15588]: sched.c:286 ast_sched_del: Attempted to delete nonexistent schedule entry 183927!

The first word sounds ok, then I draw a breath...

boostjet*CLI> iax2 show channels

Channel               Peer             Username    ID (Lo/Rem)  Seq (Tx/Rx)  Lag      Jitter  JitBuf  Format
IAX2/itconv-taht-out  71.198.23.124    transconf   00019/00023  00106/00197  00040ms  1412ms  0200ms  speex

I hard capped the maximum jitter buffer at 200, but I think that there is something else convincingly wrong

On the sending side, I see:

Apr 21 05:32:29 WARNING[3362]: translate.c:157 framein: no samples for speextolin
Apr 21 05:32:29 WARNING[3362]: translate.c:190 framein: speextolin did not update samples 0

While this is a warning, not an "error", I think it's interacting with the jb badly here. Not sending a packet? not telling the jb to not expect a packet for a while, that it's OK to not get anything?

I know that not a lot of people care about speex, but it's on a rtp standards track finally, and it's the only thing out there that supposedly does vad and plc natively... and when it works, it sounds great...

By: zoa (zoa) 2006-04-21 01:10:10

did you enable it for iax2 ?

We did not implement anything for iax2 yet.
The statistics you show are from the iax2 jitter buffer (steve kann's), not ours.

Where did you configure what jitter buffer ?

zoa

By: mtaht (mtaht) 2006-04-21 03:20:54

Ah, I am confused, then, I thought that this had become a generic jitterbuffer in test-this-branch with AST_JB enabled.

Yes, it's enabled in both sip and iax, but I guess I'm using two different jitterbuffers implementations now?

By: zoa (zoa) 2006-05-01 07:46:39

yes you are using the two at the same time

By: Rosario Pingaro (rpingar) 2006-05-01 12:36:02

Zoa,

any new update about fixing the actual issues?

Regards
Rosario

By: zoa (zoa) 2006-05-02 02:39:29

What bug are you referring to ?


Mar 17 10:57:09 WARNING[22866]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=-1, ts=166273064
Mar 17 10:57:09 WARNING[22866]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=-1, ts=166273064

These bugs are because of invalid timing info coming from iax2.c
I'm not aware of any other bugs. (you will not see this with sip-> zap)

We are investigating where those invalid timestamps come from.

By: Rosario Pingaro (rpingar) 2006-05-02 02:46:29

sorry to note that I don't have any sip->iax call.

My calls are all sip->woomera and woomera->sip.

The problem is not only about logs it is also about choppy audio.

Reagrds
Rosario

By: Rosario Pingaro (rpingar) 2006-05-02 02:49:06

in my log everything is at 0 also the ts

WARNING[21689]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=0, ts=0

By: Serge Vecher (serge-v) 2006-05-10 10:39:30

all, please note that the latest code has moved to a new location:

http://svn.digium.com/view/asterisk/team/group/rtpjitterbuffer

please test and report your findings -- thank you!

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-05-11 05:59:42

perhaps this should be added to the trunk soon? we really don't want to wait until 1.6 or 1.8 or 3.0 to get this, do we?

By: Serge Vecher (serge-v) 2006-05-11 09:13:21

roy: you have tested this feature for a while, right? So why not write a small note to the effect "I've downloaded the latest branch code and tested it for a week with no problems"? Testing feature and reporting on it is what facilitates their inclusion into trunk.

thanks,

P.S. I'll see that you get some karma for testing too.

By: zoa (zoa) 2006-05-15 12:18:49

Could everybody who sees things as:

Mar 17 10:57:09 WARNING[22866]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=-1, ts=166273064
Mar 17 10:57:09 WARNING[22866]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=-1, ts=166273064

on sip to sip or sip to zap calls report this to us and help us find where they come from ? We dont see this on our servers.

Zoa

By: Russell Bryant (russell) 2006-05-22 11:46:44

I see all of these options documented in sip.conf.sample and zapata.conf.sample, but none of them are implemented in the code.  Is this something that got lost in a later patch that was applied to this branch?

By: Russell Bryant (russell) 2006-05-22 11:55:37

Nevermind!  I found where they are implemented.  Oops!  :)

By: Michael Willmott (navgar) 2006-05-25 07:15:20

I've just build the rtpjitterbuffer branch this morning to test and had a few issues.  My existing asterisk configuration only uses SIP and ZAP channels.  Here's is what I found.

non-bridged calls (navigating the menu/playback extensions) sound fine.
SIP clients in a bridged call have severly disrupted audio (it sounds like the audio frames are being received too slowly, with lots of breaks in the audio stream).  This is found on both ends of a SIP-SIP call, and only on the SIP end of a SIP-ZAP call.
The audio disruption is present, regardless of the setting of jb-enable and jb-force in both sip.conf and zapata.conf.
When jb-force is used, I see messages "Recieved frame with invalid timing info: has_timing_info=0, len=0, ts=0" appearing very fast on the console.

Just rebuilt to trunk and confirmed the problem is not present.  Is there anything I can try to help you investigate this ?

By: Russell Bryant (russell) 2006-05-25 08:20:31

Thank you for the feedback.  I started testing this a couple of days ago and was seeing the same kind of issues.  However, when I tried the version of this code for the 1.2 branch (svn/asterisk/team/oej/jitterbuffer-1.2), everything worked fine.

I sent an email directly to Slav to see if he could try out the trunk version to see what results he sees.  Hopefully we can figure out what is causing these problems very soon ...

By: zoa (zoa) 2006-05-26 03:59:05

royk just emailed me to say that he has issues even with the 1.2 version.

It seems to work most of the time but sometimes he has those timestamp errors and garbled audio at that time.

We are using the jitter buffer in production on several machines and have never seen such an error. :(

Is somebody on this list who saw such an error before capable of taking such a machine out of production for testing and identifying the issue ?


Roy also sees memory leaks, we see no memory leak at all, (Even after a million calls).

Zoa.

By: Russell Bryant (russell) 2006-05-26 10:30:16

The issues that file and I saw when testing, and it appears that navgar saw as well, were immediately apparent.  It was not something that happened over time.

Just a reminder, anything that is going to be in Asterisk 1.4 has to be merged by this Wednesday (May 31st).  It really is a shame that everyone was using the 1.2 version and not testing the trunk version as well so that we may have identified these issues sooner and had more time to figure out what was going on.  

I'm going to do my best to spend time on this between now and then, but as most of us, I have a lot of other obligations as well.

By: zoa (zoa) 2006-05-26 10:47:51

Slav is going to try to fix the trunk version next week.

We used the 1.2 version to make sure we didn't chase bugs introduced by other developments.

Btw, royk sees a memory leak when using jbforce on sip-> sip calls.
(a memory leak that is freed when the jb is disabled and sip is reloaded).

We tested the 1.2 version with millions of calls and do not see a memory leak in this version. Could you stresstest it with the empirix, Russel ?

Zoa

By: Russell Bryant (russell) 2006-05-26 12:49:36

I absolutely understand your perspective for using 1.2 during the development of this feature.  My comments were aimed at all of the testers, not you guys.  Slav, if there is anything I can do to help you out while you are looking at this, please let me know!

By: Rosario Pingaro (rpingar) 2006-05-26 16:09:10

Any possibility to see SIP jitter buffer to work also with WOOMERA channel?

How could I help to have it included?

Infact I have tested almost each version out on this page but I get always:
WARNING[21689]: abstract_jb.c:347 ast_jb_put: Recieved frame with invalid timing info: has_timing_info=0, len=0, ts=0

Regards
Rosario

By: Russell Bryant (russell) 2006-05-26 17:02:30

Given that there is no woomera channel driver in the official Asterisk tree, that is not really an appropriate question to ask here.  However, given the generic nature of the way this has been implemented, it would be trivial to update any channel driver to support it.

By: Rosario Pingaro (rpingar) 2006-05-27 01:29:25

what could I ask to Tony Minnesale to add?

Is there any refenrence code?

Regards
Rosario

By: Roy Sigurd Karlsbakk (rkarlsba) 2006-05-29 05:23:24

Testing so far shows very good audio even with a heavily overloaded link. Latency can be really bad with high traffic, but that is logical.

Keep up the good work :)

roy

By: Hekuran (haks) 2006-05-29 11:03:42

I have build the rtpjitterbuffer from the new link but asterisk is unnable to load the g729 codec. so it segfaults:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1075625952 (LWP 29601)]
__load_resource (resource_name=0x816b54f "codec_g729.so",
   cfg=<value optimized out>) at loader.c:734
734             if (!m->unload_module && !(m->flags & NO_UNLOAD) )

By: Russell Bryant (russell) 2006-05-29 11:13:59

Unfortunately, this is expected behavior.  The g729 codec has not been updated to work with the trunk yet.  The trunk is still awaiting changes that need to be made before a final g729 binary for 1.4 can be made.

By: Michael Willmott (navgar) 2006-05-31 06:01:20

Just finished preliminary testing of revision 30987 on SIP-SIP calls with both jb off and forced.  Previously reported issues appear to have been resolved.  Currently rebuilding production system to see how things go in SIP-ZAP, will report back findings later.

By: Rosario Pingaro (rpingar) 2006-05-31 07:24:25

howto patch woomera channel to support this generic jitter buffer?

Regards
Rosario

By: Michael Willmott (navgar) 2006-05-31 10:09:00

I've now been running this branch in production for 4 hours with a constant load of about 6-8 SIP-ZAP calls with no notable issues.

By: Russell Bryant (russell) 2006-05-31 12:02:19

This has now been merged into the trunk in revision 31052.  If there are any futher bug reports, please open a new issue, as I think Mantis may explode if we keep using this one.

Thank you everyone, especiall Slav!