[Home]

Summary:ASTERISK-24464: [patch]ICE Candidates Gathering causes abort in pjproject code due to too small network interface array size
Reporter:Andreas Steinmetz (anstein)Labels:
Date Opened:2014-10-28 10:41:36Date Closed:2017-12-18 09:25:13.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:Resources/res_rtp_asterisk
Versions:11.13.0 Frequency of
Occurrence		Constant
Related
Issues:
Environment:LinuxAttachments:( 0) abort_pjproject.txt
( 1) ice-candidates-length.patch
Description:The array size for network gathering for the ice candidates list  is too small (16). This can cause the required candidates not to be included which lateron triggers a pjproject abort.
Thinking of VLANs and that every network interface has typically at least two addresses an array size of 64 seems to be more reasonable.
Comments:By: Andreas Steinmetz (anstein) 2014-10-28 10:50:11.874-0500

Fix array size for ice candidates (use a more reasonable value)
By: Matt Jordan (mjordan) 2014-10-29 10:01:29.137-0500

While this may still be needed, can you provide a backtrace from the abort in {{pjproject}}? If possible, it would be good to add defensive code to prevent that from happening as well.
By: Andreas Steinmetz (anstein) 2014-11-02 08:36:19.625-0600

Below is the requested backtrace with the sip address obfuscated, though it is now from 11.13.1 which shouldn't matter.  With the array being to small Asterisk offers only IPv6 ICE candidates whereas the client offers only IPv4 ICE candidates thus there can't be any match.

Anyway there should be a way to limit the adresses asterisk offers, e.g. by 'iceallow' and 'icedeny' commands using the acl code. Offering all network addresses of a server can be an unwanted information leak showing possible points of attack either against the server itself or systems on the attached networks. Note that the addresses are easy to view when using e.g. "about:webrtc" in firefox.

[Edit by Rusty - removed abort trace and attached as abort_pjproject.txt]

By: Andreas Steinmetz (anstein) 2014-11-02 13:04:23.609-0600

It seems, just adjusting the array size isn't enough. The SDP sent by Asterisk seems to be limited somewhere else to 16. This results in the SDP containing only "a=candidate:XXXXXXXX 1 UDP" (note the " 1 ") lines and audio setup fails (no audio in call).
With a dirty hack to limit the amount of addresses to just one IPv4 and IPv6 address the SDP then looks like:
{noformat}
a=candidate:H76f0ae12 1 UDP 2130706431 [fda8:de2d:e95f:4811::1] 6054 typ host

a=candidate:Ha010a01 1 UDP 2130706431 10.1.10.1 6054 typ host

a=candidate:H76f0ae12 2 UDP 2130706430 [fda8:de2d:e95f:4811::1] 6055 typ host

a=candidate:Ha010a01 2 UDP 2130706430 10.1.10.1 6055 typ host
{noformat}
Now audio setup succeeds (two way audio).

Note that in the first case (no audio) Asterisk then seems to hang internally:

1. All peers get unregistered, reregistrations fail
2. Lots of CLI commands just do nothing anymore
3. Asterisk cannot be killed by SIGTERM, needs SIGKILL

To reproduce this just configure sufficient interfaces. Use e.g. VLANs not connected anywhere. SIPML5 will do as WebRTC client using Firefox 33.0 on Linux. Note that for Firefox in "about:config" the value "network.dns.disableIPv6=true" needs to be set to prevent Firefox from parsing an IPv6 address in the connection line of the SDP for IPv4 multicast (Firefox bug) and then bailing out.
As a remark I cannot and even if I could, would not report the Firefox bug due to behaviour at mozilla.org.
By: Rusty Newton (rnewton) 2014-11-17 09:50:15.176-0600

Attaching Andreas' backtrace of the abort and removing it from the comment.
By: Rusty Newton (rnewton) 2014-11-17 09:59:55.823-0600

[~anstein] go ahead and put your patch up on Reviewboard per the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Thanks!
By: Sean Bright (seanbright) 2017-02-15 10:58:14.460-0600

PJSIP limits the number of candidates to 16 (See [here|http://www.pjsip.org/pjnath/docs/html/group__PJNATH__CONFIG.htm#gaab843b3c2b1fc2236234a3902f55d0c7]). So you would have to recompile PJSIP with different options.
By: Sean Bright (seanbright) 2017-02-15 11:35:03.999-0600

The assertion failure is pretty odd as well. It implies that {{pj_ice_sess_create_check_list}} was not called, which it was. What version of pjsip were/are you using?

Try setting {{PJ_ICE_MAX_CAND}} to {{64}} in your {{config_site.h}} when compiling pjsip and then apply your patch to Asterisk and see if that works.
By: Andreas Steinmetz (anstein) 2017-02-15 16:43:03.916-0600

As for the version I really can't recall anymore (more than two years) but the bug states that it was 11.13. Anyhow the thing is that the pjsip candidates limit is way to low for a dual stack network. Though it fits the bill for an IPv4 only network one has to multiply the  value with at least 3 (better 4) to fit the bill for dual stack (ipv4 address, link local address, any kind of global address at least per interface). Setting a value of 64 as the limit should be the default nowadays.

Anyway, I will not be able to get around to test this for quite a while due to current job demand as well as health issues. I hope you don't mind.

By: Sean Bright (seanbright) 2017-02-15 16:45:03.302-0600

Thanks Andreas.
By: Joshua C. Colp (jcolp) 2017-12-18 09:25:13.079-0600

We now used a bundled PJSIP and our default configuration allows 32 candidates. If this is still a problem under Asterisk 13 or above feel free to reopen.