[Home]

Summary:ASTERISK-29780: bridge: Sending answer towards called party causes bidirectional audio to drop permanently due to AST_CONTROL_SRCCHANGE
Reporter:N A (InterLinked)Labels:
Date Opened:2021-11-28 15:51:16.000-0600Date Closed:
Priority:MinorRegression?
Status:Open/NewComponents:Channels/chan_iax2 Core/Bridging Core/Channels
Versions:18.8.0 Frequency of
Occurrence
Constant
Related
Issues:
Environment:Attachments:( 0) A_to_B_directly_fail.txt
( 1) A_to_B_directly_with_just_console_channel_driver_success.txt
( 2) A_to_C_to_B,_log_from_C_fail.txt
( 3) A_to_C_to_B_fail.txt
( 4) example.wav
( 5) Issue.txt
( 6) ken-iax-nonworking.txt
( 7) ken-iax-working.txt
( 8) ken-nonworking-bridged.txt
( 9) ken-working-normal.txt
(10) min_example_side_a.txt
(11) min_example_side_b.txt
Description:We have discovered that if a call is made over IAX2 into an Asterisk system that completes the call using DAHDI, if the D() option is used on the DAHDI side to send any DTMF tones to the called party on answer, that will cause audio to stop passing for the remainder of the call. This appears to be a complex interaction of factors between the two sides of the call. Several conditions must be just right, and if so the call will fail every single time.

- Call must complete using DAHDI to the destination line on Side B. No issues with SIP.
- The D option must be used in Dial on Side B to send DTMF digits to the destination endpoint on answer. Digits do not matter. No issues if no DTMF sent.
- Only IAX2 between A and B has been tested. Trunk is not encrypted or authenticated.
- Side A must use the G option for Dial to split the call on answer. No issues if call is not split.

If A originates a call to B in the Asterisk CLI using the console channel driver, there are no audio issues.
If using an ATA registered to A to make a call through B, where the first thing that Asterisk A does is call B, there are no audio issues.
If the call on side A is set up in just the right particular way, then the no audio issue can occur. This will occur regardless of whether A ends up calling B directly or the call goes through an intermediary (say, C), which then directly calls B.

The exact mechanism of this bug appears to be that DAHDI on Side B will send a control frame over IAX2 to the originating side. On a good call, that is all that happens. However, on a bad call, this control frame is "echoed" back to Side B, and this causes audio to drop in both directions for the remainder of the call.

Side A tested with several versions of Asterisk 18. Side B is running Asterisk 11. The issue itself is on Side A, however. It is merely an issue that is made obvious in response to a control frame being sent from Side B. This therefore appears to a bug with core bridging and/or IAX2.

Using iax2 debug / debug 10, I have narrowed down the exact point where the issue starts to this:

[2021-11-28 14:35:46] DEBUG[26626][C-000000c9]: chan_iax2.c:5762 iax2_indicate: Indicating condition -1 <---------------------------- NOT NORMAL.
[2021-11-28 14:35:46] DEBUG[26625][C-000000c9]: pbx.c:2938 pbx_extension_helper: Launching 'Set'
[2021-11-28 14:35:46] DEBUG[1268]: chan_iax2.c:3414 send_packet: Sending 7823 on 12206/6856 to ASTERISK-SIDE-B-IP
[2021-11-28 14:35:46] Tx-Frame Retry[000] -- OSeqno: 002 ISeqno: 004 Type: CONTROL Subclass: (255?) <---------------------------- transmit 255? back to side B. NOT NORMAL.
[2021-11-28 14:35:46]     Timestamp: 07823ms  SCall: 12206  DCall: 06856 ASTERISK-SIDE-B-IP

This always starts directly after the "Indicating condition -1", which does NOT show up on a good call, only on bad calls.

This does, furthermore, appear to be linked to a CONTROL frame of 255?. Usually, Side B sends one of these to Side A. It seems that on bad calls, this CONTROL frame of 255? is echoed back to Side B. Additionally, I see "Blocked sending condition 26" which does not appear in a good call.
Condition 26 appears to be AST_CONTROL_SRCCHANGE = 26, /*!< Media source has changed and requires a new RTP SSRC */. -1 isn't a condition, so I'm assuming that means some failure has occured which may be why this is happening.

[Nov 28 09:43:28] Rx-Frame Retry[ No] -- OSeqno: 002 ISeqno: 002 Type: CONTROL Subclass: ANSWER
[Nov 28 09:43:28]     Timestamp: 07774ms  SCall: 13234  DCall: 12944 ASTERISK-SIDE-B-IP
[Nov 28 09:43:28] DEBUG[18882][C-00000354]: chan_iax2.c:10404 socket_process_helper: Received packet 2, (4, 4)
[Nov 28 09:43:28] DEBUG[18882][C-00000354]: chan_iax2.c:3414 send_packet: Sending 7774 on 12944/13234 to ASTERISK-SIDE-B-IP
[Nov 28 09:43:28] Tx-Frame Retry[-01] -- OSeqno: 002 ISeqno: 003 Type: IAX     Subclass: ACK
[Nov 28 09:43:28]     Timestamp: 07774ms  SCall: 12944  DCall: 13234 ASTERISK-SIDE-B-IP
[Nov 28 09:43:28] DEBUG[18882][C-00000354]: chan_iax2.c:11996 socket_process_helper: For call=12944, set last=7774
[Nov 28 09:43:28] Rx-Frame Retry[ No] -- OSeqno: 003 ISeqno: 002 Type: CONTROL Subclass: (255?) <----------------------------------- we get a 255? from Side B. This is normal.
[Nov 28 09:43:28]     Timestamp: 07777ms  SCall: 13234  DCall: 12944 ASTERISK-SIDE-B-IP
[Nov 28 09:43:28] DEBUG[18882][C-00000354]: chan_iax2.c:10404 socket_process_helper: Received packet 3, (4, -1)
[Nov 28 09:43:28] DEBUG[18882][C-00000354]: chan_iax2.c:3414 send_packet: Sending 7777 on 12944/13234 to ASTERISK-SIDE-B-IP
[Nov 28 09:43:28] Tx-Frame Retry[-01] -- OSeqno: 002 ISeqno: 004 Type: IAX     Subclass: ACK
[Nov 28 09:43:28]     Timestamp: 07777ms  SCall: 12944  DCall: 13234 ASTERISK-SIDE-B-IP
.....
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: chan_iax2.c:5743 iax2_indicate: Indicating condition 26 <---------------------- NOT NORMAL.
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: chan_iax2.c:7745 send_command: Callno 11602: Blocked sending control frame 26. <---------------------- NOT NORMAL.
[Nov 28 09:43:28] DEBUG[18870]: cdr.c:1470 cdr_object_finalize: Finalized CDR for IAX2/ASTERISK-SIDE-B-IP:4569-12944 - start 1638110600.454928 answer 1638110608.228645 end 1638110608.232358 dur 7.777 bill 0.003 dispo ANSWERED
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: chan_iax2.c:5743 iax2_indicate: Indicating condition 26 <---------------------- NOT NORMAL.
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: chan_iax2.c:7745 send_command: Callno 12944: Blocked sending control frame 26. <---------------------- NOT NORMAL.
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: chan_iax2.c:5743 iax2_indicate: Indicating condition -1 <----------------------------- NOT NORMAL.
[Nov 28 09:43:28] TE-Frame Retry[000] -- OSeqno: 007 ISeqno: 009 Type: CONTROL Subclass: (255?) <---------------------------- We forward the 255? to originating switch.
[Nov 28 09:43:28]     Timestamp: 09821ms  SCall: 11602  DCall: 17130 ASTERISK-SIDE-A-IP



For reference, this is what happens immediately prior to the indicating condition 26. This is a call from A -> C -> B (debug from C):
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge_channel.c:2911 bridge_channel_internal_join: Bridge d13d477a-3765-433c-8f13-7f57369fdd64: 0x7fa8d407dba0(IAX2/ASTERISK-SIDE-B-IP:4569-12944) is joining
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge_channel.c:2251 bridge_channel_internal_push_full: Bridge d13d477a-3765-433c-8f13-7f57369fdd64: pushing 0x7fa8d407dba0(IAX2/ASTERISK-SIDE-B-IP:4569-12944)
[Nov 28 09:43:28]     -- Channel IAX2/ASTERISK-SIDE-B-IP:4569-12944 joined 'simple_bridge' basic-bridge <d13d477a-3765-433c-8f13-7f57369fdd64>
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: stasis_bridges.c:290 bridge_snapshot_update_create: Update: 0x7fa8b401db98  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: stasis_bridges.c:270 bridge_snapshot_update_dtor: Update: 0x7fa8b401db98  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge_native_rtp.c:774 native_rtp_bridge_compatible: Bridge 'd13d477a-3765-433c-8f13-7f57369fdd64' can not use native RTP bridge as two channels are required
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge.c:535 find_best_technology: Bridge technology native_rtp is not compatible with properties of existing bridge.
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge.c:525 find_best_technology: Bridge technology holding_bridge does not have any capabilities we want.
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge.c:525 find_best_technology: Bridge technology softmix does not have any capabilities we want.
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge.c:549 find_best_technology: Chose bridge technology simple_bridge
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge.c:1058 smart_bridge_operation: Bridge d13d477a-3765-433c-8f13-7f57369fdd64 is already using the new technology.
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: bridge.c:448 bridge_channel_complete_join: Bridge d13d477a-3765-433c-8f13-7f57369fdd64: 0x7fa8d407dba0(IAX2/ASTERISK-SIDE-B-IP:4569-12944) is joining simple_bridge technology
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: stasis_bridges.c:290 bridge_snapshot_update_create: Update: 0x7fa8b4019e88  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76886][C-00000354]: stasis_bridges.c:270 bridge_snapshot_update_dtor: Update: 0x7fa8b4019e88  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge_channel.c:2911 bridge_channel_internal_join: Bridge d13d477a-3765-433c-8f13-7f57369fdd64: 0x7fa8d411a650(IAX2/sfcu-11602) is joining
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge_channel.c:2251 bridge_channel_internal_push_full: Bridge d13d477a-3765-433c-8f13-7f57369fdd64: pushing 0x7fa8d411a650(IAX2/sfcu-11602)
[Nov 28 09:43:28]     -- Channel IAX2/sfcu-11602 joined 'simple_bridge' basic-bridge <d13d477a-3765-433c-8f13-7f57369fdd64>
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: stasis_bridges.c:290 bridge_snapshot_update_create: Update: 0x7fa8d405bf88  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: stasis_bridges.c:270 bridge_snapshot_update_dtor: Update: 0x7fa8d405bf88  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge_native_rtp.c:647 native_rtp_bridge_compatible_check: Bridge 'd13d477a-3765-433c-8f13-7f57369fdd64'.  Checking compatability for channels 'IAX2/ASTERISK-SIDE-B-IP:4569-12944' and 'IAX2/sfcu-11602'
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge_native_rtp.c:657 native_rtp_bridge_compatible_check: Bridge 'd13d477a-3765-433c-8f13-7f57369fdd64' can not use native RTP bridge as channel 'IAX2/sfcu-11602' has features which prevent it
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge.c:535 find_best_technology: Bridge technology native_rtp is not compatible with properties of existing bridge.
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge.c:525 find_best_technology: Bridge technology holding_bridge does not have any capabilities we want.
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge.c:525 find_best_technology: Bridge technology softmix does not have any capabilities we want.
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge.c:549 find_best_technology: Chose bridge technology simple_bridge
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge.c:1058 smart_bridge_operation: Bridge d13d477a-3765-433c-8f13-7f57369fdd64 is already using the new technology.
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: bridge.c:448 bridge_channel_complete_join: Bridge d13d477a-3765-433c-8f13-7f57369fdd64: 0x7fa8d411a650(IAX2/sfcu-11602) is joining simple_bridge technology
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: channel.c:5752 set_format: Channel IAX2/sfcu-11602 setting read format path: ulaw -> ulaw
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: channel.c:5752 set_format: Channel IAX2/ASTERISK-SIDE-B-IP:4569-12944 setting write format path: ulaw -> ulaw
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: channel.c:5752 set_format: Channel IAX2/ASTERISK-SIDE-B-IP:4569-12944 setting read format path: ulaw -> ulaw
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: channel.c:5752 set_format: Channel IAX2/sfcu-11602 setting write format path: ulaw -> ulaw
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: stasis_bridges.c:290 bridge_snapshot_update_create: Update: 0x7fa8d4024cb8  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: stasis_bridges.c:270 bridge_snapshot_update_dtor: Update: 0x7fa8d4024cb8  Old: d13d477a-3765-433c-8f13-7f57369fdd64  New: d13d477a-3765-433c-8f13-7f57369fdd64
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: chan_iax2.c:5743 iax2_indicate: Indicating condition 26 <---------------------- NOT NORMAL.
[Nov 28 09:43:28] DEBUG[76877][C-00000354]: chan_iax2.c:7745 send_command: Callno 11602: Blocked sending control frame 26. <---------------------- NOT NORMAL.
[Nov 28 09:43:28] DEBUG[18870]: cdr.c:1470 cdr_object_finalize: Finalized CDR for IAX2/ASTERISK-SIDE-B-IP:4569-12944 - start 1638110600.454928 answer 1638110608.228645 end 1638110608.232358 dur 7.777 bill 0.003 dispo ANSWERED
.....
Comments:By: Asterisk Team (asteriskteam) 2021-11-28 15:51:18.741-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/].

By: Joshua C. Colp (jcolp) 2021-11-29 04:22:27.387-0600

The -1 means to stop any tones which are being played.

As well, please ATTACH the complete debug logs from both servers.

By: N A (InterLinked) 2021-11-29 05:09:10.137-0600

All right, here are some debug logs at debug 10 + iax2 debug. Except for that one that says "from C", the logs are from either A, or C if it was in the call path (originating or middle side, upstream from terminating switch).

A simple call that succeeded is included for reference.

Also, am I intentionally assigned to this issue? This is about as much as I know here.


By: Joshua C. Colp (jcolp) 2021-11-29 05:25:30.622-0600

When waiting for feedback it goes to the person that it is waiting for feedback from, and then once a comment/change occurs it automatically transitions out into triage.

By: N A (InterLinked) 2021-12-04 20:28:32.491-0600

Below is a minimally reproducible dialplan for the originating side of the call, for Asterisk 18:

The ConfBridge sections are primarily to get app_confbridge out of our way so we can hear everything.

extensions.conf:
[fail-test] ; to minimize the # of things in the call, test using the console channel driver: "channel originate Local/s@fail-test application Wait 35"
exten => s,1,Answer()
   same => n,MixMonitor(fail-${EPOCH}.wav)
   same => n,Originate(Local/s@bridge,exten,outgoing-ring,2579,1,,a)
   same => n,Goto(bridge,s,1)

[bridge]
exten => s,1,Answer()
   same => n,ConfBridge(testbridge,silentbridge,incogtalker)

[outgoing-ring]
exten => 2579,1,Set(CDR_PROP(disable)=1)
   same => n,Answer()
   same => n,Dial(Local/${EXTEN}@originate-local/n,,G(split))
   same => n(split),Hangup()
   same => n(callee),Goto(bridge,s,1)

[originate-local] ; actual outgoing call
exten => 2579,1,Dial(IAX2/REDACTED/REDACTED) ; <-- this is a call to another Asterisk which uses DAHDI to complete a call to an analog switch using an analog trunk. On its end, it uses the D option with Dial(DAHDI...) and if DTMF is sent on answer, that's when all this weirdness happens!
   same => n,Hangup()

confbridge.conf:
; these are provided mainly so that the confbridge is silent and not in our way, so we can hear everything that's happening, all the time
[silentbridge]
type=bridge
sound_join=/var/lib/asterisk/sounds/en/silence/1
sound_leave=/var/lib/asterisk/sounds/en/silence/1

[incogtalker]
type=user
dsp_drop_silence=no
quiet=yes
dtmf_passthrough=yes
answer_channel=no

By: N A (InterLinked) 2021-12-04 20:30:16.495-0600

Looks like the formatting got messed up on that comment... here it is as a txt.

This is the minimally reproducible dialplan for this bug on the originating side.

By: N A (InterLinked) 2021-12-04 20:34:23.832-0600

Attached is an audio recording of this phenomenon (using the minimally reproducible dialplan code).

As can be heard, there are a few seconds of ringback on the analog switch that is called. For testing, a milliwatt test line on the switch is called with DTMF sent to it on answer (by the Asterisk on Side B).

The line answers and a brief blip of milliwatt tone can be heard before audio is cut out so that DTMF is sent. However, the audio never returns.

On a normal call, obviously, MW returns and continues to be audible.

By: N A (InterLinked) 2021-12-29 07:00:08.214-0600

Minimum dialplan from Side B (DAHDI terminating)

By: N A (InterLinked) 2022-01-26 20:03:44.569-0600

I encountered almost exactly this issue with another environment, which led me to suspect something larger was going.

Based on my testing, I believe the issue is that if you use the G dial option to fork on answer, and then Answer() the called side of the leg, which is bridged over IAX2 with DAHDI on the other end, this basically destroys the call.

What happened with the original issue is there would be no audio after the call was answered, in either direction.

In this case, on a working call, DAHDI would dial pulse into the end switch and ring the phone normally. In the broken case, DAHDI would get stuck either at the beginning or in the middle of inpulsing, and digits would never finish getting sent. However, I still had an audio path; I could hear trunk and battery noise and a bunch of echo (the trunks on that end are very echoey, so that's not unusual to this, on a good call, there is also lots of echo, as the echo cancellation is wonky over there.) Point is, I can hear stuff, the call just doesn't go through.

In BOTH cases, my sending an ANSWER frame (the CONTROL 255?) in the iax debug causes DAHDI on the terminating side over the IP link to do something very weird.

So the bug is either:
1) When a call is answered post-split TOWARDS the called party (which just answered), we're sending something we shouldn't be.
The 255? subclass in the iax debugs corresponds to the answer, and sending an answer is sending an answer, so I'm not sure that's the issue.

However, at least, after five months, I have finally determined the workaround here is to simply not send an Answer, which shouldn't be needed anyways, really. However, explicitly sending an Answer shouldn't cause all hell to break loose, so this is still a bug, and the workaround confirms that the answer is responsible for causing this (forking the call on answer may not be directly relevant either, but as that's the way to send something to the called party after they answer, it's hard to say whether or not it's relevant.)

2) DAHDI reacts VERY badly to things if it receives an answer from the calling party. Given that the ANSWER is all that gets sent over IAX2, I can't see how anything else on the originating side would matter.

Again, this isn't all DAHDI calls. It seems to be only when DTMF or features are used. With the original issue, it was when the D option was used to send a tone to the caller on answer. With this case, the Tt options were specified for transfer. However, it also seems to happen without any options, which puzzles me. On the original issue, "vanilla" DAHDI dials with no features completed just fine either way. With this new case, everything seems to be broken and the only way to unbreak it is to not send the answer from Side A in the first place.

In short, sending an answer to a DAHDI switch when it's the called party MAY cause the call to bork, if certain features such as Send DTMF to caller on answer, and possibly others, are being used.

Attached are 4 debugs which prove this:
1) core set debug 10 for a broken call (on Side B)
2) core set debug 10 for a good call (on Side B)
3) iax2 debug for a broken call (from Side A)
4) iax2 debug for a good call. (from Side A)

Note that the core debugs didn't have iax2 debug enabled so they won't show up. Here are some things to note:
only on working:

only on nonworking:
[2022-01-25 20:36:12] DEBUG[1806][C-00000018] chan_dahdi.c: Requested indication -1 on channel
DAHDI/23-1

ANALOG_EVENT_RINGBEGIN occurs 4x in working, 1x in failure
ditto for ANALOG_EVENT_RINGOFFHOOK
ANALOG_EVENT_DIALCOMPLETE occurs 1x in each.

In both, a control subclass 255? frame is sent from Side B to Side A. (this is the answer on Side B)
However, only in the broken call is a 255? sent back from Side A to Side B in response:
[2022-01-27 00:26:43] Tx-Frame Retry[000] -- OSeqno: 002 ISeqno: 003 Type: CONTROL Subclass: (255?)
This is the answer from Side A towards the caller.

Additionally, I have confirmed that removing the answer also successfully works around the original node being called so this is the exact same root underlying issue in both cases.

As can be seen by comparing the debugs - specifically the differences between them - the answer frame is what spawns the issue.