[Home]

Summary:ASTERISK-28423: ARI causes STASIS Deadlock
Reporter:Ross Beer (rossbeer)Labels:
Date Opened:2019-05-22 07:47:51Date Closed:2020-01-30 09:15:57.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Resources/res_stasis
Versions:13.26.0 GIT Frequency of
Occurrence		Frequent
Related
Issues:
causesASTERISK-28679 stasis application is destroyed after its creation
is duplicated byASTERISK-28540 Deadlock In Stasis/ARI
Environment:CentOS 7Attachments:
Description:When using WebSocket ARI connections and the connection drops multiple times, stasis deadlocks.
Comments:By: Asterisk Team (asteriskteam) 2019-05-22 07:47:52.336-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.
By: Friendly Automation (friendly-automation) 2019-11-14 11:16:39.654-0600

Change 13172 merged by Friendly Automation:
stasis: Don't hold app_registry and session locks unnecessarily

[https://gerrit.asterisk.org/c/asterisk/+/13172|https://gerrit.asterisk.org/c/asterisk/+/13172]
By: Friendly Automation (friendly-automation) 2019-11-14 11:23:49.851-0600

Change 13189 merged by Friendly Automation:
stasis: Don't hold app_registry and session locks unnecessarily

[https://gerrit.asterisk.org/c/asterisk/+/13189|https://gerrit.asterisk.org/c/asterisk/+/13189]
By: Friendly Automation (friendly-automation) 2019-11-14 21:59:30.166-0600

Change 13171 merged by Friendly Automation:
stasis: Don't hold app_registry and session locks unnecessarily

[https://gerrit.asterisk.org/c/asterisk/+/13171|https://gerrit.asterisk.org/c/asterisk/+/13171]
By: Friendly Automation (friendly-automation) 2019-11-14 22:08:21.319-0600

Change 13190 merged by Friendly Automation:
stasis: Don't hold app_registry and session locks unnecessarily

[https://gerrit.asterisk.org/c/asterisk/+/13190|https://gerrit.asterisk.org/c/asterisk/+/13190]
By: George Joseph (gjoseph) 2020-01-08 07:55:29.896-0600

[~rossbeer] Reports issue is still happening.

By: Kevin Harwell (kharwell) 2020-01-08 09:44:42.595-0600

Looks like the current patch here might have caused another bug: ASTERISK-28679
By: Kevin Harwell (kharwell) 2020-01-24 12:41:44.769-0600

From what I can tell the [original patch|https://gerrit.asterisk.org/c/asterisk/+/13171] does indeed fix a deadlock that involved the "apps_registry" and "event_session" objects.

This is just a guess as I've only done a code inspection, and haven't seen a backtrace for the latest reported deadlock, but I think fixing that exposed another deadlock further "down". This time involving "app" and "event_session" objects:

Partial call stack for Thread A:
{noformat}
stasis_app_message_handler --> ao2_lock(session)
app_send
app_update --> ao2_lock(app)
__stasis_app_register
...
{noformat}
Partial call stack for Thread B:
{noformat}
stasis_app_event_allowed --> ao2_lock(app)
stasis_app_message_handler --> ao2_lock(session)
app_send
sub_channel_update_handle
...
{noformat}
By: Friendly Automation (friendly-automation) 2020-01-30 09:15:59.053-0600

Change 13700 merged by Friendly Automation:
stasis/app: don't lock an app before a call to send

[https://gerrit.asterisk.org/c/asterisk/+/13700|https://gerrit.asterisk.org/c/asterisk/+/13700]
By: Friendly Automation (friendly-automation) 2020-01-30 09:25:27.043-0600

Change 13676 merged by George Joseph:
stasis/app: don't lock an app before a call to send

[https://gerrit.asterisk.org/c/asterisk/+/13676|https://gerrit.asterisk.org/c/asterisk/+/13676]
By: Friendly Automation (friendly-automation) 2020-01-30 09:26:02.416-0600

Change 13699 merged by George Joseph:
stasis/app: don't lock an app before a call to send

[https://gerrit.asterisk.org/c/asterisk/+/13699|https://gerrit.asterisk.org/c/asterisk/+/13699]
By: Friendly Automation (friendly-automation) 2020-01-30 09:43:57.156-0600

Change 13716 merged by George Joseph:
stasis/app: don't lock an app before a call to send

[https://gerrit.asterisk.org/c/asterisk/+/13716|https://gerrit.asterisk.org/c/asterisk/+/13716]
By: Friendly Automation (friendly-automation) 2020-01-30 09:44:30.898-0600

Change 13717 merged by George Joseph:
stasis/app: don't lock an app before a call to send

[https://gerrit.asterisk.org/c/asterisk/+/13717|https://gerrit.asterisk.org/c/asterisk/+/13717]
By: Friendly Automation (friendly-automation) 2020-01-30 09:44:58.899-0600

Change 13718 merged by George Joseph:
stasis/app: don't lock an app before a call to send

[https://gerrit.asterisk.org/c/asterisk/+/13718|https://gerrit.asterisk.org/c/asterisk/+/13718]
By: Friendly Automation (friendly-automation) 2020-01-30 09:55:41.587-0600

Change 13701 merged by Friendly Automation:
stasis/app: don't lock an app before a call to send

[https://gerrit.asterisk.org/c/asterisk/+/13701|https://gerrit.asterisk.org/c/asterisk/+/13701]