Asterisk
  1. Asterisk
  2. ASTERISK-12877

channel get stuck on ast_queue_frame when hanging up

    Details

    • Type: Bug Bug
    • Status: Closed
    • Severity: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Target Release Version/s: None
    • Component/s: Channels/General
    • Labels:
      None
    • Mantis ID:
      13676
    • Regression:
      No

      Description

      From my understanding, as result of a channel stuck on ast_queue_frame when hanging up and not unlocking their &p->lock a bunch of AgentCallBackLogin() threads get stuck trying to get the agents list.

      They are plenty of warnings messages on the logger output:

      • channel.c: Avoiding deadlock for channel
      • channel.c: Dropping voice to exceptionally long queue on IAX/...
      • chan_iax2.c: Max retries exceeded to host ... on IAX2/.. (type = ., subclass = ., ts=1., seqno=.)

      The scenario is a production system on a call center enviroment, calls are made and received through an IAX2 facility. This use to append after 40 to 60 minutes uptime with about 20 logged agents using AgentCallBackLogin() mainly doing outbound calls.

                • ADDITIONAL INFORMATION ******

      ===
      === Thread ID: 3058826128 (pbx_thread started at [ 2645] pbx.c ast_pbx_start())
      === --> Lock #0 (channel.c): MUTEX 2613 ast_write &chan>lock 0x90e8420 (1)
      === --> Lock #1 (chan_agent.c): MUTEX 630 agent_write &p>lock 0x8fa8f10 (1)
      === --> Lock #2 (channel.c): MUTEX 2613 ast_write &chan>lock 0x8fe63f8 (1)
      === --> Lock #3 (channel.c): MUTEX 3416 ast_do_masquerade &clone>lock 0x91901c0 (1)
      === --> Lock #4 (chan_local.c): MUTEX 515 local_hangup &p>lock 0x9037428 (1)
      === --> Tried and failed to get Lock ASTERISK-1 (channel.c): MUTEX 962 ast_queue_hangup &chan>lock 0x90e7b78 (1)
      === -------------------------------------------------------------------
      ===
      === Thread ID: 3041868688 (pbx_thread started at [ 2645] pbx.c ast_pbx_start())
      === --> Lock #0 (channel.c): MUTEX 2613 ast_write &chan>lock 0x90e7b78 (1)
      === --> Waiting for Lock #1 (chan_local.c): MUTEX 309 local_write &p>lock 0x9037428 (1)
      === — ---> Locked Here: chan_local.c line 515 (local_hangup)
      === -------------------------------------------------------------------
      ===
      === Thread ID: 3009919888 (pbx_thread started at [ 2645] pbx.c ast_pbx_start())
      === --> Lock #0 (chan_agent.c): MUTEX 2023 __login_exec &(&agents)>lock 0x585a28 (1)
      === --> Lock #1 (chan_agent.c): MUTEX 2026 __login_exec &chan>lock 0xb7993a68 (1)
      === --> Waiting for Lock #2 (chan_agent.c): MUTEX 2027 __login_exec &p>lock 0x8fa8f10 (1)
      === — ---> Locked Here: chan_agent.c line 630 (agent_write)
      === -------------------------------------------------------------------
      ===
      === Thread ID: 3032038288 (pbx_thread started at [ 2645] pbx.c ast_pbx_start())
      === --> Waiting for Lock #0 (chan_agent.c): MUTEX 2023 __login_exec &(&agents)>lock 0x585a28 (1)
      === — ---> Locked Here: chan_agent.c line 2023 (__login_exec)
      === -------------------------------------------------------------------

      1. 13676.patch
        0.4 kB
        Mark Michelson
      2. ast_show_channels
        12 kB
      3. ast_show_locks
        52 kB
      4. ast_show_threads
        18 kB
      5. bt_full_thread_174
        25 kB
      6. bt_thread_apply_all_bt
        296 kB

        Activity

        Hide
        Octavio Ruiz added a comment -

        Same symptoms with SVN-branch-1.4-r148257.

        Show
        Octavio Ruiz added a comment - Same symptoms with SVN-branch-1.4-r148257.
        Hide
        Phoebe Anderson added a comment -

        This may be related to issue 0013645. As of 1.4.22, IAX2 doesn't seem to hangup properly.

        Show
        Phoebe Anderson added a comment - This may be related to issue 0013645. As of 1.4.22, IAX2 doesn't seem to hangup properly.
        Hide
        Mark Michelson added a comment -

        This appears to be an old-fashioned deadlock. The two competing threads are the first two listed, where the locking order is different between the same mutexes.

        I am uploading a patch which should alleviate the issue.

        Show
        Mark Michelson added a comment - This appears to be an old-fashioned deadlock. The two competing threads are the first two listed, where the locking order is different between the same mutexes. I am uploading a patch which should alleviate the issue.
        Hide
        Mark Michelson added a comment -

        Please try 13676.patch. Thanks!

        Show
        Mark Michelson added a comment - Please try 13676.patch. Thanks!
        Hide
        Octavio Ruiz added a comment -

        Looks like solved, by now that system have more than 60 min uptime; anyway please let me do further tests to make sure the problem has gone.

        Show
        Octavio Ruiz added a comment - Looks like solved, by now that system have more than 60 min uptime; anyway please let me do further tests to make sure the problem has gone.
        Hide
        Octavio Ruiz added a comment -

        More than 4 hours uptime with ~ 80 agents logged in, confirm that it's solved. There is no evidence of more deadlocks. Thank you Mark.

        Show
        Octavio Ruiz added a comment - More than 4 hours uptime with ~ 80 agents logged in, confirm that it's solved. There is no evidence of more deadlocks. Thank you Mark.
        Hide
        Mark Michelson added a comment -

        No problem. I'll get this patch committed. Thanks for testing!

        Show
        Mark Michelson added a comment - No problem. I'll get this patch committed. Thanks for testing!
        Hide
        Digium Subversion added a comment -

        Repository: asterisk
        Revision: 148912

        U branches/1.4/channels/chan_local.c

        ------------------------------------------------------------------------
        r148912 | mmichelson | 2008-10-14 12:23:15 -0500 (Tue, 14 Oct 2008) | 9 lines

        Deadlock prevention in chan_local.

        (closes issue ASTERISK-12877)
        Reported by: tacvbo
        Patches:
        13676.patch uploaded by putnopvut (license 60)
        Tested by: tacvbo

        ------------------------------------------------------------------------

        http://svn.digium.com/view/asterisk?view=rev&revision=148912

        Show
        Digium Subversion added a comment - Repository: asterisk Revision: 148912 U branches/1.4/channels/chan_local.c ------------------------------------------------------------------------ r148912 | mmichelson | 2008-10-14 12:23:15 -0500 (Tue, 14 Oct 2008) | 9 lines Deadlock prevention in chan_local. (closes issue ASTERISK-12877 ) Reported by: tacvbo Patches: 13676.patch uploaded by putnopvut (license 60) Tested by: tacvbo ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=148912

          People

          • Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development