[Home]

Summary:ASTERISK-28437: Taskprocessor doesn't process the tasks
Reporter:sungtae kim (pchero)Labels:pjsip
Date Opened:2019-06-05 09:15:23Date Closed:2019-06-18 07:46:27
Priority:MinorRegression?
Status:Closed/CompleteComponents:Core/Stasis
Versions:15.7.2 Frequency of
Occurrence		One Time
Related
Issues:
Environment:Attachments:( 0) core-asterisk-running-2019-06-05T13-01-31+0000-brief.txt
( 1) core-asterisk-running-2019-06-05T13-01-31+0000-full.txt
( 2) core-asterisk-running-2019-06-05T13-01-31+0000-locks.txt
( 3) core-asterisk-running-2019-06-05T13-01-31+0000-thread1.txt
Description:Hi Asterisk team,

One of Asterisk's taskprocessor doesn't process the tasks. Seems like stuck some deadlock, but couldn't figure out why.

{noformat}
CLI> core show taskprocessors

Processor                                      Processed   In Queue  Max Depth  Low water High water
ast_msg_queue                                          0          0          0        450        500
CCSS_core                                              0          0          0        450        500
dns_system_resolver_tp                                 0          0          0        450        500
hep_queue_tp                                           0          0          0        450        500
iax2_transmit                                          0          0          0        450        500
pjsip/default-0000000a                           1207948          0          2        450        500
pjsip/default-0000000b                           1207946          0          1        450        500
pjsip/default-0000000c                           1207949          0          2        450        500
pjsip/default-0000000d                           1207948          0          2        450        500
pjsip/default-0000000e                           1207949          0          2        450        500
pjsip/default-0000000f                           1207950          0          2        450        500
pjsip/default-00000010                           1207947          0          2        450        500
pjsip/default-00000011                           1207945          0          1        450        500
pjsip/distributor-0000001e                        267542          0          4        450        500
pjsip/distributor-0000001f                        272156          0          6        450        500
pjsip/distributor-00000020                        264634          0          5        450        500
pjsip/distributor-00000021                        268282          0          5        450        500
pjsip/distributor-00000022                        270238          0          7        450        500
pjsip/distributor-00000023                        270059          0          4        450        500
pjsip/distributor-00000024                        270525          0          6        450        500
pjsip/distributor-00000025                        270730          0          4        450        500
pjsip/distributor-00000026                        272161          0          5        450        500
pjsip/distributor-00000027                        273404          0          6        450        500
pjsip/distributor-00000028                        274615          0          9        450        500
pjsip/distributor-00000029                        273141          0          4        450        500
pjsip/distributor-0000002a                        269424          0          7        450        500
pjsip/distributor-0000002b                        269376          0          5        450        500
pjsip/distributor-0000002c                        269265          0          6        450        500
pjsip/distributor-0000002d                        270574          0          4        450        500
pjsip/distributor-0000002e                        269473          0          4        450        500
pjsip/distributor-0000002f                        275079          0          5        450        500
pjsip/distributor-00000030                        270892          0          5        450        500
pjsip/distributor-00000031                        274360          0          4        450        500
pjsip/distributor-00000032                        274523          0          4        450        500
pjsip/distributor-00000033                        268106          0          7        450        500
pjsip/distributor-00000034                        269143          0          6        450        500
pjsip/distributor-00000035                        273046          0          4        450        500
pjsip/distributor-00000036                        270502       1354          5        450        500
pjsip/distributor-00000037                        274427          0          6        450        500
pjsip/distributor-00000038                        268568          0          5        450        500
pjsip/distributor-00000039                        271983          0          4        450        500
pjsip/distributor-0000003a                        271705          0          4        450        500
pjsip/distributor-0000003b                        269543          0          5        450        500
pjsip/distributor-0000003c                        271750          0          5        450        500
...
289 taskprocessors
{noformat}

I'm doubting about this part, but not clear how to digging in deeper.
{noformat}
Thread 76 (LWP 7723):
#0  0x00007fe4216d6f5c in ?? () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:107 from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe40cea1460 in ?? () from /usr/lib/asterisk/modules/res_stasis.so
#2  0x0000558571255db2 in __ast_string_field_empty_buffer ()
#3  0x00007fe4216d487c in pthread_cond_signal@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:117
#4  0x0000558570e8deb8 in __ast_cond_signal (filename=<optimized out>, lineno=<optimized out>, func=<optimized out>, cond_name=<optimized out>, cond=<optimized out>) at lock.c:508
#5  0x00007fe40ce9c5da in exec_command_on_condition (can_exec_fn=0x0, data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:273
#6  exec_command (data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:283
#7  stasis_app_send_command_async (control=0x7fe207c56520, command_fn=<optimized out>, data=<optimized out>, data_destructor=<optimized out>) at stasis/control.c:843
#8  0x00007fe40ce9d43b in dial_bridge_after_cb (data=0x7fe207c56520, chan=<optimized out>) at stasis/control.c:941
#9  dial_bridge_after_cb_failed (reason=<optimized out>, data=0x7fe207c56520) at stasis/control.c:951
#10 0x0000558570dbbdf8 in after_bridge_cb_failed (node=0x7fe2b6c95630) at bridge_after.c:70
#11 after_bridge_cb_run_discard (reason=AST_BRIDGE_AFTER_CB_REASON_DESTROY, after_bridge=0x7fe206a3bab0) at bridge_after.c:99
#12 after_bridge_cb_destroy (data=0x7fe206a3bab0) at bridge_after.c:117
#13 0x0000558570e3e60c in ast_datastore_free (datastore=datastore@entry=0x7fe207638580) at datastore.c:68
#14 0x0000558570dffa56 in ast_channel_destructor (obj=0x7fe185b2ffd0) at channel.c:2255
#15 0x0000558570da1310 in __ao2_ref (user_data=0x7fe185b2ffd0, delta=delta@entry=-1, tag=tag@entry=0x7fe386409b29 "", file=file@entry=0x7fe3864099bf "chan_pjsip.c", line=line@entry=2312, func=func@entry=0x7fe386409580 <__PRETTY_FUNCTION__.29394> "hangup_data_destroy") at astobj2.c:584
#16 0x00007fe3863fdcad in hangup_data_destroy (obj=0x7fe206717bc0) at chan_pjsip.c:2312
#17 0x0000558570da1310 in __ao2_ref (user_data=user_data@entry=0x7fe206717bc0, delta=delta@entry=-1, tag=tag@entry=0x7fe386409b29 "", file=file@entry=0x7fe3864099bf "chan_pjsip.c", line=line@entry=2354, func=func@entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:584
#18 0x0000558570da16bb in __ao2_cleanup_debug (obj=obj@entry=0x7fe206717bc0, tag=tag@entry=0x7fe386409b29 "", file=file@entry=0x7fe3864099bf "chan_pjsip.c", line=line@entry=2354, function=function@entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:657
#19 0x00007fe386402590 in hangup (data=0x7fe206717bc0) at chan_pjsip.c:2354
#20 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=tps@entry=0x5585723c4a30) at taskprocessor.c:970
#21 0x0000558570f51dd0 in execute_tasks (data=0x5585723c4a30) at threadpool.c:1322
#22 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=0x55857212d280) at taskprocessor.c:970
#23 0x0000558570f52304 in threadpool_execute (pool=0x55857212d760) at threadpool.c:351
#24 worker_active (worker=0x7fe3a8282340) at threadpool.c:1105
#25 worker_start (arg=arg@entry=0x7fe3a8282340) at threadpool.c:1024
#26 0x0000558570f5d949 in dummy_start (data=<optimized out>) at utils.c:1258
#27 0x00007fe4216ce494 in start_thread (arg=0x7fe3724f8700) at pthread_create.c:456
#28 0x00007fe420949acf in __libc_ifunc_impl_list (name=<optimized out>, array=0x7fe3724f8700, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:387
#29 0x0000000000000000 in ?? ()
{noformat}

After restarting the Asterisk, everything works fine. But we want to prevent this again in the future.

However, no idea what causes this problem and how to fix this if this happens again.
Comments:By: Asterisk Team (asteriskteam) 2019-06-05 09:15:24.799-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.
By: sungtae kim (pchero) 2019-06-05 09:16:41.546-0500

Backtrace files
By: Chris Savinovich (csavinovich) 2019-06-14 11:12:13.753-0500

Hi Sungtae, we don't support 15 anymore. Would it be possible that you replicate this issue in 16?
Thanks
Chris

By: Richard Mudgett (rmudgett) 2019-06-14 11:26:10.854-0500

Per the Asterisk versions page [1], the maintenance (bug fix) support for the Asterisk branch you are using has ended. For continued maintenance support please move to a supported branch of Asterisk. After testing with a supported branch, if you find this problem has not been resolved, please open a new issue against the latest version of that Asterisk branch.

Thanks!

[1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions


By: sungtae kim (pchero) 2019-06-18 07:46:27.647-0500

Hi Chris, Richard,

Thanks for your comments.
I will create a new ticket when we facing the same issue with the newest Asterisk. :)