Summary: | ASTERISK-28437: Taskprocessor doesn't process the tasks | ||
Reporter: | sungtae kim (pchero) | Labels: | pjsip |
Date Opened: | 2019-06-05 09:15:23 | Date Closed: | 2019-06-18 07:46:27 |
Priority: | Minor | Regression? | |
Status: | Closed/Complete | Components: | Core/Stasis |
Versions: | 15.7.2 | Frequency of Occurrence | One Time |
Related Issues: | |||
Environment: | Attachments: | ( 0) core-asterisk-running-2019-06-05T13-01-31+0000-brief.txt ( 1) core-asterisk-running-2019-06-05T13-01-31+0000-full.txt ( 2) core-asterisk-running-2019-06-05T13-01-31+0000-locks.txt ( 3) core-asterisk-running-2019-06-05T13-01-31+0000-thread1.txt | |
Description: | Hi Asterisk team,
One of Asterisk's taskprocessor doesn't process the tasks. Seems like stuck some deadlock, but couldn't figure out why. {noformat} CLI> core show taskprocessors Processor Processed In Queue Max Depth Low water High water ast_msg_queue 0 0 0 450 500 CCSS_core 0 0 0 450 500 dns_system_resolver_tp 0 0 0 450 500 hep_queue_tp 0 0 0 450 500 iax2_transmit 0 0 0 450 500 pjsip/default-0000000a 1207948 0 2 450 500 pjsip/default-0000000b 1207946 0 1 450 500 pjsip/default-0000000c 1207949 0 2 450 500 pjsip/default-0000000d 1207948 0 2 450 500 pjsip/default-0000000e 1207949 0 2 450 500 pjsip/default-0000000f 1207950 0 2 450 500 pjsip/default-00000010 1207947 0 2 450 500 pjsip/default-00000011 1207945 0 1 450 500 pjsip/distributor-0000001e 267542 0 4 450 500 pjsip/distributor-0000001f 272156 0 6 450 500 pjsip/distributor-00000020 264634 0 5 450 500 pjsip/distributor-00000021 268282 0 5 450 500 pjsip/distributor-00000022 270238 0 7 450 500 pjsip/distributor-00000023 270059 0 4 450 500 pjsip/distributor-00000024 270525 0 6 450 500 pjsip/distributor-00000025 270730 0 4 450 500 pjsip/distributor-00000026 272161 0 5 450 500 pjsip/distributor-00000027 273404 0 6 450 500 pjsip/distributor-00000028 274615 0 9 450 500 pjsip/distributor-00000029 273141 0 4 450 500 pjsip/distributor-0000002a 269424 0 7 450 500 pjsip/distributor-0000002b 269376 0 5 450 500 pjsip/distributor-0000002c 269265 0 6 450 500 pjsip/distributor-0000002d 270574 0 4 450 500 pjsip/distributor-0000002e 269473 0 4 450 500 pjsip/distributor-0000002f 275079 0 5 450 500 pjsip/distributor-00000030 270892 0 5 450 500 pjsip/distributor-00000031 274360 0 4 450 500 pjsip/distributor-00000032 274523 0 4 450 500 pjsip/distributor-00000033 268106 0 7 450 500 pjsip/distributor-00000034 269143 0 6 450 500 pjsip/distributor-00000035 273046 0 4 450 500 pjsip/distributor-00000036 270502 1354 5 450 500 pjsip/distributor-00000037 274427 0 6 450 500 pjsip/distributor-00000038 268568 0 5 450 500 pjsip/distributor-00000039 271983 0 4 450 500 pjsip/distributor-0000003a 271705 0 4 450 500 pjsip/distributor-0000003b 269543 0 5 450 500 pjsip/distributor-0000003c 271750 0 5 450 500 ... 289 taskprocessors {noformat} I'm doubting about this part, but not clear how to digging in deeper. {noformat} Thread 76 (LWP 7723): #0 0x00007fe4216d6f5c in ?? () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:107 from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe40cea1460 in ?? () from /usr/lib/asterisk/modules/res_stasis.so #2 0x0000558571255db2 in __ast_string_field_empty_buffer () #3 0x00007fe4216d487c in pthread_cond_signal@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:117 #4 0x0000558570e8deb8 in __ast_cond_signal (filename=<optimized out>, lineno=<optimized out>, func=<optimized out>, cond_name=<optimized out>, cond=<optimized out>) at lock.c:508 #5 0x00007fe40ce9c5da in exec_command_on_condition (can_exec_fn=0x0, data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:273 #6 exec_command (data_destructor=<optimized out>, data=<optimized out>, command_fn=<optimized out>, control=0x7fe207c56520) at stasis/control.c:283 #7 stasis_app_send_command_async (control=0x7fe207c56520, command_fn=<optimized out>, data=<optimized out>, data_destructor=<optimized out>) at stasis/control.c:843 #8 0x00007fe40ce9d43b in dial_bridge_after_cb (data=0x7fe207c56520, chan=<optimized out>) at stasis/control.c:941 #9 dial_bridge_after_cb_failed (reason=<optimized out>, data=0x7fe207c56520) at stasis/control.c:951 #10 0x0000558570dbbdf8 in after_bridge_cb_failed (node=0x7fe2b6c95630) at bridge_after.c:70 #11 after_bridge_cb_run_discard (reason=AST_BRIDGE_AFTER_CB_REASON_DESTROY, after_bridge=0x7fe206a3bab0) at bridge_after.c:99 #12 after_bridge_cb_destroy (data=0x7fe206a3bab0) at bridge_after.c:117 #13 0x0000558570e3e60c in ast_datastore_free (datastore=datastore@entry=0x7fe207638580) at datastore.c:68 #14 0x0000558570dffa56 in ast_channel_destructor (obj=0x7fe185b2ffd0) at channel.c:2255 #15 0x0000558570da1310 in __ao2_ref (user_data=0x7fe185b2ffd0, delta=delta@entry=-1, tag=tag@entry=0x7fe386409b29 "", file=file@entry=0x7fe3864099bf "chan_pjsip.c", line=line@entry=2312, func=func@entry=0x7fe386409580 <__PRETTY_FUNCTION__.29394> "hangup_data_destroy") at astobj2.c:584 #16 0x00007fe3863fdcad in hangup_data_destroy (obj=0x7fe206717bc0) at chan_pjsip.c:2312 #17 0x0000558570da1310 in __ao2_ref (user_data=user_data@entry=0x7fe206717bc0, delta=delta@entry=-1, tag=tag@entry=0x7fe386409b29 "", file=file@entry=0x7fe3864099bf "chan_pjsip.c", line=line@entry=2354, func=func@entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:584 #18 0x0000558570da16bb in __ao2_cleanup_debug (obj=obj@entry=0x7fe206717bc0, tag=tag@entry=0x7fe386409b29 "", file=file@entry=0x7fe3864099bf "chan_pjsip.c", line=line@entry=2354, function=function@entry=0x7fe3864095d1 <__PRETTY_FUNCTION__.29416> "hangup") at astobj2.c:657 #19 0x00007fe386402590 in hangup (data=0x7fe206717bc0) at chan_pjsip.c:2354 #20 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=tps@entry=0x5585723c4a30) at taskprocessor.c:970 #21 0x0000558570f51dd0 in execute_tasks (data=0x5585723c4a30) at threadpool.c:1322 #22 0x0000558570f4ad28 in ast_taskprocessor_execute (tps=0x55857212d280) at taskprocessor.c:970 #23 0x0000558570f52304 in threadpool_execute (pool=0x55857212d760) at threadpool.c:351 #24 worker_active (worker=0x7fe3a8282340) at threadpool.c:1105 #25 worker_start (arg=arg@entry=0x7fe3a8282340) at threadpool.c:1024 #26 0x0000558570f5d949 in dummy_start (data=<optimized out>) at utils.c:1258 #27 0x00007fe4216ce494 in start_thread (arg=0x7fe3724f8700) at pthread_create.c:456 #28 0x00007fe420949acf in __libc_ifunc_impl_list (name=<optimized out>, array=0x7fe3724f8700, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:387 #29 0x0000000000000000 in ?? () {noformat} After restarting the Asterisk, everything works fine. But we want to prevent this again in the future. However, no idea what causes this problem and how to fix this if this happens again. | ||
Comments: | By: Asterisk Team (asteriskteam) 2019-06-05 09:15:24.799-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: sungtae kim (pchero) 2019-06-05 09:16:41.546-0500 Backtrace files By: Chris Savinovich (csavinovich) 2019-06-14 11:12:13.753-0500 Hi Sungtae, we don't support 15 anymore. Would it be possible that you replicate this issue in 16? Thanks Chris By: Richard Mudgett (rmudgett) 2019-06-14 11:26:10.854-0500 Per the Asterisk versions page [1], the maintenance (bug fix) support for the Asterisk branch you are using has ended. For continued maintenance support please move to a supported branch of Asterisk. After testing with a supported branch, if you find this problem has not been resolved, please open a new issue against the latest version of that Asterisk branch. Thanks! [1] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions By: sungtae kim (pchero) 2019-06-18 07:46:27.647-0500 Hi Chris, Richard, Thanks for your comments. I will create a new ticket when we facing the same issue with the newest Asterisk. :) |