Summary:	ASTERISK-25504: Asterisk with pjsip driver crashes codec related?
Reporter:	Carl Fortin (phonefxg)	Labels:
Date Opened:	2015-10-29 04:09:41	Date Closed:	2015-11-30 13:00:31.000-0600
Priority:	Major	Regression?
Status:	Closed/Complete	Components:
Versions:	13.5.0	Frequency of Occurrence
Related Issues:
Environment:	Asterisk Realtime 13.5 PJSIP Driver mysql Ver 14.14 pjproject 2.3 spandsp 0.0.6 jansson 2.7 CentOS 6.6 64 bits on Vmware Number of endpoints : > 700 Numbers of calls : 3000/day Our Hardware: Phones : Cisco SPA514G FW: 7.5.7 ATA : Audiocodes MP124 T1 : Mediatrix 3532 ISDN to SIP gateway CPU : Quadcore Intel(R) Xeon(R) CPU E5-2650 RAM : 3 GB	Attachments:	( 0) backtrace_2.txt ( 1) backtrace.txt ( 2) Endpoint_detail.txt ( 3) High_CPU.txt ( 4) My_debug_log.txt
Description:	We are running asterisk 13.5 pjsip in a production environment and I had 2 crashes yesterday.  It happened shortly after enabling the g722 codec to a couples of phones in our ps_endpoints database. I'm not sure if the problem is coming from this modification but the same phone with the g722 codec made asterisk crashed 2 times. Can someone look and my backtrace and my full debug log to see if it could be related to a codec negotiation?
Comments:	By: Asterisk Team (asteriskteam) 2015-10-29 04:09:43.229-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Carl Fortin (phonefxg) 2015-10-29 04:11:18.369-0500 My backtrace and full debug log By: Carl Fortin (phonefxg) 2015-11-04 13:21:11.040-0600 Endpoint details By: Rusty Newton (rnewton) 2015-11-04 17:47:29.290-0600 There are relevant fixes in 13.6.0. Please test with that version of Asterisk and report back. If the problem still occurs in 13.6.0 then please provide refs.txt generated with these instructions: https://wiki.asterisk.org/wiki/display/AST/Reference+Count+Debugging In addition include a new debug log that includes the DEBUG log channel turned up to 5 (along with VERBOSE turned up to 5). https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information By: Carl Fortin (phonefxg) 2015-11-04 18:57:10.738-0600 I'm also wondering about my pjsip version (2.3) . Should I update to the latest one or it makes no difference? I will test with version 13.6. It could take a while, since the problem seems to happen randomly. By: Carl Fortin (phonefxg) 2015-11-05 11:12:12.602-0600 I can confirm that this has nothing to do with codecs because I just got another crash. I will update to 13.6. See my backtrace_2 in case it is something else. By: Carl Fortin (phonefxg) 2015-11-05 11:12:31.507-0600 Backtrace By: Rusty Newton (rnewton) 2015-11-06 07:36:53.009-0600 Crash looks the same. Yes please update to pjsip 2.4.5 and Asterisk 13.6. Thank you. By: Carl Fortin (phonefxg) 2015-11-10 19:39:16.455-0600 I have updated Asterisk to 13.6 and pjsip to 2.4.5. I have also enabled DONT OPTIMIZE,MALLOC_DEBUG under Compiler Flags. I did not enabled REF_DEBUG for now because the refs file created in  /var/log/asterisk was getting huge as more calls were made (about 600 MB). The file would have filled the harddisk in a day. I will test version 13.6 for a while and will activate REF_DEBUG if the bug is back. I will report back in a few days. By: Carl Fortin (phonefxg) 2015-11-23 07:38:59.555-0600 I had asterisk running over a week without problem, however I had a High CPU and asterisk unable to stop the service shortly after setting  remove_existing = yes in my ps_aors database. The high CPU happened when Asterisk received 24 registrations update at once from our ATA. The CPU was high from 3:43 AM until a reboot at 12:30 PM . See my full debug log. This bug might not be related to this issue because there is no core crash. You can also see a Broken pipe in my log: Nov 21 03:43:11 ERROR[16486] res_pjsip_registrar.c: Failed to update contact 'sip:U_5444@10.188.34.26:5061' expiration time to 3600 seconds. Nov 21 03:43:49 ERROR[17349] utils.c: write() returned error: Broken pipe Could it be a deadlock problem? By: Carl Fortin (phonefxg) 2015-11-23 07:41:56.033-0600 High CPU upon registration update. By: Carl Fortin (phonefxg) 2015-11-23 07:48:08.187-0600 I have also notice a broken pipe when I use the command to show memory allocations: memory show allocations By: Rusty Newton (rnewton) 2015-11-23 18:32:10.901-0600 Let's not confuse this issue. The high CPU issue sounds different enough to split it out. Perhaps sometimes the high CPU leads to a crash.. but that is yet to be seen. For now, if the crash occurs again report back on this issue. Otherwise please create a new issue for the high CPU problem and attach all relevant debug to that new issue. Most preferably any details that may lead to reproduction of the issue. By: Carl Fortin (phonefxg) 2015-11-25 19:52:00.287-0600 Yeah it looks like something else, and the problem did not reappear. I will create an issue if the problem of high cpu comes back. I guess you can can close it as we did not have any core crash. By: Rusty Newton (rnewton) 2015-11-30 13:00:31.781-0600 Sounds good. Thanks.