| Summary: | ASTERISK-19889: Asterisk crashes due to memory corruption | ||||
| Reporter: | Scottie Winn (nhutasterisk) | Labels: | |||
| Date Opened: | 2012-05-20 15:15:37 | Date Closed: | 2012-09-19 20:27:49 | ||
| Priority: | Critical | Regression? | |||
| Status: | Closed/Complete | Components: | |||
| Versions: | 10.4.0 | Frequency of Occurrence | Frequent | ||
| Related Issues: |
| ||||
| Environment: | Centos 5.8 Final | Attachments: | ( 0) backtrace6.txt ( 1) valgrind.txt ( 2) valgrind10_4.txt ( 3) valgrind10_4-leak-check-full.txt | ||
| Description: | Seems to be at the manager API | ||||
| Comments: | By: Scottie Winn (nhutasterisk) 2012-05-20 15:20:25.673-0500 Backtrace By: Matt Jordan (mjordan) 2012-05-21 08:08:40.412-0500 Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks! By: Scottie Winn (nhutasterisk) 2012-05-21 21:59:28.490-0500 The backtrace6.txt was for 10.4 The valgrind.txt is for 10.5.0-rc1. Please let me know if you want me to reverse back to 10.4. Thanks By: Richard Mudgett (rmudgett) 2012-05-22 10:41:33.242-0500 FYI You need to send the issue back otherwise it could fall off the radar waiting for you to respond. By: Scottie Winn (nhutasterisk) 2012-05-22 11:24:08.922-0500 I will produce valgrind file for asterisk 10.4 tonight and upload it. Thanks By: Scottie Winn (nhutasterisk) 2012-05-22 20:24:40.903-0500 These two files were generated under 10.4. Thanks By: Rusty Newton (rnewton) 2012-05-23 17:13:02.194-0500 Scottie, thanks for the data. Can you additionally provide a detailed description of how the system is used (whats happening on the system when it crashes? queues with lots of inbound calls? AMI?, AGI?, what calls are being made? database connections? etc) and a full log with at least DEBUG 5 up to the time of the crash. By: Scottie Winn (nhutasterisk) 2012-05-23 19:36:16.201-0500 Hi Rusty! The crashes happened even though I eliminated AMI connections. There were about 40 queues and each of them had less than 5 callers. We didn't have any crashes when we're using 1.6.2.X. I looked at the log 5 minutes before the crash and the only database connections were to astdb.sqlite and there were no AGI called. I reversed back to 1.6.2.24 last night and so far asterisk hasn't crashed yet. Usually asterisk would crash at least once by now when we're using 10.4 or 10.5-rc1. Most of the time when asterisk crashed, it didn't leave any segfault messages in the message log, but it generated core files in the tmp directory. Please let me know if you still want me to upload the log file for asterisk. Thanks. By: Matt Jordan (mjordan) 2012-05-24 07:33:45.052-0500 Scottie: I know you've moved back to 1.6.2.X - for any of the valgrind logs you uploaded, did any capture the output from an instance of Asterisk that crashed? While there are a number of warnings in those outputs, none appear to be indicative of something that would cause the problems you had. By: Scottie Winn (nhutasterisk) 2012-05-24 10:23:02.670-0500 Hi Matt! All valgrind10_4 files were captured from Asterisk 10.4 and valgrind.txt were from 10.5-rc1. While running valgrind on Asterisk 10.4, Asterisk didn't crash because I ran it for a short period of time. It's on a production server so I couldn't afford to leave it running for long. We are on day 2 of 1.6.2.24 and haven't crashed yet. Please let me know if you need me to do anything more. Thanks. By: Rusty Newton (rnewton) 2012-05-25 09:54:33.915-0500 Scottie, as Matt Jordan said "While there are a number of warnings in those outputs, none appear to be indicative of something that would cause the problems you had." You'll need to get Valgrind output during the crash for a developer to be able to find anything useful in the output. Otherwise we won't have a way to investigate this issue, unless you can provide exact instructions for reproduction. By: Matt Jordan (mjordan) 2012-06-21 12:38:59.150-0500 Scottie: If you're using app_voicemail, do you mind trying the patch on ASTERISK-19923? A memory corruption has been identified in that application, and the patch on that issue has fixed it for some users experiencing memory corruption crashes. The patch was originally written for 1.8, but - at least for me - applied cleanly to a fresh checkout of Asterisk 10 as well. Thanks Matt By: Rusty Newton (rnewton) 2012-07-23 19:03:37.709-0500 Scottie, have you had a chance to try this patch? Are you still experiencing the issue? By: Scottie Winn (nhutasterisk) 2012-08-17 10:02:01.892-0500 I will install 10.7 on our production server and report if the bug is still there. Thanks By: Rusty Newton (rnewton) 2012-09-19 20:27:40.848-0500 Suspended due to lack of activity. Please request a bug marshal in #asterisk-bugs on the IRC network irc.freenode.net to reopen the issue should you have the additional information requested. Further information can be found at https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines | ||||