Summary:	ASTERISK-24190: IMAP voicemail causes segfault
Reporter:	Nick Adams (Narkov)	Labels:
Date Opened:	2014-08-07 17:18:39	Date Closed:	2014-10-30 04:13:24
Priority:	Critical	Regression?
Status:	Closed/Complete	Components:	Applications/app_voicemail Applications/app_voicemail/IMAP
Versions:	11.11.0	Frequency of Occurrence	Constant
Related Issues:
Environment:	Ubuntu 14.04.1 Linux 3.13.0-32-generic x86_64	Attachments:	( 0) backtrace.txt ( 1) issue24190_dynamic_msgarray2.patch ( 2) myDebugLog
Description:	When using IMAP storage for Voicemail, I can reliably cause Asterisk to segfault. I noticed a number of random segfaults over the past few days and on further investigation, was able to reproduce it by running "voicemail show users for default". That produces the following segfault: {quote}Aug  8 07:50:10 timmy kernel: [1025781.678733] asterisk[20482]: segfault at 10d ip 00007fe97c22ab78 sp 00007fe8a0d09900 error 4 in libc-client.so.2007e.0[7fe97c1fb000+101000]{quote} This happens on multiple boxes so I've ruled out memory corruption or some other nastiness. voicemail.conf is at defaults except for: {quote}imapserver=mail.mydomain.com imapport=993 imapgreetings=yes greetingsfolder=INBOX.Voicemail imapparentfolder=INBOX. imapfolder=INBOX.Voicemail imapflags=ssl [default] 0400000000 => 0400000000,Some User,,,attach=yes|imapuser=user@mydomain.com|imappassword=PASSWORD {quote} The relevant INBOX.Voicemail IMAP folder has about 300 voicemails in it. I think this may be related to users having forwarded messages out of those folders resulting in additional IMAP flags being added - though I can't be sure.
Comments:	By: Nick Adams (Narkov) 2014-08-07 17:19:18.833-0500 Full debug log attached. By: Nick Adams (Narkov) 2014-08-07 17:32:29.909-0500 If I move the forwarded voicemail messages out of the IMAP folder, Asterisk seems to be able to run that command without any segfault. It seems there is an issue parsing the IMAP flags/keywords. This particular IMAP server is Dovecot 2.2.9. By: Matt Jordan (mjordan) 2014-08-07 19:15:55.206-0500 Thank you for your bug report. In order to move your issue forward, we require a backtrace[1] from the core file produced after the crash. Also, be sure you have DONT_OPTIMIZE enabled in menuselect within the Compiler Flags section, then: make install After enabling, reproduce the crash, and then execute the backtrace[1] instructions. When complete, attach that file to this issue report. [1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace By: Walter Doekes (wdoekes) 2014-08-08 05:48:15.656-0500 {noformat} [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: saving mailbox message number 266 as message 253. Interactive set to 0 [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: saving mailbox message number 267 as message 254. Interactive set to 0 [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: saving mailbox message number 268 as message 255. Interactive set to 0 [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: saving mailbox message number 269 as message 256. Interactive set to 0 [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: saving mailbox message number 270 as message 257. Interactive set to 0 [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: support@mydomain.com not found in vmstates [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: support@mydomain.com not found in vmstates [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: support@mydomain.com not found in vmstates [Aug  8 07:52:29] DEBUG[21051] app_voicemail.c: support@mydomain.com not found in vmstates {noformat} {code} #define VMSTATE_MAX_MSG_ARRAY 256 ...        long msgArray[VMSTATE_MAX_MSG_ARRAY]; ...        ast_debug(3, "saving mailbox message number %lu as message %d. Interactive set to %d\n", number, vms->vmArrayIndex, vms->interactive);        vms->msgArray[vms->vmArrayIndex++] = number; {code} Memory corruption.. By: Rusty Newton (rnewton) 2014-08-08 12:49:23.942-0500 [~Narkov] Regarding [~wdoekes] comment that probably means we will need [Valgrind|https://wiki.asterisk.org/wiki/display/AST/Valgrind] or MALLOC_DEBUG output to further investigate. Instructions at the link. By: Walter Doekes (wdoekes) 2014-08-08 13:12:57.213-0500 Well.. in this case the cause of corruption was quite obvious: unchecked bounds of myArray. I attached a patch which might fix it. Nick: if you could try the patch and report back. I'm not sure about proper freeing of the acquired memory though. The passing of the vms parameters in app_voicemail are non-standard, to put it mildly. (And the fact that there's both a vm_state and a vmstate struct doesn't make things better.) By: Nick Adams (Narkov) 2014-08-09 00:25:19.274-0500 I've applied the patch and cannot reproduce the segfault. This looks to have fixed it! Did you need any additional debug output to validate the fix? Thanks very much for the prompt resolution Walter and Rusty. By: Walter Doekes (wdoekes) 2014-08-09 01:55:15.817-0500 I worry about (more) memory leaking after the patch. Unless another developer can say for sure where the freeing should be done, it'd be nice to get some MALLOC_DEBUG output. https://wiki.asterisk.org/wiki/display/AST/MALLOC_DEBUG+Compiler+Flag Primarily the {{memory atexit list}} after running a few {{voicemail show users for default}}. By: Nick Adams (Narkov) 2014-08-09 02:12:35.395-0500 The mmlog output seems rather sparse: {quote}1407567993 - New session{quote} I ran a number of {{voicemail show users for default}} and the output of {{memory show summary}} remained static after each iteration: {quote}92777 bytes in         22 allocations in file app_voicemail.c{quote} Does that give you the information you need? By: Walter Doekes (wdoekes) 2014-08-11 03:34:51.316-0500 And if you then {{module unload app_voicemail.so}}? By: Nick Adams (Narkov) 2014-08-21 23:35:04.880-0500 I've done {{module unload app_voicemail.so}} and {{module load app_voicemail.so}} a few times without issue. Do you need me to check anything else? By: Nick Adams (Narkov) 2014-10-28 17:50:32.127-0500 I've now run this patch in production for a number of months and the issue hasn't reoccured. Is it possible to get this merged and released? By: Walter Doekes (wdoekes) 2014-10-29 03:10:48.641-0500 Ok. Thanks for bumping this and the feedback. I'll put it up for review. Perhaps someone else can shed some light on the odd memory management.