[Home]

Summary:ASTERISK-17439: Crash in 'core show locks'
Reporter:Kirill Katsnelson (kkm)Labels:
Date Opened:2011-02-19 20:52:07.000-0600Date Closed:2012-09-05 08:30:49
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
duplicatesASTERISK-20090 Crash when running 'core show locks' with BETTER_BACKTRACES enabled
Environment:Attachments:( 0) ASTERISK-17439-1.6.1.0-patch.txt
( 1) ASTERISK-17439-trunk-patch.txt
Description:Typed 'core show locks', and Asterisk dumped its core on me.

****** ADDITIONAL INFORMATION ******

(gdb) bt
#0  0x00007f1c79141c70 in strlen () from /lib/libc.so.6
#1  0x00007f1c7910a69a in vfprintf () from /lib/libc.so.6
#2  0x00007f1c791323ca in vsnprintf () from /lib/libc.so.6
#3  0x0000000000556e93 in __ast_str_helper (buf=0x7f1c55172818, max_len=0, append=1, fmt=0x5d10a9 "\t%s\n", ap=0x7f1c55172610)
   at strings.c:72
#4  0x0000000000568fe2 in ast_str_append_va (buf=0x7f1c55172818, max_len=0, fmt=0x5d10a9 "\t%s\n", ap=0x7f1c55172610)
   at /root/dist/asterisk.1.8.72/include/asterisk/strings.h:787
ASTERISK-1  0x00000000005692b2 in ast_str_append (buf=0x7f1c55172818, max_len=0, fmt=0x5d10a9 "\t%s\n")
   at /root/dist/asterisk.1.8.72/include/asterisk/strings.h:859
ASTERISK-2  0x000000000056a6a4 in append_backtrace_information (str=0x7f1c55172818, bt=0x32403e8) at utils.c:746
ASTERISK-3  0x000000000056a8a7 in append_lock_information (str=0x7f1c55172818, lock_info=0x322df60, i=1) at utils.c:772
ASTERISK-4  0x000000000056ac6e in handle_show_locks (e=0x8372e0, cmd=-4, a=0x7f1c55172a80) at utils.c:883
ASTERISK-5  0x000000000048febf in ast_cli_command_full (uid=0, gid=0, fd=21, s=0x7f1c55172b70 "core show locks") at cli.c:2474
ASTERISK-6 0x0000000000490068 in ast_cli_command_multiple_full (uid=0, gid=0, fd=21, size=16, s=0x7f1c55172de0 "core show locks")
   at cli.c:2497
ASTERISK-7 0x000000000043cf1c in netconsole (vconsole=0x86fc20) at asterisk.c:1266
ASTERISK-8 0x000000000056aed5 in dummy_start (data=0x2d222f0) at utils.c:971
ASTERISK-9 0x00007f1c7894b3ba in start_thread () from /lib/libpthread.so.0
ASTERISK-10 0x00007f1c791a702d in clone () from /lib/libc.so.6
ASTERISK-11 0x0000000000000000 in ?? ()
(gdb) fr 6
ASTERISK-2  0x000000000056a6a4 in append_backtrace_information (str=0x7f1c55172818, bt=0x32403e8) at utils.c:746
746                             ast_str_append(str, 0, "\t%s\n", symbols[frame_iterator]);
(gdb) p frame_iterator
$10 = 20
(gdb) p symbols[20]
$11 = 0x6962732f7273752f <Address 0x6962732f7273752f out of bounds>
(gdb) p symbols[19]
$12 = 0x381965e "/usr/sbin/asterisk [0x56aed5]"
(gdb)
Comments:By: Kirill Katsnelson (kkm) 2011-02-19 20:55:07.000-0600

Ah, and this:

(gdb) p *bt
$13 = {addresses = {0x4ea973, 0x4e37d1, 0x445a24, 0x4f95de, 0x7f1c58112ee9, 0x7f1c58113800, 0x525bec, 0x7f1c621a78db,
   0x7f1c621a7fb9, 0x4738f9, 0x47568a, 0x7f1c57cf8588, 0x7f1c57cfdca0, 0x7f1c57d00799, 0x503e51, 0x50de18, 0x50f8e1, 0x5103c1,
   0x512106, 0x56aed5, 0x7f1c7894b3ba, 0x7f1c791a702d, 0x7f1c7894b3ba, 0x7f1c791a702d, 0x7f1c791a702d, 0x512106, 0x56aed5,
   0x7f1c7894b3ba, 0x7f1c791a702d, 0x56aed5, 0x7f1c7894b3ba, 0x7f1c791a702d}, num_frames = 22, alloced = 0}
By: Kirill Katsnelson (kkm) 2011-02-19 21:03:46.000-0600

0x6962732f7273752f = "/usr/sbi" in reverse. Buffer overflow.
By: David Woolley (davidw) 2012-01-30 10:09:02.955-0600

I thought I'd already reported this one, with an analysis, but I haven't found it yet.

As I remember it, the problem here is that num_frames is read twice, once to decide how many frames for which to resolve the addresses (parameter to bactktrace_symbols) and once to determine the number of frames to actually output (following for loop).  If the backtrace is actually being populated during this process, the second pass makes unsafe assumptions about the result of the first pass.

Incidentally, I'm not sure how a bug that only presents when trying to debug another problem can be said to affect the majority of users, so this can only really be "minor" severity!

I note that the original bug report didn't identify a version.  It is actually presenting for me on 1.6.1.0 and I haven't finished checking whether it has got fixed in passing in a supported version.
By: David Woolley (davidw) 2012-01-30 10:26:55.167-0600

I cannot see anything that would obviously fix this in 1.6.2 branch, 1.8 branch or trunk.  E.g. trunk (blame output):

118173 mmichelson
298961   tilghman       if ((symbols = ast_bt_get_symbols(bt->addresses, bt->num_frames))) {
118173 mmichelson               int frame_iterator;
118173 mmichelson              
118173 mmichelson               for (frame_iterator = 0; frame_iterator < bt->num_frames; ++frame_iterator) {

By: David Woolley (davidw) 2012-01-30 11:05:59.155-0600

Adding patch against our version, trunk patch to follow.  Not tested yet.
By: David Woolley (davidw) 2012-01-30 11:14:27.865-0600

Adding trunk patch.  This should also cover 1.6.2 and 1.8.  This hasn't been checked for compilablity.
By: Kirill Katsnelson (kkm) 2012-01-31 17:31:38.605-0600

I'll give it a test when we have fewer calls at night. Today or tomorrow.
By: Matt Jordan (mjordan) 2012-09-05 08:30:33.094-0500

We independently ran into this same issue in the Asterisk Test Suite.  Unfortunately, we missed the fact that this issue was opened and ended up writing a very similar patch to David's that got committed just recently.  (Given the amount of test failures this issue ended up causing in the test suite, I really, really wish I had written a better JIRA query and found this issue!  I may end up having to make components for the better known compile time options)

I'm going to go ahead and close this issue out to ASTERISK-20090 as a result.