[libopcodes] Decoder crash and incorrect results

[Nathan Jay]

Hello,

I have been testing the libopcodes library with randomized binary input in the 
form of 15-byte arrays. I have been using a 64-bit machine running Redhat 
Enterprise Linux 6. I have found that the library will segfault given certain 
inputs to print_insn_i386. Here is the code I am using to decode:

inst[] holds the bytes from left to right, zero indexed in a buffer.

INIT_DISASSEMBLE_INFO(disInfo, outf, (fprintf_ftype)fprintf);
disInfo.buffer = (bfd_byte*)(inst);
disInfo.buffer_length = nBytes;
disInfo.arch = bfd_arch_i386; 
disInfo.mach = bfd_mach_x86_64_intel_syntax;

print_insn_i386((bfd_vma)0, &disInfo);

Here are several inputs that cause segfaults:

8f eb 5c ec 72 4b 4e 3c 98 df e0 ef 1d 83 00
8f ac 65 ce 46 25 66 a4 81 97 92 ea b8 19 00
8f ae f7 ef 02 51 c2 50 bc 2d f3 fb 44 65 00
8f ce 47 ee 0e 2d 08 38 70 78 aa 73 29 4e 00
8f 2d 9e ed f2 12 7d 23 19 e3 49 50 8a 50 00

In addition, I have created a filter to avoid segfaults when using the 
libopcodes decoder. Here is the C code used to filter out crashing bytes:

inst[] holds the bytes from left to right, zero indexed.

if (inst[0] == (char)0x8f                                            &&
    (0x03 & (inst[1] >> 3)) == 1                                 &&
    (0x01 & (inst[2] >> 2)) == 1                                 &&
    ((0xf0 & inst[3]) == 0xe0 || (0xf0 & inst[3]) == 0xc0) &&
    ((0x0f & inst[3]) > 0x0b)) {

    return 1; //libopcodes would have crashed.
}

Additionally, these sequences can be prefixed as well and will cause a crash. 
(I have not yet produced a filter for this, but I may soon.)

Several of the decodings produced by libopcodes were also inaccurate according 
to the x86 documentation.

For example, the rep prefixes only affects string instructions (like scas, 
movs, etc.), yet these are shown in other cases, such as:

repz loopne 0x5b

Lock prefixes are also allowed even when they cause an Illegal Instruction 
signal. Lock prefixes should require a memory operand, but libopcodes will 
produce code like:

lock mov bh, 0x9f

If I can give you more information about the machine I am running on, or 
libopcodes version, let me know. If there is a nice way to get information 
about the libopcodes version, I'd be happy to pass it along. This version was 
compiled from a pull from the development source tree done on February 2nd with 
SHA:1b18aa1e79a0b343087d08075f117e821c33b930

All the best,
Nathan Jay