Test each and every modr/m value

[superfury]

I see that not all modr/m registers and r/m abd sib bits are validated? Can it be extended to test each set of bits for validity as well as special combinations with the modr/m and sib bytes(32-bit mode)? so testing all mod+reg, mod+rm, scale, index and all base values for valid results(and exceptions to the default rules on them, like with (e)sp/(e)bp)?

[superfury]

I've found some problems with the 32-bit ModR/M value, in particular problems with handling and calculating the offset from the 32-bit displacement-only mode using the SIB byte. It was doubling the displacement by applying it at both the index and base calculations, adding them together afterwards. I've also reordered the whole SIB memory operand calculations: the base now uses either a register or 0(for EBP with MOD=0), while the index is followed and added with the displacement, while ESP as an index just gives the displacement instead(not adding anything to the opcode. So it goes from the earlier DISPLACEMENT+0 to DISPLACEMENT(instead of calculating DISPLACEMENT*2 by adding it twice) in the EBP+MOD=0 case, while BASE+DISPLACEMENT is now generated for the pure displacement mode with index ESP, all cases should now work properly).

[superfury]

It's continuing without crashing now, with that latest modr/m fix! :D

[superfury]

I've found even more ModR/M offset problems: the 16-bit memory writes through the BIU were using the ModR/M parameter object(implicit C object to uint_32 conversion) instead of the proper 16/32-bit address that's stored in a variable within it. The same applies to 16-bit and 32-bit modr/m memory checking(protection checks).

[superfury]

Another not-so-obvious bug found: EBP as an index(specified using SIB index bits) defaults to DS, not SS. That was even a bug in the capstone disassembler. Maybe even NASM(and it's optimization of EBP+EBP(1) using it incorrectly, which uses SS by default, but the 'optimized' form(EBP2) uses SS instead) still. That will cause trouble with segmented memory where DS!=SS.

[barotto]

Thanks for the heads up. I've added a better tester for 32-bit addressing (ModRM + SIB). It's still missing a test for the default segment tho.

[superfury]

Afaik there's still no default segment checks for the various memory operands? ModR/M seems to be running fine.