I have reverse engineered some control paths of an old MS-DOS tool from Broadcom which came as a UPX compressed linear executable prepended with Protected Mode extender and required tools such as IDA 4.1, IDA 5.0, to be reverse engineered. This post is a collection of notes explaining how to get from binary to disassembled and decompiled code for applications packaged in a similar way.

MS-DOS protected mode extender

The following is what file shows on the tool:

$ file B57UDIAG.EXE 
B57UDIAG.EXE: MS-DOS executable, LE executable for MS-DOS, PMODE/W DOS extender, UPX compressed

After going through the first compressed layer,

Expand code - Upx decompression 
$ upx -d B57UDIAG.EXE 
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2023
UPX 4.0.2       Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 30th 2023

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    800005 <-    243351   30.42%    watcom/le    B57UDIAG.EXE

Unpacked 1 file.
$ file B57UDIAG.EXE 
B57UDIAG.EXE: MS-DOS executable, LE executable for MS-DOS, PMODE/W DOS extender


we get a Linear Executable with PMODE/W DOS extender. PMODE/W is a stub prepended to the original binary which implements DOS Protected Mode Interface and enables protected mode programs to run in MS-DOS environment, with access to all system memory available. The extender takes care of the complexity coming from supporting protected mode, such as descriptor tables and memory management. Application code in B57UDIAG.EXE does not seem to be compressed by PMODE/W (strings does return what we would expect to see from the Broadcom tool), however IDA does not recognize the Linear Executable and disassembles only PMODE/W stub, so the two need to be split up.

As a side note, if application code was compressed by PMODE/W, a tool called PMWUNLIT.EXE (version v1.20, mentioned in multiple places online such as [1], [2], [3]) can be used for the extraction.

Expand code - PMWUNLIT.EXE sha256 
sha256 of PMWUNLIT.EXE: dcbdaf0665b4d252620e0ff786a05e7c0ca9d9ee42986fe1d942ef0359db68be


PMWUNLIT.EXE runs in MS-DOS, but it should be totally doable to build a modern alternative based on PMODE/W source code (as of July 2023, PMODE/W is released under MIT license).

In order to separate PMODE/W stub and application code, DOS/32 extender is distrubuted with a tool which is capable of removing the stub, also working with PMODE/W. The tool is called sb.exe and is packaged in dos32a-912-bin.zip, of which I could find a copy only on Internet Archive. Running sb.exe on the upx unpacked B57UDIAG.EXE file results in the extraction of the LE binary:

C:\>sb.exe /U B57UDIAG.EXE
SB/32A -- Bind Utility version 9.1.2
Copyright (C) 1996-2006 by Narech K.

Application Name:   "B57UDIAG.EXE"
Application Size:   3165490 bytes
Application Type:   PMODE/W DOS Extender bound to
                    LE-style file format Linear Executable

  Unbinding file:   "B57UDIAG.EXE"
Destination file:   "B57UDIAG.le"
Destination size:   3153970 bytes (99.6%)

Disassembling LE binaries

We now have a linear executable binary and we need a disassembler which is capable of dissecting this format. All pointers I gathered onlined suggested that IDA 4.1 is the last free version of IDA disassembler which can open LEs. The alternative is to use IDA Pro. IDA 4.1 is a console application for Windows 32 bits and can be executed via wine (wineconsole). Reverse engineering can however take place in a more modern environment, as IDA 4.1 serves just as an extractor to build the disassembler database (idb file). IDA Freeware 5.0 is the latest version capable of opening the idb file generated by IDA 4.1 and it is the one I extensively used to analyze B57UDIAG.EXE.

The alternative to IDA is using Ghidra with custom LE loaders, such as ghidra-lx-loader. For my extercise on Broadcom tool, I used both. Admittedly, I found IDA to be more reliable in disassembling application code.

The availability of the two versions of IDA mentioned above is as follows:

  • IDA Freeware 5.0 can be sourced from ScummVM, who got permissions to re-distribute the binary.
  • IDA 4.1 can be found online from repositories of old MS-DOS tools

Once the Linear Executable has been split from PMODE/W stub, it can be simply loaded into IDA 4.1 to generate the database.

The idb file can then be fed into IDA Freeware 5.0 in a modern Windows environment.