[GUIDE] Fixing the new version of SProtect

[devoltz]

SProtect: the variant where the block table sits on the MPQ header

(dwBlockTablePos = 0, which means "the block table starts at the header")

Hello.

This is a companion to the Completely deprotect S/SSProtect guide. That guide covers the S/SSProtect family in general. This one covers the specific variant where the map will not open in any editor and, if you follow the general guide, you end up staring at offset 0x200 with the "block table" apparently made of your own MPQ header.

Everything in the general guide still applies here: the same three edits fix the same three things. What is different is where the block table is, and that is the part that makes people give up. So let us do this properly.

Before we start

- What is the MPQ header?

• 32 bytes at offset 0x200 of a .w3x. It tells the game where the hash table and the block table are, and how big they are.

- What is the hash table?

• The lookup table. Each 16-byte entry holds two hashes of a file name and the index of that file in the block table.

- What is the block table?

• One 16-byte entry per file: where its data starts, how big it is packed, how big it is unpacked, and its flags.

- What is "dwBlockTablePos = 0" supposed to mean?

• Every position inside an MPQ is counted from the start of the MPQ header (the archive's origin), not from the start of the file. So a position of 0 points at the very first byte of the archive, which is exactly where the header itself is written. The block table's first two entries therefore share their 32 bytes with the header, and the header wins.

Tools you need

• HxD
• MPQ Helper (to decrypt/encrypt the tables)
• Hex Edit Macro (to mass-edit every 16th byte)
• Optional but much faster: Python 3 and the script at the bottom of this post

Step 1: Recognise it

Open the map in HxD and search for 4D 50 51 1A (Datatype: Hex-values). You should land on 0x200 and see something like this:

Code: Select all

000200  4D 50 51 1A 53 50 72 6F 74 65 63 74 00 00 0F 00   MPQ.SProtect....
000210  00 D3 53 0F 00 00 00 00 00 10 00 00 C5 0A 00 00   ..S.............

Read it field by field. All values are little-endian, so read them backwards:

• 0x200: 4D 50 51 1A = "MPQ\x1A", the signature.
• 0x204: 53 50 72 6F = "SPro". This is dwHeaderSize, overwritten with text.
• 0x208: 74 65 63 74 = "tect". This is dwArchiveSize, overwritten with text. (Together the two fields spell "SProtect", which is the signature of the protector.)
• 0x20C: 00 00 = wFormatVersion = 0 (MPQ v1). Fine.
• 0x20E: 0F 00 = wBlockSize = 15. Fine. Sector size = 512 << 15 = 16 MiB.
• 0x210: 00 D3 53 0F = dwHashTablePos = 257151744.
• 0x214: 00 00 00 00 = dwBlockTablePos = 0. This is the one that matters.
• 0x218: 00 10 00 00 = dwHashTableSize = 4096 entries.
• 0x21C: C5 0A 00 00 = dwBlockTableSize = 2757 entries.

Two quick sanity checks before you go further:

1. Does 0x200 + dwHashTablePos + dwHashTableSize * 16 land exactly on the end of the file? Here: (512 + 257151744) + 65536 = 257152256 + 65536 = 257217792 = file size. Yes. The hash table is the last thing in the file, which is why nothing looks wrong.
2. The block table needs 2757 * 16 = 44112 bytes and dwBlockTablePos says it starts at 0x200. That is inside the header. That is the whole trick.

Symptoms you will have seen: most editors and converters refuse the archive or show an empty file list (w3x2lni stops at 0%), and any script that parses the MPQ by hand blows up with a block index in the hundreds of millions.

Step 2: What the protection actually does

Three things, and you have to undo all three. The first is the location, the other two are inside the tables.

1) The block table is placed at position 0, under the MPQ header.

The protector moves the block table to the very beginning of the archive, so the 32-byte header is written on top of its first two entries. Those two entries are destroyed forever. The protector knows this, and makes sure no file uses block index 0 or 1. Nothing is lost. But every tool that reads the header as written will look for the block table somewhere it cannot be.

2) Every hash entry is poisoned.

A clean hash table entry looks like this, decrypted:

Code: Select all

hashA     hashB     locale  platform  blockIndex
XXXXXXXX  XXXXXXXX  0000    0000      00000ABC

In these maps it looks like this:

Code: Select all

hashA     hashB     locale  platform  blockIndex
XXXXXXXX  XXXXXXXX  0412    XX00      40000ABC

So: locale is a fixed 0x0412 instead of 0, platform is a random byte, and the block index has 0x40000000 forced into its top byte. A reader that uses the index as-is reads entry #1073742524 of a 2757-entry table. A reader that filters on locale = 0 finds nothing at all.

3) Every block entry claims to be single-unit.

Clean block entries have flags like 0x80000200 (exists + compressed). In these maps every single entry is 0x81000200, because the protector ORs in 0x01000000, which is MPQ_FILE_SINGLE_UNIT. That flag tells the reader "this file has no sector offset table, the whole block is one compressed unit". The data still has its sector offset table. So a reader that believes the flag tries to inflate the offset table as if it were file data, and every extracted file comes out corrupt.

If you fix only the header, or only the hash table, you will get a map that opens and produces garbage. Fix all three.

Step 3: Removing it

Method A: the script (recommended)

Save this as sprotect_fix.py, put it next to your map and run:

Code: Select all

python sprotect_fix.py MyMap.w3x

It writes MyMap_fixed.w3x. It fixes the header, un-poisons the hash table, clears the fake single-unit flag, blanks the two destroyed entries and moves the block table to the end of the file, where it belongs. The file data is never touched. No dependencies beyond Python 3.

Spoiler:

Code: Select all

#!/usr/bin/env python3
"""
sprotect_fix.py - removes the "SProtect / blockTablePos = 0" protection from a
Warcraft III .w3x map and writes a clean, unprotected .w3x.

No dependencies. Python 3.6+.

    python sprotect_fix.py MyMap.w3x
    python sprotect_fix.py MyMap.w3x -o MyMap_fixed.w3x

What the protection does (all three must be undone):

  1. MPQ header: dwHeaderSize + dwArchiveSize (offsets 0x204..0x20B) are
     overwritten with the ASCII text "SProtect", and dwBlockTablePos is set
     to 0 - which literally means "the block table starts at the MPQ header",
     so the first 32 bytes of the block table live under the header itself.

  2. Hash table: every occupied entry has locale = 0x0412 (not 0), a random
     platform value, and dwBlockIndex = 0x40000000 | real_index.

  3. Block table: every entry has 0x01000000 (MPQ_FILE_SINGLE_UNIT) forced
     into the flags, although the file data still carries a sector offset
     table. A reader that believes the flag inflates the sector table as if it
     were file data and every file comes out corrupt.

This script fixes the header, fixes both tables and relocates the block table
to the end of the archive (it cannot stay where it is - the header lives
there). The file data is never touched.
"""

import argparse
import os
import struct
import sys
import zlib

MPQ_MAGIC = b'MPQ\x1a'
HASH_TABLE_INDEX, HASH_NAME_A, HASH_NAME_B, HASH_FILE_KEY = 0, 1, 2, 3
MPQ_FILE_COMPRESS = 0x00000200
MPQ_FILE_SINGLE_UNIT = 0x01000000
MPQ_FILE_EXISTS = 0x80000000

CRYPT = []

def init_crypt():
    seed = 0x00100001
    table = [0] * 0x500
    for i in range(0x100):
        idx = i
        for _ in range(5):
            seed = (seed * 125 + 3) % 0x2AAAAB
            t1 = (seed & 0xFFFF) << 0x10
            seed = (seed * 125 + 3) % 0x2AAAAB
            t2 = seed & 0xFFFF
            table[idx] = t1 | t2
            idx += 0x100
    return table

def hash_string(s, htype):
    seed1, seed2 = 0x7FED7FED, 0xEEEEEEEE
    for ch in s.upper().replace('/', '\\').encode('latin-1'):
        seed1 = (CRYPT[(htype << 8) + ch] ^ ((seed1 + seed2) & 0xFFFFFFFF)) & 0xFFFFFFFF
        seed2 = (ch + seed1 + seed2 + (seed2 << 5) + 3) & 0xFFFFFFFF
    return seed1

def decrypt(data, key):
    seed = 0xEEEEEEEE
    out = bytearray()
    for v in struct.unpack('<%dI' % (len(data) // 4), data[:len(data) // 4 * 4]):
        seed = (seed + CRYPT[0x400 + (key & 0xFF)]) & 0xFFFFFFFF
        ch = v ^ ((key + seed) & 0xFFFFFFFF)
        key = (((~key << 0x15) + 0x11111111) | (key >> 0x0B)) & 0xFFFFFFFF
        seed = (ch + seed + (seed << 5) + 3) & 0xFFFFFFFF
        out += struct.pack('<I', ch)
    return bytes(out)

def encrypt(data, key):
    seed = 0xEEEEEEEE
    out = bytearray()
    for ch in struct.unpack('<%dI' % (len(data) // 4), data[:len(data) // 4 * 4]):
        seed = (seed + CRYPT[0x400 + (key & 0xFF)]) & 0xFFFFFFFF
        v = ch ^ ((key + seed) & 0xFFFFFFFF)
        key = (((~key << 0x15) + 0x11111111) | (key >> 0x0B)) & 0xFFFFFFFF
        seed = (ch + seed + (seed << 5) + 3) & 0xFFFFFFFF
        out += struct.pack('<I', v)
    return bytes(out)

def find_mpq(buf):
    for off in range(0, min(len(buf) - 4, 0x1000), 0x200):
        if buf[off:off + 4] == MPQ_MAGIC:
            return off
    return None

def looks_like_sector_table(buf, pos, psize, usize, sector):
    """The file data starts with an offset table whose first entry is its own size."""
    n = (usize + sector - 1) // sector
    if n < 1 or pos + 4 > len(buf):
        return False
    first = struct.unpack_from('<I', buf, pos)[0]
    if first != (n + 1) * 4 or first > psize:
        return False
    last = struct.unpack_from('<I', buf, pos + n * 4)[0]
    return last <= psize + 64

def fix(path, out_path):
    buf = bytearray(open(path, 'rb').read())
    off = find_mpq(buf)
    if off is None:
        sys.exit('no MPQ header found (is this a .w3x?)')

    magic, hsize, asize, ver, bshift, hpos, bpos, hcount, bcount = \
        struct.unpack_from('<4sIIHHIIII', buf, off)
    print('MPQ at 0x%X' % off)
    print('  headerSize = 0x%08X (%s)' % (hsize, buf[off + 4:off + 8]))
    print('  archiveSize = 0x%08X (%s)' % (asize, buf[off + 8:off + 12]))
    print('  blockSize shift = %d, hashTablePos = %d, blockTablePos = %d' % (bshift, hpos, bpos))
    print('  hashTableSize = %d, blockTableSize = %d' % (hcount, bcount))

    if ver != 0:
        sys.exit('only MPQ version 0 (v1) is supported; this one is %d' % ver)

    sector = 512 << bshift
    hkey = hash_string('(hash table)', HASH_FILE_KEY)
    bkey = hash_string('(block table)', HASH_FILE_KEY)

    # -- hash table ---------------------------------------------------------
    htab = bytearray(decrypt(bytes(buf[off + hpos:off + hpos + hcount * 16]), hkey))
    fixed_hash = 0
    for i in range(hcount):
        n1, n2, loc, plat, bi = struct.unpack_from('<IIHHI', htab, i * 16)
        if n1 == 0xFFFFFFFF and n2 == 0xFFFFFFFF and bi == 0xFFFFFFFF:
            continue                      # empty slot
        if bi == 0xFFFFFFFE:
            continue                      # deleted slot
        if bi >= bcount:                  # mask the 0x40000000 the protector added
            bi &= 0x00FFFFFF
        if loc != 0 or plat != 0 or bi != struct.unpack_from('<I', htab, i * 16 + 12)[0]:
            fixed_hash += 1
        struct.pack_into('<IIHHI', htab, i * 16, n1, n2, 0, 0, bi)
    print('  hash entries repaired: %d' % fixed_hash)

    # -- block table --------------------------------------------------------
    # dwBlockTablePos == 0 means "the block table starts at the MPQ header", so
    # its first entries are overwritten by the 32-byte header itself.
    HEADER_V1 = 0x20
    btab_pos = bpos
    overlap = 0
    if btab_pos < HEADER_V1:
        overlap = max(0, min(HEADER_V1, btab_pos + bcount * 16) - btab_pos)
    btab = bytearray(decrypt(bytes(buf[off + btab_pos:off + btab_pos + bcount * 16]), bkey))

    fixed_block = 0
    for i in range(bcount):
        fpos, psize, usize, flags = struct.unpack_from('<4I', btab, i * 16)
        if i * 16 < overlap:
            # these entries live under the MPQ header and are destroyed - they
            # are referenced by no hash entry, so blank them out
            struct.pack_into('<4I', btab, i * 16, 0, 0, 0, 0)
            continue
        if not (flags & MPQ_FILE_EXISTS) or usize == 0:
            continue
        if flags & MPQ_FILE_SINGLE_UNIT and flags & MPQ_FILE_COMPRESS:
            if looks_like_sector_table(buf, off + fpos, psize, usize, sector):
                flags &= ~MPQ_FILE_SINGLE_UNIT
                fixed_block += 1
        struct.pack_into('<4I', btab, i * 16, fpos, psize, usize, flags)
    print('  block entries repaired: %d (first %d entries blanked, they were under the header)'
          % (fixed_block, overlap // 16))

    # -- rebuild the archive ------------------------------------------------
    new_bpos = len(buf) - off                       # append the block table at the end
    buf += encrypt(bytes(btab), bkey)
    buf[off + hpos:off + hpos + hcount * 16] = encrypt(bytes(htab), hkey)

    struct.pack_into('<4sIIHHIIII', buf, off,
                     MPQ_MAGIC, 0x20, len(buf) - off, 0, bshift,
                     hpos, new_bpos, hcount, bcount)

    if not out_path:
        base, ext = os.path.splitext(path)
        out_path = base + '_fixed' + ext
    open(out_path, 'wb').write(buf)
    print('written: %s (%d bytes)' % (out_path, len(buf)))

def main():
    global CRYPT
    CRYPT = init_crypt()
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('map')
    ap.add_argument('-o', '--out', default=None)
    a = ap.parse_args()
    fix(a.map, a.out)

if __name__ == '__main__':
    main()

Method B: by hand in HxD

This is the same three edits as the general guide, plus one extra move at the beginning. Work on a copy.

▬ 3.1: pull out and fix the hash table ▬

• Start offset: 0x200 + dwHashTablePos. Here 512 + 257151744 = 257152256, which is also file size minus 65536, so it is easier to just go to the end of the file and select the last 65536 bytes (dwHashTableSize × 16).
• Copy that block into a new file, open it in MPQ Helper, decrypt with the code 7037AFC3.
• Open the decrypted file in Hex Edit Macro:
  1. START VALUE = 8, OVERWRITE VALUE = 00 00 00 00, press Modify. (This zeroes locale + platform of every entry.)
  2. START VALUE = F, OVERWRITE VALUE = 00, press Modify. (This clears the 0x40 the protector forced into the top byte of the block index.)
• Save. Decrypting correctly is easy to confirm: an empty slot must read FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF, and an occupied slot must have a block index smaller than dwBlockTableSize.
• Encrypt it again with MPQ Helper (7037AFC3) and paste it back over the same 65536 bytes in HxD.

▬ 3.2: pull out and fix the block table ▬

• Start offset: 0x200 + dwBlockTablePos = 512 + 0 = 0x200. Yes, really. That is your MPQ header. Length is dwBlockTableSize × 16 = 2757 × 16 = 44112 bytes (0xAC50).
• Copy those 44112 bytes into a new file, decrypt with A3B383EC in MPQ Helper.
• In Hex Edit Macro: START VALUE = 2F, OVERWRITE VALUE = 80, press Modify. (That is byte 15 of every entry, the top byte of the flags: it turns 0x81000200 back into 0x80000200, i.e. it removes the fake single-unit flag.)
• In HxD, zero the first 32 bytes of the decrypted file. Those are the two entries the header destroyed, and no file uses them. Leaving them as garbage is survivable (nothing points at them), but zeroing keeps things tidy and stops editors from listing two phantom files.
• Save, encrypt with A3B383EC, and do not paste it back where it came from.

▬ 3.3: move the block table to the end of the map ▬

• Go to the very end of the .w3x in HxD and append the 44112 encrypted bytes there. Call that offset BLOCKOFF. In our example the map was 257217792 bytes, so BLOCKOFF = 257217792.
• Two numbers come out of it, and neither is the offset itself:
  • the new dwBlockTablePos = BLOCKOFF - 0x200 = 257217792 - 512 = 257217280 = 0x0F54D300, which in little-endian bytes is 00 D3 54 0F
  • the new dwArchiveSize = new file size - 0x200 = (257217792 + 44112) - 512 = 257261392 = 0x0F557F50, which in little-endian bytes is 50 7F 55 0F
• Use a calculator. Getting these two examples confused is the single most common way to end up with a map that still will not open.

▬ 3.4: repair the first line of the MPQ header ▬

• 0x204: dwHeaderSize, write 20 00 00 00 (= 32)
• 0x208: dwArchiveSize, write the new file size minus 0x200. Here 50 7F 55 0F (from step 3.3).
• 0x20C: wFormatVersion, write 00 00 (already fine)
• 0x20E: wBlockSize, leave it alone
• 0x210: dwHashTablePos, leave it alone
• 0x214: dwBlockTablePos, write the new value from step 3.3. Here 00 D3 54 0F.
• 0x218 / 0x21C: the two table sizes, leave them alone

Save. That is the whole job.

Step 4: Check your work

1. Open the fixed map in MPQ Editor. You should get a real file list. If the list is empty, your hash table re-encryption went wrong (you probably wrote it back in plaintext, or at the wrong offset).
2. Extract war3map.j and open it in a text editor. It must start with something like

   Code: Select all

   globals

   and be readable. If you get binary garbage, the fake single-unit flag is still there (step 3.2).
3. If a few files come out empty or corrupt, do not panic: check whether their block entries have a data position of 0. Only the first two are supposed to be blank.

Note: some of these maps compress a handful of files with PKWARE implode (the sector data starts with the byte 08 instead of 02 for zlib). That has nothing to do with the protection, and it does not stop the fix. MPQ Editor handles it, and so do most Warcraft modding tools. Only a home-made extractor will choke on those, and only on those.

Note: the protector does not encrypt anything. All keys are the standard MPQ keys (7037AFC3 for the hash table, A3B383EC for the block table), which is why this is a deprotection job and not a decryption job. If MPQ Helper produces garbage at those codes, you are pointed at the wrong offset. Go back to step 1 and recompute the positions from the header. Remember the sizes are per-map: it is always dwHashTableSize × 16 and dwBlockTableSize × 16, not the numbers in this post.

Why the two destroyed entries are harmless

You may wonder how a map can ship with two of its block table entries overwritten. It is on purpose. The protector writes the block table first, then puts the header on top of it, and then rebuilds the hash table so that no file points at index 0 or 1.

You can verify this yourself: after decrypting the hash table (step 3.1), look at the low three bytes of every block index. In our example the 2755 occupied slots use indices 2 through 2756, all distinct, and 0 and 1 never appear. That is also the fastest way to tell this protection apart from a map where the block table is genuinely damaged: if 0 or 1 appear in a hash entry, the file they point to is gone and you cannot get it back.

Summary

• Recognise: "SProtect" spelled across dwHeaderSize + dwArchiveSize, and dwBlockTablePos = 0.
• Understand: dwBlockTablePos = 0 means the block table lives on top of the 32-byte header, and its first two entries are gone.
• Fix: zero bytes 8-11 and byte 15 of every hash entry; set byte 15 of every block entry to 0x80; move the block table to the end; repair dwHeaderSize, dwArchiveSize and dwBlockTablePos.
• Check: MPQ Editor shows the files, and war3map.j opens as text.

Tested on U9_Heaven_RPG_S2_3.2_Fix6.w3x (245 MiB, 2755 files). Extracted files after the fix are byte-identical to the originals.
https://m16tool.xyz/Game/HVR/Main/Main

Have fun.