[GUIDE] Fixing w3protect ultimate pro max tier lmao

[devoltz]

W3P: the fake header, the missing HM3W and the 30,000 fake files

(how to open maps like Zombie Defense 0.25xmas_fix_w3p, and hide the junk from MPQ Editor)

Hello.

This guide is for maps protected with the protector that leaves _w3p or w3pro in the file name. It comes in two strengths.

The light one only writes 02 01 20 00 into dwHeaderSize, which puts MPQ Editor into read-only mode. That is a 4-byte fix and it is covered at the end.

The heavy one is what this guide is really about. Our example is Zombie_Defense_0.25xmas_fix_w3p.w3x (51,073,855 bytes). It has no HM3W header, a fake MPQ header in the usual place, the real one hidden a bit further down, a decoy script, and about 30,000 fake files that make MPQ Editor, "Extract all" and most converters useless. The map itself plays fine, because the game ignores every one of these tricks.

Before we start

- What is HM3W?

• The first 512 bytes of a .w3x: "HM3W", 4 zero bytes, the map name, the map flags and the player count. The World Editor and the map list read it. The MPQ archive starts right after it, at 0x200.

- What is the MPQ header?

• 32 bytes that start with 4D 50 51 1A ("MPQ\x1A"). They say where the hash table and the block table are, and how big they are. Every position in the archive is counted from the header, not from the start of the file.

- What is "user data"?

• A header that starts with 4D 50 51 1B ("MPQ\x1B"). StarCraft II uses it to point at the real MPQ header. Warcraft III does not read it at all. The StormLib source says it plainly: Warcraft III does not check for user data, and many map protectors abuse that.

- How does the game find the MPQ header?

• It checks every multiple of 0x200 from the start of the file. It takes the first "MPQ\x1A" whose header size is at least 0x20, unless its hash table or block table would start past the end of the file. That one is thrown away and the search goes on. This last rule is the whole trick.

Tools you need

• HxD
• MPQ Editor (Ladik's)
• For the fake files: Python 3 and the script at the bottom of this post. There is no sane way to deal with 30,000 entries by hand.

Step 1: Recognise it

Open the map in HxD and look at the first 0x420 bytes:

Code: Select all

000000  4D 50 51 1B 39 05 00 00 39 05 00 00 39 05 00 00   MPQ.9...9...9...
000010  42 DC BC C2 45 F6 B2 3C 5F 80 F3 D4 7C 0D 54 38   (random bytes up to 0x1FF)
...
000200  4D 50 51 1A 02 01 20 00 38 B9 ED 01 00 00 03 00   MPQ.....8.......
000210  D6 44 8B 3A 6F 86 8D 81 15 6D 87 61 11 3C 4B 7D   (random bytes up to 0x3FF)
...
000400  4D 50 51 1A 20 00 00 00 40 4F FB 02 00 00 03 00   MPQ.....@O......
000410  FF DA F3 02 FF DA 03 03 00 00 01 00 44 77 00 00   ............Dw..

Three things are wrong, and you can see all of them here:

• 0x000: no "HM3W". Instead there is a user data header, "MPQ\x1B", with all three of its fields set to 0x539 (1337). 0x539 is not even a multiple of 0x200, and what sits there is random bytes.
• 0x200: an MPQ header that is fake. The signature is right and the header size is 02 01 20 00 = 0x00200102, which passes the "at least 0x20" test. Everything after that is random: the hash table would start at 0x200 + 0x3A8B44D6, which is about 980 MB into a 51 MB file, and the table sizes are in the billions.
• 0x400: the real header. It is clean:
  • 0x404: 20 00 00 00 = dwHeaderSize 32. Fine.
  • 0x408: 40 4F FB 02 = dwArchiveSize. Wrong, but the game ignores it.
  • 0x40C: 00 00 = format version 0. 0x40E: 03 00 = sector size 512 << 3 = 4096.
  • 0x410: FF DA F3 02 = dwHashTablePos. 0x414: FF DA 03 03 = dwBlockTablePos.
  • 0x418: 00 00 01 00 = 65,536 hash entries. 0x41C: 44 77 00 00 = 30,532 block entries.

Sanity check: 0x400 + 0x0303DAFF + 30,532 × 16 = 0x030B533F = 51,073,855, the file size. The block table is the last thing in the file and the hash table sits right before it. Remember the base is 0x400 here, not 0x200.

Symptoms you will have seen:

• Converters and old tools say the file is not a valid MPQ, crash, or show an empty file list. They stopped at 0x200 and read the tables from nowhere.
• MPQ Editor opens it but lists about 30,500 files, almost all of them "File0000xxxx.xxx".
• "Extract all" never ends, and the disk fills up.
• Extracting Scripts\war3map.j fails, while war3map.j works.

Step 2: What the protection actually does

1) The HM3W header is replaced with user data.

Warcraft III never reads user data, so the map still plays. But anything that checks for HM3W no longer sees a map, and anything that follows the user data (a tool that thinks it is a StarCraft II archive) lands on garbage at 0x539.

2) A fake header at 0x200.

The game finds it first, sees that its tables would start past the end of the file, drops it and keeps looking. It finds the real header at 0x400. A tool that just takes the first "MPQ\x1A" it finds never gets there. The 02 01 20 00 in its header size is the same value the light variant writes into the real header, which is how you know both come from the same protector.

3) About 30,000 fake files.

The block table has 30,532 entries and the hash table points at 30,530 of them. Only 529 are real files. The fakes come in three kinds:

• Twins: exact copies of a real file's block entry (same position, same sizes, same flags) under a random name. Almost every real file has between 1 and 11 of them (8 have none). 2,014 in total. They read fine, because they are the real data.
• Near twins: copies of a real entry with the sizes shifted by a few bytes. The data is real but the sizes are not, so they cannot be read.
• Random slices: entries that point at random places inside the archive, with random sizes, and about 8,000 of them flagged as encrypted.

Each fake has its own random name in the hash table, so MPQ Editor shows every one of them. Together they claim 248 GB of unpacked data, which is why "Extract all" never finishes. The game only ever opens files by their real names, so it never touches any of this.

4) A decoy script.

Scripts\war3map.j has the exact unpacked size of the real war3map.j (1,778,027 bytes), but its sector offset table is random numbers, so it cannot be read. The game loads war3map.j. A tool that prefers Scripts\war3map.j fails or hangs.

5) No (listfile).

Nothing unusual here, but it means the real files show up as unnamed too, until you load a listfile.

Step 3: Removing it

Method A: the script (recommended)

Save the script from the spoiler at the bottom as w3p_fix.py, put it next to your map and run:

Code: Select all

python w3p_fix.py MyMap.w3x
python w3p_fix.py MyMap.w3x --clean
python w3p_fix.py MyMap.w3x --clean --listfile=names.txt

The first line only repairs the headers. It writes MyMap_fixed.w3x with a real HM3W header (name, flags and player count taken from the map's own war3map.w3i), the fake header's signature zeroed, and dwArchiveSize corrected. Nothing else changes.

--clean also hides the fake files. It needs no names, and on our map it hides 27,987 of the 30,530 entries. With a listfile it also hides the twins of every file the listfile names: 29,066 hidden, 1,464 left. The 1,464 are the 529 real files plus the twins of real files whose names are unknown. Those twins stay on purpose: if you do not know a file's name, you cannot tell which of its twins is the entry the game opens, and deleting the wrong one breaks the file.

How does it know a file is fake? A real file must be readable the way the game reads it. For a compressed file that means: the sector offset table starts with its own size, only goes up, and ends exactly at the packed size of the block, and every sector unpacks to exactly the size it should. A file that fails this cannot be opened by the game, so hiding it changes nothing in game. The same goes for an entry that covers part of a file that does pass: two real files never share bytes. Encrypted files with no known name are checked the same way StormLib reads them, by recovering the key from the sector table.

I checked this against 14,039 real files from 12 maps (Chinese KK maps, Korean RPGs from m16, Wurst maps and these ones) and every single one passed. On this map, the only named file that fails is the decoy Scripts\war3map.j, which is exactly what should happen.

"Hiding" means: the hash entry of each fake gets the block index FF FF FF FE (deleted) and its block entry becomes all zeros. A deleted entry does not break the search for other names, so every real name still resolves to the same data, and MPQ Editor reuses deleted slots when you add a file. No file data is moved or changed.

Where do I get a listfile? The bigger the better. Any of the well known Warcraft III listfiles helps, and names mined from the map's own script and object data help more. On this map I recovered 293 names.

Method B: by hand in HxD

The header part is easy to do by hand. The fake files are not (30,000 entries), so for those use the script, or just live with them (see Step 4). Work on a copy.

▬ 3.1: write the HM3W header over the user data ▬

• Select 0x000 to 0x1FF and fill it with zeros.
• At 0x000 type 48 4D 33 57 00 00 00 00 ("HM3W" and four zeros).
• At 0x008 type the map name as text, then one 00.
• Right after that 00: the map flags (4 bytes) and the player count (4 bytes), both little-endian. You find both in war3map.w3i. For this map: flags 68 DC 03 00 (0x0003DC68) and players 10 00 00 00 (16).

Getting the flags or the player count wrong is not fatal: the map list may show the wrong number of players, but the game takes everything else from war3map.w3i.

▬ 3.2: kill the fake header ▬

• At 0x200, replace 4D 50 51 1A with 00 00 00 00.

Now nothing can mistake it for a header, and every tool lands on 0x400. Do not touch the rest of it.

▬ 3.3: optional, fix dwArchiveSize ▬

• At 0x408 write the file size minus 0x400. Here 51,073,855 - 1,024 = 0x030B4F3F, which is 3F 4F 0B 03.

The game ignores this field, but some tools use it. Save. That is the header part done.

Step 4: Check your work

1. Open the fixed map in MPQ Editor. It must open in edit mode. Without --clean you still see about 30,500 entries. With --clean about 2,500, and with --clean and a listfile about 1,460.
2. Load your listfile in MPQ Editor so the real names show up.
3. Extract war3map.j and open it in a text editor. It must start with

   Code: Select all

   globals

4. Compare with the original in HxD (Analysis, File compare). The header fix changes only the first 0x204 bytes and 0x408-0x40B. --clean also changes the hash table and the block table (the last 1.5 MB of the file). Nothing in between may change.

Warning: do not use "Extract all" or "Compact" on a map that still has its fake files. Both go through every entry, and the fakes add up to hundreds of gigabytes. Hide them first, or extract by name with a listfile.

Note: the World Editor wants more than the headers. This map's script was run through an optimizer, so the trigger files are gone (war3map.wtg, war3map.wct). Opening it in the World Editor is a separate job.

The light variant

Castle Fight DE beta 9.33, Forest Defense 0.21g and Sunken Defense 05r have none of the above. Their real header sits at 0x200 as usual, and the only change is dwHeaderSize:

Code: Select all

000200  4D 50 51 1A 02 01 20 00 ...

Write 20 00 00 00 at 0x204 and MPQ Editor opens it in edit mode. The script does the same.

Two more things you may run into with these maps:

• They use a sector size of 16 MB (0x20E = 0F 00), so every file is one sector, and Castle Fight packs some files with bzip2 (the sector starts with 10 42 5A 68, the byte 0x10 followed by "BZh"). Tools that only know zlib fail on those files. MPQ Editor reads them fine.
• Castle Fight fills its whole hash table: 65,536 entries for 272 files, up to 280 entries per file. Nothing is fake data, but with no free slot MPQ Editor cannot add a file. python w3p_fix.py CastleFight.w3x --aliases --listfile=names.txt marks the extra entries of every named file as deleted (32,135 on that map) and leaves the name's own entry alone.

Summary

• Recognise: "MPQ\x1B" at 0x000 instead of HM3W, "MPQ\x1A 02 01 20 00" at 0x200 with nonsense after it, and the real header at 0x400.
• Understand: the game ignores user data and skips a header whose tables are outside the file. The fakes are entries the game never opens.
• Fix: write HM3W over 0x000-0x1FF, zero the signature at 0x200, optionally fix dwArchiveSize at 0x408. Hide the fake files with the script.
• Check: MPQ Editor opens it in edit mode, war3map.j extracts as text, and nothing outside the headers and tables changed.

Tested on Zombie_Defense_0.25xmas_fix_w3p.w3x (51,073,855 bytes, 30,530 entries, 529 real files). After the fix and after --clean, every named file extracts byte-identical to the original, and all 1,464 remaining entries read the same data as before.

Have fun.

Spoiler:

Code: Select all

#!/usr/bin/env python3
"""
What the protection does:

  1. Offset 0 holds an MPQ "user data" header (MPQ\\x1B, a StarCraft II
     feature) instead of the HM3W map header. Warcraft III never reads user
     data, so the map still plays, but anything that looks for HM3W does not
     see a map.

  2. Offset 0x200 holds a FAKE MPQ header: a valid "MPQ\\x1A" signature, a
     header size of 0x00200102, and random numbers where the table positions
     go. The game scans for the header every 0x200 bytes and throws away any
     header whose hash table or block table starts past the end of the file,
     then keeps scanning. Tools that stop at the first "MPQ\\x1A" read garbage.

  3. The real header sits further down (0x400 in the maps we have seen).
     The simpler variant of this protector has no decoys and only writes
     0x00200102 into dwHeaderSize of the real header, which puts MPQ Editor
     into read-only mode.

What this script changes (the data of the files is never touched):

  - writes an HM3W header at offset 0 when the file does not start with one
    (name, flags and player count come from the map's own war3map.w3i);
  - zeroes the signature of every fake header before the real one;
  - repairs dwHeaderSize, wFormatVersion and dwArchiveSize of the real header.

With --clean it also hides the FAKE FILES, so MPQ Editor stops listing them:

    python w3p_fix.py MyMap.w3x --clean
    python w3p_fix.py MyMap.w3x --clean --listfile=names.txt

A block is fake when the game could not read it: its sector table does not
start at its own size, does not end exactly at the packed size, or a sector
does not unpack to the exact size; or when it overlaps part of a block that
IS readable (two real files never share bytes). Their hash entries are marked
"deleted" (lookups of real names still work, and MPQ Editor reuses the slots)
and their block entries are zeroed. File data is never touched.
With --listfile, exact twins of a NAMED file (same position, sizes and flags,
under a random name) are hidden too. Twins of files whose names you do not
know stay, because you cannot tell which twin the game opens.

With --aliases --listfile=names.txt it frees a hash table that the protector
filled completely with extra entries for the same files (Castle Fight DE:
65,536 entries for 272 files, so MPQ Editor cannot add a file). The extra
entries of every NAMED file are marked deleted; the name's own entry stays.
"""

import argparse
import bz2
import os
import re
import struct
import sys
import zlib

CRYPT = []


def init_crypt():
    seed = 0x00100001
    table = [0] * 0x500
    for i in range(0x100):
        idx = i
        for _ in range(5):
            seed = (seed * 125 + 3) % 0x2AAAAB
            t1 = (seed & 0xFFFF) << 0x10
            seed = (seed * 125 + 3) % 0x2AAAAB
            t2 = seed & 0xFFFF
            table[idx] = t1 | t2
            idx += 0x100
    return table


def hash_string(s, htype):
    seed1, seed2 = 0x7FED7FED, 0xEEEEEEEE
    for ch in s.upper().replace('/', '\\').encode('latin-1'):
        seed1 = (CRYPT[(htype << 8) + ch] ^ ((seed1 + seed2) & 0xFFFFFFFF)) & 0xFFFFFFFF
        seed2 = (ch + seed1 + seed2 + (seed2 << 5) + 3) & 0xFFFFFFFF
    return seed1


def decrypt(data, key):
    seed = 0xEEEEEEEE
    out = bytearray()
    n = len(data) // 4
    for v in struct.unpack('<%dI' % n, data[:n * 4]):
        seed = (seed + CRYPT[0x400 + (key & 0xFF)]) & 0xFFFFFFFF
        ch = v ^ ((key + seed) & 0xFFFFFFFF)
        key = (((~key << 0x15) + 0x11111111) | (key >> 0x0B)) & 0xFFFFFFFF
        seed = (ch + seed + (seed << 5) + 3) & 0xFFFFFFFF
        out += struct.pack('<I', ch)
    return bytes(out) + data[n * 4:]


def encrypt(data, key):
    seed = 0xEEEEEEEE
    out = bytearray()
    for ch in struct.unpack('<%dI' % (len(data) // 4), data[:len(data) // 4 * 4]):
        seed = (seed + CRYPT[0x400 + (key & 0xFF)]) & 0xFFFFFFFF
        v = ch ^ ((key + seed) & 0xFFFFFFFF)
        key = (((~key << 0x15) + 0x11111111) | (key >> 0x0B)) & 0xFFFFFFFF
        seed = (ch + seed + (seed << 5) + 3) & 0xFFFFFFFF
        out += struct.pack('<I', v)
    return bytes(out)


def find_header(buf):
    """The header the game opens: scan every 0x200 bytes, ignore user data
    (MPQ\\x1B), accept MPQ\\x1A with a header size of at least 0x20, and skip
    it if a table starts past the end of the file."""
    fakes = []
    for off in range(0, len(buf) - 32, 0x200):
        if buf[off:off + 4] != b'MPQ\x1a':
            continue
        hsize, asize, ver, bshift, hpos, bpos, hcount, bcount = struct.unpack_from('<IIHHIIII', buf, off + 4)
        if hsize < 0x20:
            continue
        if (off + hpos) & 0xFFFFFFFF > len(buf) or (off + bpos) & 0xFFFFFFFF > len(buf):
            fakes.append(off)
            continue
        return off, fakes
    return None, fakes


def read_file(buf, off, name):
    """One file from the archive, by name (zlib, bzip2 or stored)."""
    hsize, asize, ver, bshift, hpos, bpos, hcount, bcount = struct.unpack_from('<IIHHIIII', buf, off + 4)
    hbase = (off + hpos) & 0xFFFFFFFF
    bbase = (off + bpos) & 0xFFFFFFFF
    htab = decrypt(bytes(buf[hbase:hbase + hcount * 16]), hash_string('(hash table)', 3))
    btab = decrypt(bytes(buf[bbase:bbase + bcount * 16]), hash_string('(block table)', 3))
    start = hash_string(name, 0) & (hcount - 1)
    a, b = hash_string(name, 1), hash_string(name, 2)
    found = None
    for probe in range(hcount):
        i = (start + probe) & (hcount - 1)
        n1, n2, loc, plat, bi = struct.unpack_from('<IIHHI', htab, i * 16)
        if n1 == a and n2 == b and bi < bcount:
            if loc == 0 and plat & 0xFF == 0 or found is None:
                found = bi
        elif bi == 0xFFFFFFFF:
            break
    if found is None:
        return None
    fpos, psize, usize, flags = struct.unpack_from('<4I', btab, found * 16)
    raw = bytes(buf[(off + fpos) & 0xFFFFFFFF:((off + fpos) & 0xFFFFFFFF) + psize])
    key = None
    if flags & 0x00010000:
        key = hash_string(name.replace('/', '\\').split('\\')[-1], 3)
        if flags & 0x00020000:
            key = ((key + fpos) ^ usize) & 0xFFFFFFFF
    sector = 512 << (bshift & 0xFF)
    nsec = max(1, (usize + sector - 1) // sector)

    def unpack(chunk, expected):
        if len(chunk) == expected:
            return chunk
        if chunk[0] == 0x02:
            return zlib.decompress(chunk[1:])
        if chunk[0] == 0x10:
            return bz2.decompress(chunk[1:])
        raise ValueError('compression 0x%02X not supported by this script' % chunk[0])

    if flags & 0x01000000:                           # single unit
        data = decrypt(raw, key) if key is not None else raw
        return unpack(data, usize)[:usize] if flags & 0x00000300 else data[:usize]
    if not flags & 0x00000300:                       # stored, not compressed
        out = bytearray()
        for s in range(nsec):
            chunk = raw[s * sector:(s + 1) * sector]
            out += decrypt(chunk, (key + s) & 0xFFFFFFFF) if key is not None else chunk
        return bytes(out[:usize])
    table = raw[:(nsec + 1) * 4]
    if key is not None:
        table = decrypt(table, (key - 1) & 0xFFFFFFFF)
    offsets = struct.unpack('<%dI' % (nsec + 1), table)
    out = bytearray()
    for s in range(nsec):
        chunk = raw[offsets[s]:offsets[s + 1]]
        if key is not None:
            chunk = decrypt(chunk, (key + s) & 0xFFFFFFFF)
        out += unpack(chunk, min(sector, usize - s * sector))
    return bytes(out[:usize])


def map_header(buf, off):
    """The 512-byte HM3W header, built from war3map.w3i (and war3map.wts for
    a TRIGSTR name)."""
    w3i = read_file(buf, off, 'war3map.w3i')
    p = 0

    def i32():
        nonlocal p
        p += 4
        return struct.unpack_from('<i', w3i, p - 4)[0]

    def cstr():
        nonlocal p
        e = w3i.index(b'\0', p)
        s = w3i[p:e]
        p = e + 1
        return s

    version = i32()
    i32()
    i32()
    if version >= 28:
        p += 16
    name = cstr().decode('utf-8', 'replace')
    cstr()
    cstr()
    cstr()
    p += 8 * 4 + 4 * 4 + 4 + 4
    flags = i32()
    p += 1                                            # tileset
    i32()                                             # loading screen number
    if version >= 25:
        cstr()                                        # loading screen model
    cstr()
    cstr()
    cstr()
    i32()                                             # game data set / loading number
    if version >= 25:
        cstr()                                        # prologue model
    cstr()
    cstr()
    cstr()
    if version >= 25:
        p += 4 + 4 + 4 + 4 + 4 + 4                    # fog type, start, end, density, color, weather
        cstr()                                        # sound environment
        p += 1 + 4                                    # light environment, water tint
    if version >= 28:
        i32()                                         # script language
    if version >= 31:
        i32()
        i32()
    players = i32()
    m = re.match(r'^TRIGSTR_(\d+)$', name.strip())
    if m:
        wts = read_file(buf, off, 'war3map.wts')
        if wts:
            for s in re.finditer(r'STRING[ \t]+(\d+)[^\n]*\n(?://[^\n]*\n)*\{\r?\n(.*?)\r?\n\}',
                                 wts.decode('utf-8', 'replace'), re.S):
                if int(s.group(1)) == int(m.group(1)):
                    name = s.group(2)
                    break
    head = b'HM3W' + b'\0' * 4 + name.encode('utf-8')[:400] + b'\0' + struct.pack('<ii', flags, players)
    return head + b'\0' * (0x200 - len(head)), name, flags, players


def detect_key(enc0, enc1, dec0, accept1):
    """StormLib's DetectFileKeyBySectorSize / DetectFileKeyByKnownContent: the
    key that turns the first encrypted dword into dec0 and whose second dword
    passes accept1."""
    k12 = ((enc0 ^ dec0) - 0xEEEEEEEE) & 0xFFFFFFFF
    for i in range(0x100):
        k1 = (k12 - CRYPT[0x400 + i]) & 0xFFFFFFFF
        k2 = (0xEEEEEEEE + CRYPT[0x400 + (k1 & 0xFF)]) & 0xFFFFFFFF
        if enc0 ^ ((k1 + k2) & 0xFFFFFFFF) != dec0:
            continue
        saved = k1
        k1 = (((~k1 << 0x15) + 0x11111111) | (k1 >> 0x0B)) & 0xFFFFFFFF
        k2 = (dec0 + k2 + (k2 << 5) + 3) & 0xFFFFFFFF
        k2 = (k2 + CRYPT[0x400 + (k1 & 0xFF)]) & 0xFFFFFFFF
        if accept1(enc1 ^ ((k1 + k2) & 0xFFFFFFFF)):
            return saved
    return None


KNOWN_STARTS = ((0x584C444D, lambda v, fs: v == 0x53524556),     # MDLX VERS (.mdx)
                (0x31504C42, lambda v, fs: v in (0, 1)),         # BLP1
                (0x46464952, lambda v, fs: v == fs - 8),         # RIFF (.wav)
                (0x00905A4D, lambda v, fs: v == 3),              # MZ
                (0x6D783F3C, lambda v, fs: v == 0x6576206C))     # <?xml


def unpack_sector(chunk, expected):
    """-> bytes, or None when this script does not know the compression."""
    if chunk[0] == 0x02:
        return zlib.decompress(chunk[1:])
    if chunk[0] == 0x10:
        return bz2.decompress(chunk[1:])
    if chunk[0] == 0x00:
        return chunk[1:]
    if chunk[0] == 0x78:                               # zlib stream with no mask byte
        return zlib.decompress(chunk)
    return None


def validate(buf, off, sector, block, key_from_name=None):
    """'ok' (the game would read it), 'bad' (it would not) or 'unsure'."""
    fpos, psize, usize, flags = block
    if not flags & 0x80000000:
        return 'bad'
    p = (off + fpos) & 0xFFFFFFFF
    if p + psize > len(buf):
        return 'bad'
    if usize == 0:
        return 'ok'
    raw = bytes(buf[p:p + psize])
    nsec = (usize + sector - 1) // sector
    comp = bool(flags & 0x00000300)
    ntab = nsec + 1 + (1 if flags & 0x04000000 else 0)
    key = key_from_name
    if flags & 0x00010000 and key is None:
        if flags & 0x01000000 or len(raw) < 8:
            return 'unsure'
        e0, e1 = struct.unpack_from('<II', raw, 0)
        if comp:
            for d0 in range(ntab * 4, ntab * 4 + 4):
                key = detect_key(e0, e1, d0, lambda v, d0=d0: v <= sector + d0)
                if key is not None:
                    key = (key + 1) & 0xFFFFFFFF
                    break
            if key is None:
                return 'bad'
        else:
            for d0, test in KNOWN_STARTS:
                key = detect_key(e0, e1, d0, lambda v, test=test: test(v, usize))
                if key is not None:
                    break
            if key is None:
                return 'unsure'

    def sector_ok(chunk, expected):
        if len(chunk) > expected:
            return 'bad'
        if comp and len(chunk) < expected:
            try:
                data = unpack_sector(chunk, expected)
            except Exception:
                return 'bad'
            if data is None:
                return 'unsure'
            if len(data) != expected:
                return 'bad'
        return 'ok'

    if flags & 0x01000000:
        data = decrypt(raw, key) if key is not None else raw
        if not comp or psize == usize:
            return 'ok' if psize == usize else 'bad'
        return sector_ok(data, usize)
    if not comp:
        return 'ok' if psize == usize else 'bad'
    table = raw[:ntab * 4]
    if len(table) < ntab * 4:
        return 'bad'
    if key is not None:
        table = decrypt(table, (key - 1) & 0xFFFFFFFF)
    offs = struct.unpack('<%dI' % ntab, table)
    if offs[0] != ntab * 4 or offs[ntab - 1] != psize:
        return 'bad'
    if any(offs[s] > offs[s + 1] for s in range(ntab - 1)):
        return 'bad'
    for s in [nsec - 1] + list(range(nsec - 1)):
        chunk = raw[offs[s]:offs[s + 1]]
        if key is not None:
            chunk = decrypt(chunk, (key + s) & 0xFFFFFFFF)
        r = sector_ok(chunk, min(sector, usize - s * sector))
        if r != 'ok':
            return r
    return 'ok'


def clean(buf, off, names):
    """Hides the fake files: deleted hash entries, zeroed block entries."""
    hsize, asize, ver, bshift, hpos, bpos, hcount, bcount = struct.unpack_from('<IIHHIIII', buf, off + 4)
    sector = 512 << (bshift & 0xFF)
    hbase = (off + hpos) & 0xFFFFFFFF
    bbase = (off + bpos) & 0xFFFFFFFF
    hkey, bkey = hash_string('(hash table)', 3), hash_string('(block table)', 3)
    htab = list(struct.unpack('<%dI' % (hcount * 4), decrypt(bytes(buf[hbase:hbase + hcount * 16]), hkey)))
    btab = list(struct.unpack('<%dI' % (bcount * 4), decrypt(bytes(buf[bbase:bbase + bcount * 16]), bkey)))
    blocks = [tuple(btab[i * 4:i * 4 + 4]) for i in range(bcount)]
    # the block of every known name (every entry of the name, like the game's probing)
    named = {}
    for name in list(names) + ['war3map.j', 'war3map.lua', 'war3map.w3e', 'war3map.w3i', 'war3map.wts',
                               'war3map.doo', 'war3map.wpm', 'war3map.shd', 'war3mapMap.blp', 'war3map.w3u',
                               '(listfile)', '(attributes)']:
        start = hash_string(name, 0) & (hcount - 1)
        a, b = hash_string(name, 1), hash_string(name, 2)
        for probe in range(hcount):
            i = (start + probe) & (hcount - 1)
            n1, n2, bi = htab[i * 4], htab[i * 4 + 1], htab[i * 4 + 3]
            if n1 == a and n2 == b and bi < bcount:
                named.setdefault(bi, name)
            elif bi == 0xFFFFFFFF:
                break
    used = sorted(set(htab[i * 4 + 3] for i in range(hcount)) - {0xFFFFFFFF, 0xFFFFFFFE})
    state = {}
    for bi in used:
        if bi >= bcount:
            continue
        key = None
        name = named.get(bi)
        if name is not None and blocks[bi][3] & 0x00010000:
            key = hash_string(name.replace('/', '\\').split('\\')[-1], 3)
            if blocks[bi][3] & 0x00020000:
                key = ((key + blocks[bi][0]) ^ blocks[bi][2]) & 0xFFFFFFFF
        state[bi] = validate(buf, off, sector, blocks[bi], key)
    # partial overlap with a readable block = fake (real files never share bytes)
    import bisect
    real = sorted((blocks[b][0], blocks[b][0] + blocks[b][1], b) for b, s in state.items() if s == 'ok' and blocks[b][1])
    starts = [r[0] for r in real]
    maxend = []
    m = -1
    for r in real:
        m = max(m, r[1])
        maxend.append(m)
    for bi, s in list(state.items()):
        if s == 'bad' or not blocks[bi][1]:
            continue
        fpos, psize, usize, flags = blocks[bi]
        k = bisect.bisect_right(starts, fpos + psize - 1) - 1
        while k >= 0 and maxend[k] > fpos:
            ro, re_, rb = real[k]
            if rb != bi and re_ > fpos and ro < fpos + psize and blocks[rb][:3] != (fpos, psize, usize):
                if s == 'unsure' or rb in named and bi not in named:
                    state[bi] = 'bad'
                break
            k -= 1
    # exact twins of a named file
    if names:
        groups = {}
        for bi, s in state.items():
            if s == 'ok':
                groups.setdefault(blocks[bi], []).append(bi)
        for group in groups.values():
            if any(b in named for b in group):
                for b in group:
                    if b not in named:
                        state[b] = 'bad'
    fake = set(b for b, s in state.items() if s == 'bad')
    entries = 0
    for i in range(hcount):
        if htab[i * 4 + 3] in fake:
            htab[i * 4 + 3] = 0xFFFFFFFE
            entries += 1
    for b in fake:
        btab[b * 4:b * 4 + 4] = [0, 0, 0, 0]
    buf[hbase:hbase + hcount * 16] = encrypt(struct.pack('<%dI' % len(htab), *htab), hkey)
    buf[bbase:bbase + bcount * 16] = encrypt(struct.pack('<%dI' % len(btab), *btab), bkey)
    kept = sum(1 for s in state.values() if s != 'bad')
    print('  fake files hidden: %d (%d hash entries marked deleted); %d files left in the list' % (len(fake), entries, kept))


def free_aliases(buf, off, names):
    """Extra hash entries pointing at a named file's block -> deleted."""
    hsize, asize, ver, bshift, hpos, bpos, hcount, bcount = struct.unpack_from('<IIHHIIII', buf, off + 4)
    hbase = (off + hpos) & 0xFFFFFFFF
    hkey = hash_string('(hash table)', 3)
    htab = list(struct.unpack('<%dI' % (hcount * 4), decrypt(bytes(buf[hbase:hbase + hcount * 16]), hkey)))
    keep = {}
    for name in list(names) + ['war3map.j', 'war3map.lua', 'war3map.w3e', 'war3map.w3i', 'war3map.wts',
                               'war3map.doo', 'war3map.wpm', 'war3map.shd', 'war3mapMap.blp', 'war3map.w3u',
                               '(listfile)', '(attributes)']:
        start = hash_string(name, 0) & (hcount - 1)
        a, b = hash_string(name, 1), hash_string(name, 2)
        for probe in range(hcount):
            i = (start + probe) & (hcount - 1)
            n1, n2, bi = htab[i * 4], htab[i * 4 + 1], htab[i * 4 + 3]
            if n1 == a and n2 == b and bi < bcount:
                keep.setdefault(bi, set()).add(i)
            elif bi == 0xFFFFFFFF:
                break
    freed = 0
    for i in range(hcount):
        bi = htab[i * 4 + 3]
        if bi in keep and i not in keep[bi]:
            htab[i * 4 + 3] = 0xFFFFFFFE
            freed += 1
    buf[hbase:hbase + hcount * 16] = encrypt(struct.pack('<%dI' % len(htab), *htab), hkey)
    print('  aliases: %d extra hash entries marked deleted (%d named files)' % (freed, len(keep)))


def fix(path, out_path, do_clean=False, names=(), aliases=False):
    buf = bytearray(open(path, 'rb').read())
    off, fakes = find_header(bytes(buf))
    if off is None:
        sys.exit('no MPQ header the game would open (is this a .w3x?)')
    print('real MPQ header at 0x%X' % off)
    for f in fakes:
        buf[f:f + 4] = b'\0\0\0\0'
        print('  fake header at 0x%X: signature zeroed' % f)

    hsize, asize, ver, bshift = struct.unpack_from('<IIHH', buf, off + 4)
    if hsize != 0x20:
        struct.pack_into('<I', buf, off + 4, 0x20)
        print('  dwHeaderSize 0x%08X -> 0x00000020' % hsize)
    if ver != 0:
        struct.pack_into('<H', buf, off + 12, 0)
        print('  wFormatVersion %d -> 0' % ver)
    if asize != len(buf) - off:
        struct.pack_into('<I', buf, off + 8, len(buf) - off)
        print('  dwArchiveSize 0x%08X -> 0x%08X' % (asize, len(buf) - off))

    if buf[:4] != b'HM3W':
        if off < 0x200:
            print('  no HM3W and no room for it (the MPQ starts at 0x%X): left as is' % off)
        else:
            head, name, flags, players = map_header(buf, off)
            buf[0:0x200] = head
            print('  HM3W written: name %r, flags 0x%X, %d players' % (name, flags, players))

    if aliases:
        if not names:
            sys.exit('--aliases needs --listfile (the names say which entry is the real one)')
        free_aliases(buf, off, names)
    if do_clean:
        clean(buf, off, names)

    if not out_path:
        base, ext = os.path.splitext(path)
        out_path = base + '_fixed' + ext
    open(out_path, 'wb').write(buf)
    print('written: %s (%d bytes)' % (out_path, len(buf)))

    check = read_file(buf, find_header(bytes(buf))[0], 'war3map.j') or \
        read_file(buf, find_header(bytes(buf))[0], 'war3map.lua')
    if check:
        print('script reads back fine: %r' % check[:40])


def main():
    global CRYPT
    CRYPT = init_crypt()
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('map')
    ap.add_argument('-o', '--out', default=None)
    ap.add_argument('--clean', action='store_true', help='hide the fake files from MPQ Editor')
    ap.add_argument('--listfile', default=None, help='known file names, one per line (hides twins too)')
    ap.add_argument('--aliases', action='store_true', help='free a hash table filled with extra entries (needs --listfile)')
    a = ap.parse_args()
    names = []
    if a.listfile:
        with open(a.listfile, encoding='utf-8', errors='replace') as f:
            names = [line.strip() for line in f if line.strip()]
    fix(a.map, a.out, a.clean, names, a.aliases)


if __name__ == '__main__':
    main()

[DER_RICHTER]

wow, bro is cooking

gj mate