Devo's Map Doctor - diagnose and remove map protection, open protected maps in the World Editor

[devoltz]

Devo's Map Doctor 1.1

Diagnose and remove the protection of Warcraft III maps, and make protected maps open in the World Editor.

Spoiler:

This is a small Windows program I made after fixing a lot of protected maps by hand, one hex editor session at a time. Everything I learned from those maps went into three buttons. You pick a map, and it tells you what was done to it, removes it, and makes a copy that the World Editor can open.

It is a single .exe of 12.8 MB. There is nothing to install and you don't need Python. Your original map is never changed.

The three buttons

1. Diagnose protection

• Lists every protection it finds, in plain English, and what the World Editor still needs to open the map. It only reads the map.

2. Remove protection

• Saves <map>_unprotected.w3x next to the map. The MPQ archive is fixed, so MPQ Editor opens it in edit mode, and the fake files are removed, so MPQ Editor no longer freezes trying to list 30,000 junk entries. You also get a real (listfile) with every file name the map itself mentions.

3. Make it open in World Editor

• Saves <map>_editor.w3x. It removes the protection first if there is any, restores the end of war3map.w3i that protectors cut off, and adds the files only the editor needs: triggers, custom script, regions, cameras, preplaced units and the imports list. The whole map script goes into the custom script section with //! inject main and //! inject config, so saving with JassHelper enabled builds the same script again.

Spoiler:

Version history

Spoiler:

1.1 -- this version is about the World Editor: the maps that the game runs and the editor refused to open now open, and they can be saved.

• The World Editor no longer crashes while opening a prepared map. The editor opens a map twice: it loads everything, and then opens the archive again and frees the first copy -- while a texture was still being read from that freed copy in the background. The read was late because some of the units the button had placed need a model that is a file INSIDE the map. Button 3 now leaves out of the editor's unit file the units whose model is a file the map itself carries and that no placed doodad uses: those are the ones that make the editor ask for a map file that late. Nothing changes in the game -- the units come from the map script, and this file is only read by the editor. On the reported map, 780 of its 824 units are placed and it opens and closes cleanly; the same map with all 824 still crashed. The report tells you how many units were left out.
• The map can be saved from the editor again. Saving was refused with Function redeclared: InitGlobals: the editor's own script declares the same functions the map script does (InitGlobals, CreateAllUnits, the unit creation of each player...), and two functions with the same name do not compile. The map's own copies of those names are now written with a devo_ prefix in the custom script, so the editor's ones and the map's ones live together. Checked with the Reforged JassHelper: the script compiles exactly as before, with no name declared twice.
• No more crash on the map's own doodad data. Some maps place doodads whose object ID exists in no data at all -- neither in the game data nor in the map's own object data. The game draws nothing for them and the editor reports Invalid object ID and goes down while opening the map. Button 3 removes exactly those doodads; every other doodad is copied byte for byte, and the report tells you the IDs and how many doodads each one affects.
• No more hang on "Loading Rects". An old protector wrote the text "..." into the record counter of the region file (war3map.w3r), so the editor tried to load 1,262,703,942 regions. The counter of every file only the editor reads -- regions, cameras, sounds, preplaced units, triggers, imports, minimap icons -- is checked and rewritten with the number of records the file really has.
• Units, regions and cameras are there when the map opens. The regions of a map whose region file the protector wiped are rebuilt from the map script (the CreateRegions function the game uses), weather and ambient sound included, and the preplaced units are rebuilt from the script's CreateAllUnits. The unit file is written in the map's own format -- measured on 271 real maps: 91-byte records for the 1.27 era, 111 (115 with the skin field) for the newer one -- and the units show up with their owner, their gold and their camp behaviour. A unit file in a format no real map uses is detected and rebuilt.
• The neutral units go to the right owners: 24 (Neutral Hostile) and 27 (Neutral Passive). The old numbers (12 and 15, the ones of the 1.27) show up in the 3.0 editor as "Player 13" and "Player 16", which are not neutral players at all -- that was the report about the team colours coming out wrong. Measured on 133 Reforged maps: every one of them uses 24 and 27.
• Lua maps work with button 3 now (the script goes into the custom script inside do ... end, and its last line keeps the map's own main and config running after you save -- see below), and maps packed with 512-byte MPQ sectors, which MPQ Editor refuses to open, are rewritten with normal 4 KB sectors.
• Files hidden behind language entries (the real file stored only under the game's language codes, and a decoy under the neutral entry that MPQ Editor and the editor open) are brought back, and replacing the file list no longer overwrites a real file that a decoy entry shared.

1.0 -- first release: the MPQ header, the fake header, the fake files, the decoy script, the virtual tables of PG2, the scrambled object IDs, SProtect and the file names.

What it removes

• A tampered MPQ header, which is why MPQ Editor opens a map read-only: the header size marks left by the Vexorian optimizer ("VxOP"), the w3p protector (02 01 20 00), PG2 and SProtect, a wrong format version, and hash or block tables placed before the header, including the negative block table position.
• A fake MPQ header in front of the real one, and an MPQ "user data" block where the HM3W header should be. The game skips both. Most tools don't.
• Fake files: the tens of thousands of junk entries the w3p protector adds, copies of real files under random names, extra fake names on real files (a hash table filled to 100%, so nothing can be added), and entries that point to data that doesn't exist. They stop being listed. The junk bytes stay inside the file as dead space, so the file size doesn't change.
• A decoy scripts\war3map.j that nothing can read, sitting next to the real script.
• Virtual file tables (PG2): tables that start inside the header and run past the end of the file, every name repeated several times, and decoy files named after public name lists. The map gets new tables with only the files it really uses, and the file data stays where it was.
• Scrambled object IDs (PG2): units, items and abilities renamed to four random bytes, with the script reaching them through sums of two rawcodes. Each one gets a clean ID in the object data, in every field that refers to it and in the script.
• SProtect: scrambled block indexes and the block table hidden under the header.

What it can't do

• Maps encrypted by the KK platform. The real map (script, terrain, objects, models) is stored encrypted outside the MPQ archive, and only the KK client can decrypt it. The program says so and stops.
• File names. Protectors delete the file list. The program gets back every name the map itself mentions: script, object data, models, textures, imports list, UI files and so on. A file that nothing mentions stays unnamed. It stays in the map, because the game may still load it, but MPQ Editor shows it without a name.
• Extra fake names on files it can't name are kept on purpose (Castle Fight has 119 such files). Without the real name there is no way to tell which entry the game uses, and deleting the wrong one breaks the map. The report tells you when this happens.
• The original GUI triggers. Optimizers and protectors compile them away, so you get the complete map script as custom script. Since 1.1 the regions, cameras and preplaced units that the script creates are rebuilt from it and do show in the editor; the items and abilities of a unit, and the sounds that only the script uses, still exist only in the game.

Is it safe?

• Your original map is never written to. The new copy goes next to it and never overwrites an existing file.
• Every copy is checked before it gets its final name: the files are read back and compared byte for byte with the original. If anything differs, nothing is saved.
• Only the archive's headers and tables are rewritten. File data stays byte for byte, except where changing it is the fix (the new object IDs and the script lines that use them).
• It doesn't connect to the internet and it doesn't change the game.

Tested on

1705 maps from 0.4 MB to 407 MB: w3p and w3pro maps (Zombie Defense, Castle Fight, Forest Defense, Sunken Defense, ORDR S2 with tens of thousands of fake files), Vexorian-optimized maps (WoW Reanimated, invalid Defense), a map with a broken header and a negative block table (Fish RPG), a PG2-protected RPG with virtual tables and 1,391 scrambled object IDs, SProtect (Heaven RPG S2), maps with entries that point to missing data, a fully encrypted archive, three clean maps (nothing to do, no false alarms).

The 1.1 changes were measured as well: 271 real maps for the layout of the preplaced unit file, 133 Reforged maps for the neutral owners, and the maps reported in this thread, opened and closed in the editor (the map of the crash report, and Legion TD 3.47g, which used to die on "Loading Units").

Small maps take a few seconds. A 100+ MB map with tens of thousands of fake files takes 2 to 5 minutes.

Spoiler:

Download

Thread attachment;
VirusTotal: https://www.virustotal.com/gui/file/371c56fd26fa516c2afc400884bf297da86474ec7594cea636ddaa0791d09c71?nocache=1

Some antivirus programs flag new, unsigned programs built with PyInstaller. It is a known false positive. The program only reads the map you pick and writes the new copy next to it.

If a map beats it, post the text from "Diagnose protection" and a link to the map, and I'll take a look. If the World Editor crashes, the newest folder under Documents\Warcraft III\Errors\ is what found this one.

Credits

• StormLib by Ladislav Zezula. The program follows its rules for reading MPQ archives, and its PKWARE DCL and WAVE decompressors are ports of StormLib code (MIT License; the notice ships with the program).
• WC3MapDeprotector by speige, for the ideas of repairing the cut war3map.w3i and of recovering file names from the map's own files.

[Vegas]

Wow, Excellent work!! i always wondered when someone would do this. xdprotect and nathan's has been obsolete for a decade at least. So this can be edited and saved?

It would be great to have and updated cheatpack injector now

[devoltz]

Wow, Excellent work!! i always wondered when someone would do this. xdprotect and nathan's has been obsolete for a decade at least. So this can be edited and saved?

It would be great to have and updated cheatpack injector now

Yeah, it can be edited and saved :p