The Deserialization VM: The Pre-VM Stage of a Lua Virtualizer

Execution splits into three stages: a pre-VM stage that deserializes the raw blob, the real VM that runs the program, and a post-VM stage for error handling and return values. The pre-VM begins by normalizing and decompressing the blob, then parses the constants, then the function prototypes (each prototype's instructions extracted by a "get next function instructions" routine), clears the globals used as scratch interface, and reads the entrypoint index. Its output is one table whose fields you can map by matching against the plaintext deserialize routine: Insts, the register tables REG_A/REG_B/REG_C (all indexed by the virtual IP), function_prototypes, stk_size, and two constant tables. The raw constants table is still encrypted for the real VM; the decrypted_constants table holds them after runtime decryption - a concrete anchor for locating the decryption routine.

A subtle protection nearly blocks analysis: simply calling print crashes, regardless of arguments. The reason is that print internally calls tostring, and in Lua even strings have a metatable - so the VM hijacks __tostring (via debug.setmetatable("", ...) / getmetatable("").__tostring) to detect and punish inspection. Two workarounds combine: use io.write (which does no formatting, like C's printf) plus a custom recursive tostring for tables, and - the real fix - patch the deserializer so that during constant parsing any constant equal to "__tostring" is overwritten with an empty string, at the earliest point constants are plaintext. That defuses the trap before the deserialization VM can arm it.

Logging every opcode shows ~20,000 deserialization instructions execute before the real VM, which itself runs ~5,000. A defining design choice appears in how closures are handled: what looks like OP_RETURN (return true, REG_C[VIP], 0) is actually a function dispatcher. Rather than nesting a child VM inside the parent (the IronBrew model, where both stay alive), the VM runs inside a pcall and returns metadata signalling "continue in this closure". The enclosing VM is stopped before the next is dispatched, so only one VM instance exists at any moment - significantly cutting memory use. A second, hidden path invokes closures through a metatable on the prototypes table. Understanding this stage is the groundwork for intercepting the real instruction stream rather than reversing the deserializer in full.