Gildor's Forums

Author Topic: Injustice - Gods Among Us file format  (Read 51090 times)
sh0ck1
Newbie
*
Posts: 9


View Profile
Injustice - Gods Among Us file format
« on: April 05, 2013, 14:43 »

FYI, the IOS game is DIFFERENT from the Xbox360/Playstation 3 game.

The PS3 / Xbox game is of interest to me. Not sure if its just me never using this before, but I could not get it to work at all with umodel.

******** CHAR_Batman.xxx ********
*** ERROR: Trying to allocate 1163022155 bytes
appMalloc:size=1163022155 <- FArray::Empty:1163022155 x 1 <- FString<< <- FPacka
geFileSummary<<:Ver=573/49 <- UnPackage::UnPackage:CHAR_Batman.xxx, ver=573/49,
game=8000 <- UnPackage::LoadPackage:CHAR_Batman.xxx <- Main

--------------------------------------------------------------------------------------------

Edit: On second look, it looks like the error is the same no matter what character model I try to extract (exact # of bytes for allocation).
« Last Edit: April 05, 2013, 14:50 by sh0ck1 » Logged
Gildor
Administrator
Hero Member
*****
Posts: 7924



View Profile WWW
Injustice - Gods Among Us file format
« Reply #1 on: April 05, 2013, 15:43 »

The package format is changed, so umodel crashed immediately when opening the package.
Logged
sh0ck1
Newbie
*
Posts: 9


View Profile
Injustice - Gods Among Us file format
« Reply #2 on: April 06, 2013, 03:09 »

Yeah, I've been looking into it with a hex editor (though I'm a total noob at this) and all the files start off similar to

9E2A83C1 0031023D 0006A08D 45524F4B 00000042 00000005 4E6F6E65 00028800 0900000A 1B000007 F5000011 19000190 69000001 4000016D 69000000 0400065C...

The amount umodel is trying to allocate is the 4th double word, but that value is a generic value in all the files ("EROK").
I know the first set is the UE3 identifier/tag. Also, the 7th set is the "PackageIdentifier" aka None from the older UE files.

Been using Gildor's posts in this thread as a guide:
http://www.gildor.org/smf/index.php/topic,882.0.html

But yes, package format is definitely changed. I have a question though. Since the package format is changed, does this imply that UE has a new package format or does it imply that the game's developers have customized it?
---
BTW Gildor, thanks for the response! I've been trying to learn to reverse engineer off and on for awhile, but your work here is incredible. I hope you release the source code one day, but until then, I will continue to try and figure things out myself! Keep going strong!
« Last Edit: April 06, 2013, 03:12 by sh0ck1 » Logged
Gildor
Administrator
Hero Member
*****
Posts: 7924



View Profile WWW
Injustice - Gods Among Us file format
« Reply #3 on: April 06, 2013, 13:41 »

Answering to your question. That's because format was changed. Perhaps they just inserted a single dword, some id, but this brokes loading completely.
And thank you for the good words Smiley
Logged
howfie
Newbie
*
Posts: 34


View Profile
Injustice - Gods Among Us file format
« Reply #4 on: April 07, 2013, 21:25 »

Yo gildor, I got an unpacker for this game but so far it's pretty useless as it would just dump a million useless binary files lol. I'm stumped at giving these files filename extensions using the import/export/string tables. Here's what I got so far with the samples someone sent me.

What I do is take all the XXX files that are compressed, and convert them into XXX files with uncompressed data. It's really easy. Decompress LZO1X data, resave header, string tables, and uncompressed data, and then zero out compression flags and voila you got uncompressed XXX files with all offsets intact.

The export tables are also pretty easy to process they look like:

Code:
0xffffff70 0x00000000 0x00000d27 0x000009ca 0x00000000 0x00000000 0x00000000 0x000f0004 0x00000000 0x000301ee 0x00dae473 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0xffffff70 0x00000000 0x00000d27 0x000009cb 0x00000000 0x00000000 0x00000000 0x000f0004 0x00000000 0x0006023f 0x00dde661 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0xffffff70 0x00000000 0x00000d27 0x000009cc 0x00000000 0x00000000 0x00000000 0x000f0004 0x00000000 0x000601ee 0x00e3e8a0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

in first line, 0x000009CA refers to name table:
Code:
property 0x09c9: WonderWoman 0x00000007 0x00000010 0x00000000
property 0x09ca: WonderWoman_Accessory_Diff 0x00000007 0x00000010 0x00000000
property 0x09cb: WonderWoman_Accessory_Norm 0x00000007 0x00000010 0x00000000
property 0x09cc: WonderWoman_Accessory_Spec 0x00000007 0x00000010 0x00000000
property 0x09cd: WonderWoman_BoneSockets 0x00000007 0x00000010 0x00000000

also in first line,
Code:
0x000301ee = size of data
0x00dae473 = offset to data

So it's easy to get the data out. But figuring out what's a texture, model file, etc. is kind of cryptic. At first I thought 0xffffff70 must be textures, but it's not that. And I've looked at the other parameters too and I just don't see it. Got any advice?

There are also two sets of other string tables that look like:

Code:
There are 0x0037 string entries.
0x0000: EngineMaterials
0x0001: EngineResources
0x0002: MwyFastPost
0x0003: MK9_EngineResources
0x0004: DCF_CharMatLibrary
0x0005: FX_Character
0x0006: FX_Global
0x0007: MwyEngineResources
0x0008: EngineFonts
0x0009: SystemArt
0x000a: NPC_MaleLarge
0x000b: DCF_SocketLibrary
0x000c: DCF_Cine_SocketLibrary
0x000d: DCF_IkSkeletonLibrary
0x000e: NPC_MaleMedium
0x000f: FX_Environment
0x0010: FX_Clash
0x0011: FX_Impact
0x0012: FX_Blood
0x0013: DCF_FX_MatLibrary
0x0014: FX_Training
0x0015: ReverbPresets
0x0016: CHAR_WonderWoman
0x0017: SND_FOLEY_Leather
0x0018: SND_SFX_Wonderwoman
0x0019: SND_SPEECH_WonderWoman_eng
0x001a: SND_VO_WonderWoman
0x001b: FX_Prop_WonderWoman
0x001c: DCF_FX_PropMatLibrary
0x001d: NPC_FemaleLarge
0x001e: FX_WonderWoman
0x001f: FX_Melee
0x0020: FX_Scenario
0x0021: NPC_Amazonian
0x0022: FX_PropSwap_WonderWoman
0x0023: ui_i_SM_WonderWoman
0x0024: WonderWoman_Clash
0x0025: Anims_Clash_WonderWoman
0x0026: WonderWoman_Expressions
0x0027: WonderWoman_Supermove
0x0028: DCF_SupermoveLibrary
0x0029: FX_Prop_NPC_Amazonian
0x002a: NPC_MaleSmall
0x002b: Anims_Supermove_Wonderwoman_V1
0x002c: Generic_Cloth
0x002d: MaleMedium_Expressions
0x002e: Amazonian_Expressions
0x002f: NPC_FemaleSmall
0x0030: Anims_HawkGirl
0x0031: CHAR_Hawkgirl
0x0032: Catwoman_Expressions
0x0033: CHAR_Catwoman
0x0034: SND_COMBAT_All
0x0035: MidwayEditorMeshes
0x0036: EditorMaterials

Code:
There are 0x0006 string entries.
0x0000: Engine
0x0001: Core
0x0002: SystemArt-d9a5d101
0x0003: MwyDecal
0x0004: CHAR_WonderWoman-fc548cbb
0x0005: MwyCinema

but these are like directory names or something and don't really tell me what type of data the data is in the export tables.
« Last Edit: April 07, 2013, 21:31 by howfie » Logged
howfie
Newbie
*
Posts: 34


View Profile
Injustice - Gods Among Us file format
« Reply #5 on: April 07, 2013, 21:45 »

nvm lol, i think i figured it out... the data about the type of data is actually somewhere else, in some table usually at the bottom of the file.

so once I get these files extracted, is there a way to run them through umodel?
Logged
sh0ck1
Newbie
*
Posts: 9


View Profile
Injustice - Gods Among Us file format
« Reply #6 on: April 08, 2013, 09:59 »

Hey howfie, congrats on your work. I'm very new to all of this, and everything you all do here is exciting to me.  Everything you just posted is very helpful to me learning. I'm curious as to what tools you use? If we need to take this to PM's thats fine as well. I've been experimenting with QuickBMS, Offzip, CrypTool but I'm not sure what others use. I can program in C++ and Java as well but I'm not sure how best to approach any of this.

Also, do you plan to release your unpacker? Or the source code for it (Im very interested in learning about this topic, so if you are okay to PM me src, I will keep it private)? If you are not releasing anything, could you guide me towards what bytes in the header are for?

I'm a little embarrassed to post what I had pieced together from everywhere else on this forum, but here goes:
int32 1 = TAG
int32  2 = unknown (looks to be the same always / File version?)
int32 3 = Possibly headersize - (Location of something about the file?) - looks like 8 bytes
int32 4,5,6,7 = "EROK" Package group info ends in None (4 longs for String?)
... (below from Gildor in another thread) ...
int32 8 = PackageFlags
int32 9 = NameCount
int32 10 = NameOffset
int32 11 = ExportCount
int32 12 = ExportOffset
int32 13 = DependsOffset - When PackageVersion >= 415
int32 14,15,16,17 = GUID
int32 18 = Generations.Count
then generations
...
...
...
int32 ExportCount
int32 NameCount
int32 NetObjectCount
...
...
...

Finally
int32 EngineVersion
int32 CookerVersion
int32 CompressionFlags
... compressed chunks
... some other fields ...


Finally, last question, you said "Decompress LZO1X data, resave header, string tables, and uncompressed data, and then zero out compression flags - and this leaves offsets intact"

Shouldn't decompressing data change offsets?
Logged
howfie
Newbie
*
Posts: 34


View Profile
Injustice - Gods Among Us file format
« Reply #7 on: April 08, 2013, 10:54 »

of course i will release it if gildor chooses not to work on it. but if he has time then i will probably not.
current source is here.
https://code.google.com/p/sticklove/source/browse/trunk/source/xentax/ripper/xb360_injustice.cpp

Quote
Finally, last question, you said "Decompress LZO1X data, resave header, string tables, and uncompressed data, and then zero out compression flags - and this leaves offsets intact"
Shouldn't decompressing data change offsets?

no, there are two types of 0x0031023D packages. one has compressed data and one has uncompressed data. the one with compressed files has an extra "list of compressed sizes offsets." you can verify this by looking at them. GalleryArt1.xxx is uncompressed type. CHAR_WonderWoman.xxx is compressed type.

Once uncompressed, the first EXPORT entry in CHAR_WonderWoman.xxx is

Code:
0x0000: 0xffffff7a 0x00000000 0x00000000 0x00000712 0x00000001 0x00000000 0x00000000 0x000f0000 0x00000000 0x00032767 0x00078ca6 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

So the data starts at offset 0x00078ca6, so WTF comes before 0x78CA6? Well take and copy CHAR_WonderWoman.xxx and delete all the compressed data from offset 0xD99 (the first compressed package entry) to the end of the file. You now have a file that is 0xD99 bytes lol.

Now let's delete the compression entries.
Set the 00000002 value to 00000000 at offset 0x6D. That be the compression flag.
Set the 00000044 value to 00000000 at offset 0x71. That be the number of compressed packages that are in the file.
Now delete everything from offset 0x75 to 0x4B4 (a total of 0x440 bytes). This is the list of compressed entry informations. Each entry is 0x10 bytes and there are 0x44 of them.

Now you have a file that is 0x959 bytes. Now look at the first few lines of your file. Do you see 0x959 somewhere? (HINT: Nod yes at offset 0x25 lol). We've converted the XXX header (compressed) to one that is uncompressed.

But still we ain't got to 0x78CA6 yet. Where do we get that? Well, if you decompress the first XXX package that was in the file, you will notice the compression entry (the first one in the list that we deleted)

Code:
00000959 0007834D 00000D99 0001BC6D

first value is 0x959, the position in the decompressed XXX for which this data starts (we got that number already).

second value is size of decompressed data.

third value is 0xD99 (remember i said delete all the compressed data starting from 0xD99?). this is the offset in the compressed XXX file for which this compressed data starts.

fourth value is 0x1BC6D, the size of the compressed data.

So tell me, what is

0x959 + 0x7834D = Huh??

Yeah, fuck yeah, 0x78CA6 lol. See? If you just saved all the decompressed data to an empty file, all the offsets in the EXPORT table will be WRONG.
Logged
howfie
Newbie
*
Posts: 34


View Profile
Injustice - Gods Among Us file format
« Reply #8 on: April 09, 2013, 09:51 »

jesus mother of freaking god gildor, how do keep track of all this bookkeeping of the object properties? for example, look at MotionTrackTotalOffset entry below... it isn't 0xC bytes of data, so in order to read these entries you have to literally bookkeep all of the different types of objects in some kind of data structure to keep track of these entry sizes! there's like thousands of these objects!

Code:
export entry 0x0172: 0xfffffff0 0x00000000 0x00000172 0x0000001f 0x00000000 0x00000000 0x00000001 0x00070000 0x00000000 0x00031cb1 0x000f8227 0x00000000 0x00000001 0x00000000 0x00000000 0x00000000 0x00000000

data at 0x000f8227 is:
00 00 05 C8 00 00 00 00 00 00 03 AF 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 1F 00 00 00 00  (SequenceName, NameProperty)
00 00 03 CF 00 00 00 00 00 00 03 AF 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 4C 00 00 00 02  (OriginalPackage, NameProperty)
00 00 03 9E 00 00 00 00 00 00 06 2F 00 00 00 00 00 00 00 0C 00 00 00 00 00 00 07 07 00 00 00 00 C4 73 10 00 00 00 00 00 00 00 00 00 (MotionTrackTotalOffset, StructProperty)
00 00 05 AA 00 00 00 00 00 00 00 C5 00 00 00 00 00 00 00 01 00 00 00 00 01 (RotationCompressionFormat, ByteProperty)
00 00 01 2C 00 00 00 00 00 00 00 54 00 00 00 00 00 00 04 C4 00 00 00 00 data (CompressedTrackOffsets, ArrayProperty)
00 00 03 4A 00 00 00 00 00 00 02 AD 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 03 (m_nVersion, IntProperty)
00 00 03 BA 00 00 00 00 (None)
+ more data
« Last Edit: April 09, 2013, 09:54 by howfie » Logged
Gildor
Administrator
Hero Member
*****
Posts: 7924



View Profile WWW
Injustice - Gods Among Us file format
« Reply #9 on: April 09, 2013, 13:01 »

I think the object property format is more complex than you think. Why don't you just look at open-source projects which are already figured our how to work with them? For example, ME3 Explorer - this project even has editing capabilities which are so important for many visitors of this forum for reasons I can't understand.
Logged
howfie
Newbie
*
Posts: 34


View Profile
Injustice - Gods Among Us file format
« Reply #10 on: April 09, 2013, 13:18 »

oh cool, thx for the reference!

EDIT:
lol, actually it's very simple I just misinterpreted where the starting offset for the StructProperty data was at Smiley.
« Last Edit: April 09, 2013, 15:29 by howfie » Logged
sh0ck1
Newbie
*
Posts: 9


View Profile
Injustice - Gods Among Us file format
« Reply #11 on: April 09, 2013, 16:54 »

hey Gildor, I actually did try ME3 Explorer and it doesn't work with these files either (I renamed them to pcc, upk etc). The ME3 wiki is really nice though and the PCC format looks almost identical to these xxx files, its just that their software requires files specific to ME3 to function correctly.

Thx for the src howfie. Finally got it to build. Been awhile since I've touched C++ and booster is crazy. Its going to take me awhile to figure out how to decipher these files, but is it possible that there are more layers of nesting? If one xxx can contain another xxx, whose to say the next one cant contain another? Could there be another export table of offsets etc? I'll be popping into your svn occasionally to check out your progress. Keep up the good work!
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7924



View Profile WWW
Injustice - Gods Among Us file format
« Reply #12 on: April 09, 2013, 17:40 »

I actually did try ME3 Explorer and it doesn't work with these files either (I renamed them to pcc, upk etc).
Of course! It shouldn't work because ME3 Explorer understands only Mass Effect version of upk format. But it's much closer to whatever-you-want-to-do than starting from scratch.
Quote
The ME3 wiki is really nice though and the PCC format looks almost identical to these xxx files, its just that their software requires files specific to ME3 to function correctly.
pcc, xxx, upk, u - these are just file name extensions. Format is the same with variations depending on game.
Quote
is it possible that there are more layers of nesting? If one xxx can contain another xxx, whose to say the next one cant contain another?
There's no nesting at all. Package compression could be implemented on 3 levels:
1) whole package compression (usually used for .u or startup* files), looks like upk in some compressed archive
2) ordinary package compression - compressed using tables in package header
3) compression of bulk data - used for textures and sounds, and for some other types
Quote
Could there be another export table of offsets etc?
No.
Quote
I'll be popping into your svn occasionally to check out your progress. Keep up the good work!
I still have no PC at home. Currently I'm typing this message from my work, it would be crazy to type such message using Android keyboard.
« Last Edit: April 11, 2013, 12:45 by Gildor » Logged
mkhacker
Newbie
*
Posts: 13


View Profile
Injustice - Gods Among Us file format
« Reply #13 on: April 09, 2013, 19:26 »

Awesome work howfie and of course Gildor...
I'm not sure how to execute you .cpp file, I notice the script includes a lot of .h resources some i'm guessing it needs to be compiled to be executed...
Any chance of a .bms script to extract the textures? Or do I need to wait for Gildor's next umodel release Wink

Logged
sh0ck1
Newbie
*
Posts: 9


View Profile
Injustice - Gods Among Us file format
« Reply #14 on: April 10, 2013, 19:32 »

Sort of a general question but definitely applies to this game - the art assets are usually stored in the assets folder etc, and the sounds separately as well (probably in a Sound folder). Where is the game data stored though? For example, in a fighting game like this, there should be a lot of data that stores attack speed, damage per move, input command to do the move, as well as the moves properties (whether its a low attack or a high attack etc)

I saw that for Mass Effect 3 on PC that they stored a lot of gameplay values/flags in the coalesced.ini file. This game is built on MK9 engine, and MK9 coalesced.eng didn't have any data like this and MK9 coalesced.ini didn't have any of these values either.  Anyone have an idea where this may be stored?
Logged
Jump to:  

Powered by SMF | SMF © 2006-2009, Simple Machines LLC
Leviathan design by Bloc | XHTML | CSS