Gildor's Forums
Gildor's Forums > English Forum > UE Viewer (English) > Unreal coding (Moderator: Juso3D) > UPK file format (again...)
Pages: [1] 2
Author Topic: UPK file format (again...)  (Read 20863 times)
michg84
Newbie
*
Posts: 12


View Profile
UPK file format (again...)
« on: March 15, 2012, 23:22 »
Hi all,

I'm trying to extract 2D textures from upk files inside an application. I've read Gildor's advices about upk files (http://www.gildor.org/smf/index.php/topic,297.msg2859.html) and I can read header, compression flags, compressed chuncks and compressed chunks blocks.
Where I get lost is where do I find the first compressed chunk ? Is it right after the last compressed chunk block ?
If so, when I try to uncompress the first block, I get an exception. I'm using LZO.net compressor/decompressor as my upk files are compressed with LZO.

Any idea will be of great help.

Thanks.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7969



View Profile WWW
Re: UPK file format (again...)
« Reply #1 on: March 16, 2012, 08:09 »
Hi.
The algorithm has description in the thread you mentioned above:
http://www.gildor.org/smf/index.php/topic,297.msg2873.html#msg2873
Logged
michg84
Newbie
*
Posts: 12


View Profile
Re: UPK file format (again...)
« Reply #2 on: March 16, 2012, 15:00 »
Yes, I read your algorithm and I integrated it in my application.
So when I get Pos onto the first compressed chunk block and read CompressedSize, I assume the reading is correct as the compressed block ends with "11 00 00".
When I pass this block to lzo.uncompress in LZO.net, I get an exception.
Which compression library are you using ?
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7969



View Profile WWW
Re: UPK file format (again...)
« Reply #3 on: March 16, 2012, 15:07 »
Quote from: michg84 on March 16, 2012, 15:00
I assume the reading is correct as the compressed block ends with "11 00 00".
11 00 00 means nothing
Quote
When I pass this block to lzo.uncompress in LZO.net, I get an exception.
Which compression library are you using ?
LZO (C-version)
Logged
michg84
Newbie
*
Posts: 12


View Profile
Re: UPK file format (again...)
« Reply #4 on: March 16, 2012, 21:24 »
OK, found the problem. LZO.net can only decompress what it compresses Huh?
Got Oberhumer sources (lzo 2.0.6) and recompiled lzo.dll
So everything went fine, I managed to uncompress the whole file. Now I've to deal with export an import tables.
Is there reliable information here : http://wiki.beyondunreal.com/Legacy:Package_File_Format ?
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7969



View Profile WWW
Re: UPK file format (again...)
« Reply #5 on: March 16, 2012, 21:54 »
That information corresponds to Unreal engine 1 and 2.
Logged
michg84
Newbie
*
Posts: 12


View Profile
Re: UPK file format (again...)
« Reply #6 on: March 17, 2012, 15:11 »
So name table is quite simple to understand. Import and export table don't.
Has each game is own structures or are they all identicals ? If so, would you mind to share some information about these structures ?
Thanks.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7969



View Profile WWW
Re: UPK file format (again...)
« Reply #7 on: March 17, 2012, 15:48 »
Most games used "standard" import/export tables. Some games has modifications there. Even name table could be modified in some games.
Logged
michg84
Newbie
*
Posts: 12


View Profile
Re: UPK file format (again...)
« Reply #8 on: March 17, 2012, 18:14 »
I realize my question was maybe stupid  Huh? as umodel can extract 2D textures from this game (actually it's The Last Remnant)
From what I see in the uncompressed file :
Name Table is variable length, with a string and 2 int
Import Table is 28 bytes long based on (Export Offset - Import Offset) / Import Count
Export Table should be 74 bytes long based on (DependsOffset - ExportOffset) / ExportCount but this doesn't match the pattern in the file. The pattern seems to say Export Table is 72 bytes long, but then there's extra bytes at the end of table. Who's right and who's wrong Grin
Logged
michg84
Newbie
*
Posts: 12


View Profile
Re: UPK file format (again...)
« Reply #9 on: March 18, 2012, 22:09 »
Despite my efforts, I couldn't get any information about import and export tables structure. Could you please give me the standard structures for these tables as you did for the header file.
Thanks.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7969



View Profile WWW
Re: UPK file format (again...)
« Reply #10 on: March 18, 2012, 22:55 »
Export entry for original engine:
Code:
Ar << E.ClassIndex << E.SuperIndex << E.PackageIndex << E.ObjectName << E.Archetype; // int, int, int, Name, int
Ar << E.ObjectFlags;    // int64
Ar << E.SerialSize;     // int
Ar << E.SerialOffset;   // int
if (Ar.ArVer < 543)  Ar << E.ComponentMap; // TMap<Name, int>
Ar << E.ExportFlags;    // int
Ar << E.NetObjectCount; // TArray<int>
Ar << E.Guid;           // Guid
Ar << E.U3unk6C;        // int
Import entry for UE2 and UE3 (identical for all games except a few UE2-based game titles):
Code:
Ar << I.ClassPackage << I.ClassName << I.PackageIndex << I.ObjectName; // Name, Name, int, Name
Logged
michg84
Newbie
*
Posts: 12


View Profile
Re: UPK file format (again...)
« Reply #11 on: March 19, 2012, 00:56 »
Thanks a lot for the serialization.
I guess Name is 64 bits long since my Import table is 28 bytes long (couldn't find type definition).
I guess also Ar.ArVer is package version, so mine is 507 and E.ComponentMap is present. Type TMap is int.
So export table seems to be variable size, depending on NetObjectCount. This explain the extra bytes I found when assuming constant size.

Well, I'll try to make my way through this.

Thanks again for your help.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7969



View Profile WWW
Re: UPK file format (again...)
« Reply #12 on: March 19, 2012, 08:29 »
Name is 2 32-bit integers.
TMap<type1, type2> is identical (by serialization) to TArray<SomeStruct>, where SomeStruct is
Code:
struct SomeStruct
{
    type1 a;
    type2 b;
};
TArray<type> is "int count" + "type[count]" records.
Quote from: michg84 on March 19, 2012, 00:56
So export table seems to be variable size, depending on NetObjectCount. This explain the extra bytes I found when assuming constant size.
Almost everything in Unreal is variable size (this is confusing for people who asking "what is placed on offset ...")
Logged
michg84
Newbie
*
Posts: 12


View Profile
Re: UPK file format (again...)
« Reply #13 on: March 19, 2012, 16:38 »
Thank you for correcting my mistake about TMap.

Now i can read complete export table and find the object that interest me.
If I seek to object SerialOffset (0x2A10E), I found what I suppose to be object properties. Once again, I've found nothing accurate for object properties in UE3.
Here's a dump of the concerned area. Object begins at 0x2A01E and is supposed to be a 2D texture.



The bitmap itself seems to begin at 0x2A2D7 as I see a repetitive pattern of 8 bytes which is accurate as the picture is an area map with mainly black color.

Could you please help me decoding these properties or give me the generic texture structure format (if it applies).
Thanks.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7969



View Profile WWW
Re: UPK file format (again...)
« Reply #14 on: March 19, 2012, 18:18 »
Did you read the properties block?
Logged
Pages: [1] 2
Jump to:  

Powered by SMF | SMF © 2006-2009, Simple Machines LLC
Leviathan design by Bloc | XHTML | CSS