Gildor's Forums

Author Topic: Universal Decooker  (Read 8842 times)
Koomazaz
Newbie
*
Posts: 7


View Profile
Universal Decooker
« on: April 16, 2020, 01:49 »

Konstantin,

I've been trying to gather information about the possibility of decooking files from one platform and recooking them for another platform.

I understand there are not currently any tools that can do a proper decooking, and you said in another forum post that it would likely take a year to develop such a tool.

I am interested in attempting to build a decooking tool.  I am a professional coder, but I have very little experience with the Unreal Engine file structures.

Would you be able to provide some guidance on how to get started on such a project?  Even some helpful documentation or notes would be helpful.  I'm willing to learn, but I need some help on getting started.  Are there any tools that get part of the way there?  What are the current roadblocks?

Thanks,
Koomazaz
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7971



View Profile WWW
Re: Universal Decooker
« Reply #1 on: April 16, 2020, 12:01 »

Initially there's just one uasset file. Epic decided to split it to .uasset and .uexp - the first one contains file header, second one - all the data. When reading uasset, engine transparently appends uexp to uasset. The logic is: when attempt to read data at position which is larger than uasset file size, it subtracts uasset file size, and reads uexp at computed position.

This was done for better loading times: when cooking the game and packaging it into pak file, engine can reorder files, so when loading it will do less seek operations. For instance, it will put all uasset files sequentially, so scanning of asset information will be done quickly, because one uasset HEADER goes after another one. Then engine requests contents of asset, and again - it will read DATA, possibly without any seek operation.

Similar with ubulk: cooker puts lowest resolution mips for textures and lower LODs of meshes into uexp (data), so level startup is fast. Level may appear blurry and lowpoly-ish. Then, when level is already started, engine loads higher resolution data is loaded from separate ubulk files, with immediate update of the screen picture.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7971



View Profile WWW
Re: Universal Decooker
« Reply #2 on: April 18, 2020, 10:15 »

Although I understand the philosophical design of this approach,
Yes I assumed that you MAY understand this. I posted the information just in case Smiley

Quote
my current unknown is how to unserialize the byte array from .umap/.uexp back into usable unreal objects.
I suppose you meant "usable in Unreal Editor". It's not "unserialize". It's rather "uncook" (there's no official term for that) - to convert cooked object to non-cooked one.

Quote
In SavePackage.cpp in the unreal source code there is

Code:
FStructuredArchive::FRecord ExportRecord = ExportSlot.EnterRecord();
Export.Object->Serialize(ExportRecord);
FStructuredArchive is relatively recent addition. Honestly I didn't understand the concept of it, it was much easier before, when everything was FArchive. I didn't dig deeper into the code to understand what it does and WHY "structured archive" was added to the engine, so I rather can't answer your question.

Quote
I'm assuming something along the lines of
Code:
FArchive <<
... Anyways I'll poke around some more :-)
All of serialization in UE (both reading and writing) is done with Object::Serialize(FArchive& Ar) or in operator<<(FArchive, Object).
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7971



View Profile WWW
Re: Universal Decooker
« Reply #3 on: April 18, 2020, 11:13 »

1. You're looking at VERY basic serialization level. You should look at ULevel::Serialize perhaps.
2. The video is about nothing.
Logged
Juso3D
Moderator
Hero Member
*****
Posts: 910


Welcome to Gildor Forums


View Profile
Re: Universal Decooker
« Reply #4 on: April 30, 2020, 17:06 »

Very Cool!
Logged

Blender + Linux User Here.
Juso3D
Moderator
Hero Member
*****
Posts: 910


Welcome to Gildor Forums


View Profile
Re: Universal Decooker
« Reply #5 on: May 01, 2020, 14:46 »

All this goes through my head, but would be awesome to try out in the future.
Logged

Blender + Linux User Here.
Gildor
Administrator
Hero Member
*****
Posts: 7971



View Profile WWW
Re: Universal Decooker
« Reply #6 on: May 01, 2020, 14:54 »

Now, Kostya, for my own sanity can you concur? Hehehe!
Level serializer is the area where I never looked at. However I see you have problems with different engine versions etc, so you probably understand the difficulty of doing thing which will work for more then just one game Smiley
Also, I believe you're not dealing with BSP (brushes) etc, just with static meshes placed on the level.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7971



View Profile WWW
Re: Universal Decooker
« Reply #7 on: May 04, 2020, 09:40 »

You'd better record a video (at least timelapse) showing the process of conversion Smiley
Logged
Digika
Jr. Member
**
Posts: 44


View Profile
Re: Universal Decooker
« Reply #8 on: May 04, 2020, 10:33 »

Quote
So I'm thinking we may manually need to build a map of custom properties and its respective sizes so that when the unknown property kicks in, at least UnderlyingArchive.Seek(StartOfProperty + Tag.Size); can be synced properly

This dont always works, some properties can pad stuff/content and even if Tag size says it is 9 the actualy bytes can be 10 until next property starts (TextProperty does that, for example)
Logged
iridium
Newbie
*
Posts: 10


View Profile
Re: Universal Decooker
« Reply #9 on: May 04, 2020, 14:48 »

sorry to break into this topic unexpectedly shocked, but is there any chance that we can "decook" maps in the future from old versions of the engine? 4.8, 4.12 etc, for example.
or it will be most likely on the latest ue4 versions?
Logged
Juso3D
Moderator
Hero Member
*****
Posts: 910


Welcome to Gildor Forums


View Profile
Re: Universal Decooker
« Reply #10 on: May 04, 2020, 16:25 »

Be good if you can get some Material Function related stuff.
Logged

Blender + Linux User Here.
Digika
Jr. Member
**
Posts: 44


View Profile
Re: Universal Decooker
« Reply #11 on: May 06, 2020, 12:56 »

Quote
an array of strings
That is FName pool

Quote
an array of import structures, what ever they are
an array of export structures, whatever they are
Import and export objects.

I think 90% of Uassets use the same package summary header that has the meta about the asset.
Logged
Soulax
Newbie
*
Posts: 8


View Profile
Re: Universal Decooker
« Reply #12 on: May 30, 2021, 17:15 »

I understand there are not currently any tools that can do a proper decooking, and you said in another forum post that it would likely take a year to develop such a tool.

Here is a lil script that uncook few stuff (at least in 4.22) Smiley You need to use it as UE4 Editor Utility Widgets !

Feel free to improve it, it's entirely experimental, so I can't really vouch for its stability.


* AssetDecooker.zip (2.76 KB - downloaded 581 times.)
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7971



View Profile WWW
Re: Universal Decooker
« Reply #13 on: May 30, 2021, 18:48 »

This is not a "script", it's C++ code Smiley
Logged
Soulax
Newbie
*
Posts: 8


View Profile
Re: Universal Decooker
« Reply #14 on: May 30, 2021, 21:23 »

This is not a "script", it's C++ code Smiley

 rolleyes  Grin
Logged
Jump to:  

Powered by SMF | SMF © 2006-2009, Simple Machines LLC
Leviathan design by Bloc | XHTML | CSS