March 29, 2024, 00:45
bigger smaller reset     1020px Wide width Full width Reset   * *

Gildor's Forums

  Homepage Facebook Read news on Twitter Youtube channel Github page
Welcome, Guest. Please login or register.
Did you miss your activation email?

« previous next »
Print
Author Topic: TFC file compression  (Read 38981 times)
Daemonjax
Newbie
*
Posts: 4


View Profile
« on: November 02, 2012, 23:02 »

EDIT - LOL - I got it (I think).  They used all three compression types, but most of the files were compressed with LZO.  It is chugging away and I am up to 2.2GB of decompressed files.  Thanks for the tool!

I'm also trying to figure out how to decompress these .tfc files. 

The reason we need to actually decompress these files is because it is the only way we'll be able to create distributable texture mods for it, by creating and applying a differential patch to it.  I can write that program, but I need to uncompress the tfc files first.

4Aces... Could you please explain the process you used to decompress the tfc?  If not 4Aces, then anyone... any help here would be appreciated.  Yes I read the FAQ and followed the links, either: the information isn't there, I'm an idiot, or it's so buried that it's become less than useful.

Are these tfc files compressed, or are they just known formats concatenated together?

I believe the tfc are compressed due to the headers of these files... for example (textures.tfc):
Code:
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  C1 83 2A 9E 00 00 02                             Áƒ*ž...

That matches the compressed headers in upk files.  When I uncompress those upk files, the header's different.  So, I think these tfc files are compressed.  Furthermore, it appears to be with lzo. 

I'f I'm wrong, someone tell me.  It would make things a lot easier if they weren't actually compressed to begin with!  I'd prefer to be wrong about this.

I can't create a program that creates useful diff patches for tfc files for other modders without being able to perform a comparison search for the original uncompressed data.  Without a way to distribute texture mods piecemeal via patches to an uncompressed tfc, the future of texture mod scene for xcom:eu looks rather bleak indeed.  Well... at least there's texmod.

I'm even willing to write a compatible decompression tool, if Gildor would willing to answer the many questions I would have on the subject.  Getting the game to load uncompressed tfc files could be a problem, but I prefer to take one step at a time.

Anyways... Thanks in advance for any info.  I bookmarked this thread.

« Last Edit: November 03, 2012, 00:34 by Daemonjax » Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #1 on: November 03, 2012, 09:47 »

TFC files are concatenated compressed texture data blocks with 32Kbyte alignment (not sure about exact alignment value though).
Logged
Daemonjax
Newbie
*
Posts: 4


View Profile
« Reply #2 on: November 03, 2012, 10:18 »

Good info, thanks. Cheesy

Just to make sure... you're referring to  LZO compression using 32kb block sizes, correct?

Or, do you mean something else entirely?
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #3 on: November 03, 2012, 10:20 »

No. Each mipmap starts at position which is multiply of 0x8000 bytes.
Logged
Ashryu
Newbie
*
Posts: 1


View Profile
« Reply #4 on: November 03, 2012, 12:49 »

Because giving as little information as possible, presented as unclearly as you can make it, gives great opportunities to talk down to people.  It's the Gildor path to a better sense of undeserved superiority, and it can be all yours for the low low price of reading these forums.
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #5 on: November 03, 2012, 14:01 »

Because giving as little information as possible, presented as unclearly as you can make it, gives great opportunities to talk down to people.  It's the Gildor path to a better sense of undeserved superiority, and it can be all yours for the low low price of reading these forums.
Sorry?
Logged
Daemonjax
Newbie
*
Posts: 4


View Profile
« Reply #6 on: November 04, 2012, 00:53 »

No. Each mipmap starts at position which is multiply of 0x8000 bytes.

So, it's not compressed with LZO then.

That sounds awesome, because I won't have to get knee deep into any compression algorithms!  Sometimes it's good to be wrong. Wink

Thanks! 


Since you mentioned mipmaps, then is it safe to assume all of the textures in tfc files are in DDS format?


Anyways, thanks again, and I'll look into this some more very soon.
« Last Edit: November 04, 2012, 07:03 by Daemonjax » Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #7 on: November 04, 2012, 10:20 »

So, it's not compressed with LZO then.
No, they are compressed. Data structures are described here:
http://www.gildor.org/smf/index.php/topic,297.0.html
Type of compression, data offset in TFC file and data size are stored in UPK files, TFC contains texture data only. Note: when texture is used from multiple UPK files all of these UPKs will have the same offset/size values. So if you want to replace texture with data of different size you have to update metadata in all UPKs.
Quote
Since you mentioned mipmaps, then is it safe to assume all of the textures in tfc files are in DDS format?
The format is selected in UPK, inside Texture2D object data. Most textures are DXT (not DDS, there's no DDS header), but there are also different formats used like RGBA8, G8 etc.
Logged
Daemonjax
Newbie
*
Posts: 4


View Profile
« Reply #8 on: November 04, 2012, 19:26 »

Thanks for explaining further. Cheesy

Ok, I think I got it now... 

To sum things up:


  • TFC files are compressed (LZO, ZLIB, LZX), and that single compression type is used throughout the entire file.
  • The compression type used isn't stored within that TFC file, but is stored in other upk files within a Texture2D object data (explained here: http://www.gildor.org/smf/index.php/topic,297.0.htm)
  • Uncompressed TFC files contain concatenated textures of various formats (most are dxt), and different resolutions (mipmaps) for streaming purposes.
  • Each of these mipmaps has a length in bytes that is some multiple of 0x8000 (32768 in decimal).

Let me know if any of the above are false. Wink

Assuming the above to be true, then the process to uncompress and extract a particular texture (including all of its mipmaps) hinges on finding its Texture2D object stored in some other upk file.

That's a decent amount of work, but I should be able to construct a differential patch using a modified texture and the original texture.  Changing texture sizes may also be possible by changing the Texture2D object data.


This is the most mod-unfriendly way of doing things I can imagine (yeah, I know things could always be worse unless you're dead).

This will take me some time to do.

Again, thanks for your help.

EDIT:  Aren't dxt files actually dds just with a different header?  http://wiki.beyondunreal.com/Legacy:DXT
« Last Edit: November 04, 2012, 20:05 by Daemonjax » Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #9 on: November 04, 2012, 23:29 »

DXT is a texture encoding format. Raw image data. DDS is an image format, on-disk. It has texture data and header.
Logged
twinj
Newbie
*
Posts: 1


View Profile
« Reply #10 on: December 23, 2012, 11:34 »

TFC is just plain wrong!
Logged
rdlady
Newbie
*
Posts: 9


View Profile
« Reply #11 on: June 07, 2017, 02:56 »

How did you decompress these upk files with the Áƒ*ž^ header?
I'm interested because I have a game that doesn't work with umodel and I'm trying to figure out how the decompression works.

Thanks!
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #12 on: June 07, 2017, 07:23 »

I'm definitely not decompressing them, as the game doesn't work.
Logged
Print 
« previous next »
Jump to:  

Powered by SMF | SMF © 2006-2009, Simple Machines LLC
Leviathan design by Bloc | XHTML | CSS