March 29, 2024, 06:23
bigger smaller reset     1020px Wide width Full width Reset   * *

Gildor's Forums

  Homepage Facebook Read news on Twitter Youtube channel Github page
Welcome, Guest. Please login or register.
Did you miss your activation email?

« previous next »
Print
Author Topic: Extracting xbox textures from mass effect 2 startup.xxx  (Read 9384 times)
Dybuk
Newbie
*
Posts: 13


View Profile
« on: February 02, 2016, 22:40 »

Greetings,

I'm trying to use UE Viewer to extract textures from the following file (xbox360 mass effect 2 startup_int.xxx) :

https://mega.nz/#!I0txUS7S!u_0WH7DNfuUMFaSlgczpXoau3Rd7oFTqG7TY_CiWwCs

My end goal is to insert them into the PC version that is missing them. This part is working already using ME3Explorer code. Getting the textures out of the xbox file correctly is my issue.

Although UE Viewer loads the file fine (force lzx compress). It seems to have problems with the DDS PF_A8R8G8B8 textures. Its as though the colour is inverted. Perhaps the colour maps are getting inverted (endian issues?). Also UE Viewer will only output the files as TGA, instead of dds which is the format I need to re-inject them with (although maybe they'll be fine after running through a converter).

I tried to extract these textures with ME3Explorer (after several endian changes) and they game out quite garbled. The same way they come out in UE Viewer when you force PC. I guess some magic tiling changes are coded into UE Viewer maybe?.

Here are three example files. The pc version, xbox version extracted using UE Viewer (tga) and using a modded ME3Explorer :

https://mega.nz/#!d00wUCqC!9ULdIXuOq9ZeQssZW_3jdwAyF3yjYMdCJN_7IhhDi3M

I've poked through the UE Viewer code a bit. I suspect it'll take me some time to actually figure out what is happening though. Any pointers would be great.

Thanks,

Dybuk
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #1 on: February 02, 2016, 22:56 »

Hi.

A8R8G8B8 texture format is not processed in any "special way", UModel doesn't perform any math on colors. Not sure if this format requires byte swapping for XBox360 or PS3 - I have no information about this. Perhaps this is game-specific feature. For some games I saw completely garbage textures in this format.

Regarding DDS export - I'm doing that only for compressed texture formats. Uncompressed formats are always saved as TGA.

Pointers to UModel code: UE3 texture loading is handled in UnTexture3.cpp, all texture processing like decompression, XBox360 decoding is in UnTexture.cpp.
Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #2 on: February 02, 2016, 23:20 »

Wow, thanks for the quick response.

I don't know much about the PF_A8R8G8B8 format. Do you know if its just an array of 32 bit INTs after the header?

I could try decompressing the xxx file. Finding the export points data block and swapping all the bytes after the properties section to the end of the data block. Then throw it at UE viewer. What could possibly go wrong?

Or maybe time to try and get UE Viewer to build Smiley

Dybuk
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #3 on: February 02, 2016, 23:29 »

This format is 32-bit uncompressed texture. This is a very simple and very old texture format. Every pixel stored as 1 byte red, 1 byte blue, green and alpha (4 bytes per pixel).

There's no "headers" exists in texture data. For details, see UnMaterial3.h, UTexture2D class. It holds all mipmaps in array "Mips" of FTexture2DMipMap type. FTexture2DMipMap itself holds texture data in bulk data block, which represents raw byte array of size (TextureWidth * TextureHeight * BytesPerPixel). This data block exists on disk either in package file or in TFC, in uncompressed or compressed form (I mean not a texture compression, but LZO/ZLib/LZX).

If more internals details will be discussed here, I'll more this thread to "Unreal coding" which exists specially for such things Smiley
Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #4 on: February 02, 2016, 23:42 »

Yeah, I'm aware there isn't a header. I've poked through the ME3Explorer code to see that it pretty much adds a DDS header to the data based on the properties read:

DDSHeader.magic = 0x20534444;
DDSHeader.Size = 0x7C;
DDSHeader.Flags = 0x081007;
DDSHeader.Caps = 0x1000;
DDSHeader.pfFlags = 0x4;
DDSHeader.pfSize = 0x20;
DDSHeader.Width = imgSize.width;
DDSHeader.Height = imgSize.height;
DDSHeader.pfFlags = 0x41;
DDSHeader.RGBBitCount = 0x20;
DDSHeader.RBitMask = 0xFF0000;
DDSHeader.GBitMask = 0xFF00;
DDSHeader.BBitMask = 0xFF;
DDSHeader.ABitMask = -16777216;

Sadly with the header it adds in this case the texture comes out quite garbled and with colour issues. Similar to UE viewer when you force pc mode. I guess I should look at your xbox decoding code to see what you are doing differently to me3explorer (or at least what you do differently when you detect xbox360). I would have thought forcing pc mode would have made the file not load at all.

Guess its time to man up and try some C++. Its been a while.

Feel free to move this thread to any forum you want.

Dybuk
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #5 on: February 02, 2016, 23:52 »

Look at last lines of UTexture2D::GetTextureData() to see what's different for XBox360 is doing.
Regarding ME3Explorer, I'm pretty sure it doesn't have XBox360 support: 1) it is intended for PC game modding, 2) that would require a lot of work to port C/C++ code to C# (LZX, texture decoding).
Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #6 on: February 03, 2016, 00:15 »

Yeah to get the file to load in me3explorer I had go make the endian selectable in some of your code and had to use your decompress tool before it loaded.

I'm trying to fix a bug in the ME2Controller mod for pc where part of the hud doesn't appear. I'm pretty sure this is down to missing textures that the flash file refers to as an external. I've already modified the me2explorer's PCCObject file to allow the adding of new export points (via a clone) and have code to add the textures.

I'm going to call it a night for now but I'll have a go at hacking your code to byte swap the data over the next days to see if that does it.

Thanks for your help,

Dybuk
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #7 on: February 03, 2016, 00:21 »

Note appReverseBytes() function in UModel's source. It is intended to swap bytes in arrays of values of any integer size.
Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #8 on: February 03, 2016, 12:50 »

I did come back to this last night for 30 mins. Tried to get umodel to build, it didn't like my visual studio location. I was clearly too cheap to buy a bigger SSD so have to install big things to other drives. After a quick Perl hack got past that. Then it looked as though the build was trying to write to the same symbol file for all the C++ files (obj\umodel-win32\vc120.pdb) which caused a conflict. I'll trawl through the build scripts later. Probably a user error Smiley

So at that point I thought I might as well try plan A. So I got out the old trusty hex editor and did a byte swap in the export point data. Loaded it into UE-Viewer. It loaded fine no problem. The colours had been switched around for my test texture, but it still wasn't right. Damn. Might be more complicated than I thought. Joy. Lets pray for user error.

I'll continue tonight. Pesky work getting in the way.

Dybuk

Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #9 on: February 03, 2016, 13:57 »

There are some instructions regarding UModel's source code (not "unreal coding" in general, UModel specific) - http://www.gildor.org/smf/index.php/board,37.0.html

Then it looked as though the build was trying to write to the same symbol file for all the C++ files (obj\umodel-win32\vc120.pdb) which caused a conflict. I'll trawl through the build scripts later. Probably a user error Smiley
I'm using Visual Studio 2010 to build UModel. VS2013 (and perhaps VS2015) has a problem with PDB like you mentioned. To solve it, parallel build tool (jom.exe) should be disabled. You can do that either in build.sh by adding "--nmake" option to vc32tools (see the end of build.sh), or by modifying "vc32tools" script itself - add 'force_nmake=1' at the bottom instead of 'force_nmake=""'.
Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #10 on: February 04, 2016, 03:28 »

I gave up in the end and set up a virtual box with windows 7, visual studio 2010. Everything work first time.

I tried adding a byte swap (as well as actually removing the one that was there). I pretty much saw the same results as when I byte swapped the export data.

You got any other thoughts on looking at the below? Left is xbox, Right is PC.



Thanks,

Dybuk
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #11 on: February 04, 2016, 12:37 »

Sorry, no idea. I never replacing resources, this is not interesting for me.
Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #12 on: February 04, 2016, 13:17 »

Sorry, no idea. I never replacing resources, this is not interesting for me.

Damn Sad Obviously its more about extracting the xbox assets and converting them to PC format. The replacing is pretty much sorted.

Guess I need to do some reading up on the dds format and do some more playing.

The annoying thing is the guy that originally started the mod managed to achieve this. In one of his posts he described using umodel to get the textures out. Sadly haven't heard form him since Aug last year.

His problem was he didn't have code to add new export points so he was just replacing the existing pc icons. However he ran out of pc icons to replace so left 5 textures out.

Even more annoying are these 5 textures are 256x1 images that appear all black in umodel. They are just used as Gradients for UI flash objects. I know they do something as when I replace them with other textures the missing ui components appear (although looking quite wrong).

I'll carry on and I'll let you know how I get on. If you get any thoughts, know of any other tools I could try, let me know.

Cheers,

Dybuk

Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #13 on: February 04, 2016, 19:43 »

Success!!

Looks like I'm really dumb Smiley

Inside CTextureData::DecodeXBox360(int MipLevel)
...

Its hard coded to do :

   // swap bytes
   if (bytesPerBlock > 1)
      appReverseBytes(buf, Mip.DataSize / 2, 2);

which is a 16bit swap.

Changing that to :

   // swap bytes
   if (bytesPerBlock > 1)
      appReverseBytes(buf, Mip.DataSize / 4, 4);

Fixes all the PF_A8R8G8B8 textures... breaks all the others though.

Guess we could put something in PixelFormatInfo so that it swaps 4 for PF_A8R8G8B8.

Any thoughts?

Dybuk
Logged
Dybuk
Newbie
*
Posts: 13


View Profile
« Reply #14 on: February 04, 2016, 20:42 »

Boooo the 256x1 textures still show up as all black... will have to dig into the code a bit more Sad

Dybuk
Logged
Print 
« previous next »
Jump to:  

Powered by SMF | SMF © 2006-2009, Simple Machines LLC
Leviathan design by Bloc | XHTML | CSS