Splinter Cell Series (SC1-4)

[spiritovod]

@Joshhhuaaa: Thank you for such detailed info Though offline version of Pandora Tomorrow was already supported in official umodel build, except for static meshes. It's just it can't be autodetected due to detection overlapping, but if respective override is set manually, everything should work (except for static meshes).

Some notes from my own researches:
1) Online (versus) format for static meshes in Pandora Tomorrow and Chaos Theory is the same, it's just in second one names table is additionally obfuscated. Double Agent online static meshes format is around the same, but actually closer to static meshes in offline versions of SC1-2.
2) Chaos Theory also contains some data in raw "bin" format (_tex.bin/_sm.bin for textures and meshes respectively), which consists of index (object names and offsets/sizes for raw data) and data itself. There is "scl" format as well, which is modified UE2 package.
3) Offline version of Double Agent is using some proprietary "hlnc+ulnc" format, which works more like UE4 asset, but with embedded UE2 packages(?).

[Joshhhuaaa]

Update: spiritovod fixed all the packages mentioned below from Splinter Cell and Splinter Cell Pandora Tomorrow (single player) that were previously crashing. Update to the latest version of the UModel build, available in this topic.


Just to update the status on UModel Compatibility for Splinter Cell and Splinter Cell: Pandora Tomorrow (Single Player), you may run into some issues when loading some packages. I'm not certain whether these specific packages can be easily fixed or if they have unique issues compared to the others.

The following packages currently cause errors or crashing:

Splinter Cell

Animation:
EMiniV.ukx (Memory: bad allocation size -939524096 bytes) [SCPT has this package and it works, SC1 is a little bigger in size]


Texture:
1_3_Cinematic_TEX.utx (Memory: bad allocation size -1476395008 bytes)
6_1_1_kolatex.utx (Property ZTest expected type byte but read bool)
lambert_office_tex.utx (Memory: bad allocation size -1140850688 bytes) [SCPT has this package and it works, SC1 is a little bigger in size]
Maps.utx (Unable to read 641882 bytes at pos=0x1000) [This package is identical to the Maps.utx seen in SCPT]


Splinter Cell: Pandora Tomorrow

Animation:
EBird.ukx (Memory: bad allocation size -1342177280 bytes)
ELada.ukx (Memory: bad allocation size -1140850688 bytes)
EParA.ukx (Access violation (C0000005) at umodel_materials.exe+0xFF549)
EParB.ukx (Memory: bad allocation size -67108864 bytes)
EWinGlass.ukx (Memory: bad allocation size -469762048 bytes)
EZodiac.ukx (Memory: bad allocation size -671088640 bytes)

StaticMesh:
431_TVS.usx - Loads package sometimes or crashes without an error message. Some meshes appear broken (picture below). Will crash on certain meshes, application just closes with no error.
EMeshCharacter.usx (Wrong import index 61)
LeafSM.usx (Memory: bad allocation size -86528 bytes)
LightGenOBJ.usx (Same behavior as 431_TVS.usx)

Texture:
111_EmbassyT.utx (Memory: bad allocation size -98112 bytes)
3_1ShipyardTXT.utx (Serializing behind stopper (DC0EB2+508 > DC11BB))
411_VillageT.utx (Property ZTest expected type byte but read bool)
ETexAccessories.utx (assertion failed: Colors.Num() == 256)
GUIContent.utx (Memory: bad allocation size -911106 bytes)
Maps.utx (Unable to read 641882 bytes at pos=0x1000)
MiningTown_Tex.utx (Memory: bad allocation size -872415220 bytes)
NucRoom2.utx (Memory: bad allocation size -1744830464 bytes)
WarehouseAD.utx (Access violation (C0000005) at umodel_materials.exe+0xFF549)


These StaticMesh packages will load correctly without textures. They currently fail with textures loaded as they rely on Texture packages that fail currently.
      2_2_TrainS.usx (assertion failed: Colors.Num() == 256) - Failing because ETexAccessories.utx
      411_VillageS.usx (Property ZTest expected type byte but read bool) - Failing because 411_VillageT.utx
      EMeshAccessories.usx (assertion failed: Colors.Num() == 256) - Failing because ETexAccessories.utx


In terms of priority, the most important packages to get fixed as they contain full map assets:
Splinter Cell: Pandora Tomorrow

StaticMesh:
431_TVS.usx

Texture:
111_EmbassyT.utx
411_VillageT.utx
ETexAccessories.utx


Example of 431_TVS.usx and LightGenOBJ.usx having broken meshes. These are the only 2 packages that can load without problem occasionally but have broken meshes. I noticed the affected meshes are usually in the Group named "breakablelight."

[Joshhhuaaa]

All of the mentioned errors should be fixed now in the updated build. Though some textures, which are using non-standard palettes, can't be loaded. Also those weird looking meshes are using non-standard rendering (triangle strips or quads), which is not supported in umodel in general - but at least they shouldn't crash.

Thanks for all these updates and fixes, I really appreciate you taking the time. I noticed the non-standard palettes not being loaded as well, I forgot to bring that up but thanks for letting me know.

I've confirmed that all packages for SC1/SCPT are now working on the latest build.

On the side note, static meshes from Pandora Tomorrow (online) and Chaos Theory (versus) should be supported now as well, without textures.

I definitely didn't expect anyone to bother adding support for these but nice work. SCCT Versus StaticMesh support is flawless with all packages that shipped with the game working and even the Ubisoft DLCs released later load without any issues.

If a StaticMesh package has been created or modified with the retail Chaos Theory Editor, it will break UModel support. Only packages created by Ubisoft and not modified will load correctly. User-created packages give the error: "Serialized FString is not null-terminated."

This is definitely low on the list in priority, I wouldn't worry about this problem if it is difficult to fix. If you are interested in looking into it, here are a few custom StaticMesh packages.


The following packages currently cause errors:

SCPT Versus

StaticMesh:
chantier_poc_stm.usx (Serializing behind stopper (A88+7068 > 12B0) [SCCT Versus has this file working with all the same StaticMeshes, plus some more.]
CTF_static.usx (Memory: bad allocation size -549494240 bytes) [SCCT Versus has this file working with all the same StaticMeshes.]
FX_WAR_warp.usx (Memory: bad allocation size -1960704 bytes) [SCCT Versus has this file working with all the same StaticMeshes.]
KRA07_Specifik_XBox.usx (Memory: bad allocation size -1080 bytes) [Highest priority to fix as it is related to a map's assets.]
POC_TESTLD_SM.usx (Memory: bad allocation size -126 bytes)
SEffectStatic.usx (wrong export index 61)
SIcoMinimap_.usx (Memory: bad allocation size -803370 bytes)
Sminimap_SM.usx (bad allocation size -10 bytes)
SoundVisionMesh.usx (Memory: bad allocation size -9572 bytes)
stm_lab.usx (Memory: bad allocation size -1368 bytes)
TestMapStaticM.usx (wrong export index 62)
SM_EgoutsNeew.usx (Serializing behind stopper (799+FECD90 > 273B) [Package was created by Ubisoft, but released as DLC later.]

None of these are too concerning, if someone wanted any of these failing SCPT Versus StaticMeshes, they can import the package into the Chaos Theory Editor and then export into .ASE format from there.


Thanks again spiritovod with all the progress.

[Joshhhuaaa]

@Joshhhuaaa: I've rolled some fixes in the updated build (meshes created in editor for Chaos Theory should be supported now), but I guess it's quite pointless trying to support all possible formats for Pandora Tomorrow, since they're using like 10 different formats or more within one game version (it can be guessed from packages versioning as well). Engine supporting all those variants at once should be a flying spaghetti monster...

No worries about the remaining SCPT Versus packages that are causing issues, I can't thank you enough, and it is impressive how fast you've been rolling out these updates.

Packages like TestMapStaticM.usx and POC_TESTLD_SM.usx are not even used in the game and are only developer test packages. The minimap/soundvision packages are also not used in the game and canceled content.  This latest build supports all StaticMeshes that are actually used in levels.

SCCT Versus StaticMesh support is perfect now, no issues at all.


I noticed you added Texture support for SCPT Versus as well. If it is any useful, here are the packages that currently fail in latest build:

SCPT Versus:

Texture:
DTK_Textures_TC.utx (wrong name index -6592)
EditorTex.utx (wrong name index -1879048191)
GUIContent.utx (wrong name index -1048564)
Krauser_TXT.utx (wrong name index 65229360)
MHOSFX_Xbox.utx (wrong name index -48)
Prison_TEX.utx (wrong name index -32)
SFont.utx (wrong name index 2013265921)
SFX_KRA_Xbox.utx (wrong name index 66216172)
SFX_SCH_Xbox.utx (wrong name index 1046167)
SGameplayObjectsTex.utx (GetTypeName: unknown type index 0)
SHUDTextures.utx (wrong name index -1048575)
SIcoMiniMap.utx (wrong name index -1342177279)
SMiniMap.utx (wrong name index -1048573)
SPersoTextures.utx (wrong name index -1048565)
TXT_Spy.utx (wrong name index -52)
TXT_WAR.utx (wrong name index -29)

In terms of priority, the most important packages to get fixed as they contain textures used in maps:
SCPT Versus:

Texture:
DTK_Textures_TC.utx
Krauser_TXT.utx
MHOSFX_Xbox.utx
Prison_TEX.utx
SFX_KRA_Xbox.utx
SFX_SCH_Xbox.utx
TXT_Spy.utx
TXT_WAR.utx

[Joshhhuaaa]

Since I don't quite understand what they're doing with some struct properties (like FMatrix), workaround is added for such cases. Now umodel will stop reading any properties after bumping into any unknown struct property. It should help to avoid related crashes, but the rest of properties would be skipped, which may affect materials.

With this update, textures now works for Pandora Tomorrow (online) and Chaos Theory (versus). Textures from Double Agent (online) partially works as well, but due to additional properties some of them can't be loaded properly at the moment.

Nice update as always, I wasn't expecting to see SCDA Versus so quickly even if it is just partial texture support. SCDA Versus StaticMesh support would be awesome to get added eventually.

SCPT Versus and SCCT Versus are very good in compatibility now, you can consider them complete, unless interested in supporting their Animation packages.

Appreciate all the recent work, most people tend to avoid the Splinter Cell series because engine is too modified. The builds have been helpful for plenty of people already within the community.



Update on the packages that current fail on latest build:

SCPT Versus

Again, these packages are no big deal in terms of priority. I would consider this game complete.

Texture:
EditorTex.utx (wrong name index -1879048191)
SFont.utx (wrong name index 2013265921)
SIcoMiniMap.utx (wrong name index -1342177279)

StaticMesh:
POC_TESTLD_SM.usx (Memory: bad allocation size -126 bytes)
SEffectStatic.usx (wrong export index 61)
SIcoMinimap_.usx (Memory: bad allocation size -803370 bytes)
Sminimap_SM.usx (bad allocation size -10 bytes)
SoundVisionMesh.usx (Memory: bad allocation size -9572 bytes)


SCCT Versus has perfect support now with this build, all Texture and StaticMesh packages are working including newly created packages with the retail editor.


SCDA Versus

Texture:
Menu_ContextItem.utx (Serializing behind stopper (4F8+A8 > 516))
MMain_Test_Particules.utx (Serializing behind stopper 15A442+3D6120 > 172E88)

StaticMesh:
BLW_SFX.usx (Serializing behind stopper (4C74+53E08 > 19BF5))
BntMap_StaticMesh.usx (Memory: bad allocation size -532643832 bytes)
HUD_STM.usx (Memory: bad allocation size -266321788 bytes)
LightVolume_SM.usx (Memory: bad allocation size -24668 bytes
MMain_Particles_Mesh.usx (TArray: index 0 is out of range (0))
NotUsedAnymore_STM.usx (Memory: bad allocation size -208 bytes)
Objectifs.usx (Memory: bad allocation size -80 bytes)
ProtoMenuPackage.usx (Serializing behind stopper (4C72+1F180 > 55E2))
Seb.usx (TArray: index 0 is out of range (0))
SFX_SHP02_STM.usx (Serializing behind stopper (4C89+53E08 > 19C0A))
SGameplayObjects.usx (Memory: bad allocation size -264 bytes)
XTST_STM.usx (Memory: bad allocation size -20 bytes)

In terms of priority, the most important packages to get fixed as they contain assets used in maps:

SCDA Versus

StaticMesh
BLW_SFX.usx (Serializing behind stopper (4C74+53E08 > 19BF5))
SFX_SHP02_STM.usx (Serializing behind stopper (4C89+53E08 > 19C0A))

[Joshhhuaaa]

Static meshes from Double Agent (online) are now supported as well. Though they're using custom material class for most materials and thus they don't work.

Very nice to see them supported regardless. Are custom material classes and those non-standard palettes from SC1/SCPT out of UModel's scope to ever support or just difficult to implement?

Also, the latest build that added SCDA Online StaticMesh support ended up having a little regression. SCCT Versus StaticMesh packages that were created in the retail Chaos Theory Editor are now crashing again. These packages are now getting the error: "Serializing behind stopper (BB9+2070 > 20A9)".

I understand that fixing one issue for one game could potentially break another game. If ensuring all games work seamlessly under a single build becomes too complicated, I've saved backups of previous builds in case regressions like this ever happen, so no worries.

[Joshhhuaaa]

@Joshhhuaaa: That regression should be fixed now. I forgot that packages from editor for Chaos Theory are using higher engine version than Double Agent.

Speaking of palettes, I have no idea how they work in context of umodel and I guess there are other things to look into aside from them. As for materials - well, the point is that even for UE4 games, where approach for custom materials is more convenient, there are cases when materials are not even parsed or linked to meshes. And in latest Splinter Cell games those are custom classes, which is a way harder to implement, so I doubt they will be supported.

I see, thanks for the fix on latest build. I think that covers about everything that UModel can support without being very difficult to implement. SCCT Versus Animation support would be useful to have if it is not difficult, but can't thank you enough for everything so far.