March 28, 2024, 12:36
bigger smaller reset     1020px Wide width Full width Reset   * *

Gildor's Forums

  Homepage Facebook Read news on Twitter Youtube channel Github page
Welcome, Guest. Please login or register.
Did you miss your activation email?

« previous next »
Print
Author Topic: working on a few features (GLTF focused))  (Read 475 times)
mdro
Newbie
*
Posts: 8



View Profile
« on: April 27, 2023, 00:29 »

I've been working on adding a few features in the past months (https://github.com/mdro/UEViewer) and now found the time and motivation to register here and post this.

The features added so far are:
  • 8 influences per bone + export (only GLTF tested, don't know if the other formats even support this)
  • partial export of materials to GLTF (based on older patches by @horsenit)
    • supports specular via currently recommended extensions (specular + IOR=0)
    • still needs to ensure that textures are exported to same folder (or subfolder)
    • still needs to flip green channel of normal maps
    • still needs to convert glossiness to roughness
    • still needs to generate combined occlusion-metallic-roughness from separate textures
    • still needs to put separate opacity into basecolor alpha
    • not sure how to treat specular power
  • exporting morph targets to GLTF
  • exporting scale keys to GLTF (based on @floxay's patches for PSA)
  • setting bufferView.target type in GLTF (may improve loading time)

Things i might do in the future:

@gildor I know you have paused working on umodel, but would you consider PRs?
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #1 on: April 27, 2023, 00:52 »

Nice job.. Just for curiosity - isn't gltf format supports only 4 morph targets?

Also, regarding "flip green channel" in normal maps. It is really NOT NEEDED. I just know this, because I worked on some glTF/Unreal stuff at my job, and doing flip is a mistake. Epic's glTF Exporter plugin does this, what makes geometry looking corrupted. I've made some research regarding that, and found that green channel should be flipped, if you'll flip tangents, flip UV's (Y coordinate) etc. However, neither Epic's GLTF Exporter, nor UModel, not flipping anything, so normal map is totally fine. The model is just using different texture coordinate system, compared to OpenGL, but it WORKS. UModel is showing Direct3D models in OpenGL, using GLSL, and no adaptation of data is needed for that.
Logged
mdro
Newbie
*
Posts: 8



View Profile
« Reply #2 on: April 27, 2023, 01:12 »

https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#skinned-mesh-attributes
you can have multiple JOINTS and WEIGHTS buffers for a mesh to save more than 4 per node.

I will have to look at the normals again, they looked better to me when flipped. Maybe the model i looked at was wrong in the game?  Grin

I'm not sure about your comment at BAKE_BONE_SCALES in the code. Do you want to remove the baked or non-baked mode?
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #3 on: April 27, 2023, 01:30 »

https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#skinned-mesh-attributes
you can have multiple JOINTS and WEIGHTS buffers for a mesh to save more than 4 per node.
Understood, thanks.

Quote
I will have to look at the normals again, they looked better to me when flipped. Maybe the model i looked at was wrong in the game?  Grin
In my experience, flipping is making bad meshes.

Quote
I'm not sure about your comment at BAKE_BONE_SCALES in the code. Do you want to remove the baked or non-baked mode?
I don't remember what I did there
https://github.com/gildor2/UEViewer/commit/8208aeb476530581b886d06250aac8537174e1f8

This was made 2 years ago, and I think I worked on scale animations and animation retargeting that time. And comment says "feature is disabled", but it is NOT. It seems, "baking" of animation scales was implemented before, and the new code allows to bypass it, and store scales separately, animating them in runtime. Probably I've added the feature trying to resolve some issue, and it didn't help (or I didn't get a feedback from some person who reported the issue), so I stopped working on it - leaving the new code for future experiments (and them leaving the project).
Logged
mdro
Newbie
*
Posts: 8



View Profile
« Reply #4 on: April 27, 2023, 01:49 »

This was made 2 years ago, and I think I worked on scale animations and animation retargeting that time. And comment says "feature is disabled", but it is NOT. It seems, "baking" of animation scales was implemented before, and the new code allows to bypass it, and store scales separately, animating them in runtime. Probably I've added the feature trying to resolve some issue, and it didn't help (or I didn't get a feedback from some person who reported the issue), so I stopped working on it - leaving the new code for future experiments (and them leaving the project).

ok, thanks. I noticed a problem with baked mode where a model got squished in animations, will look at it some day. Probably something around the root bone.
Logged
mdro
Newbie
*
Posts: 8



View Profile
« Reply #5 on: April 29, 2023, 15:56 »

I have tested the normal maps again, they work as-is in the previewers in VS Code (Babylon.js, Cesium, Filament, Three.js), but not in Blender and Godot import.
So the only fix would be converting the whole coordinate system as you described Sad
Logged
Gildor
Administrator
Hero Member
*****
Posts: 7978



View Profile WWW
« Reply #6 on: April 29, 2023, 19:37 »

Probably Blender ignores tangent space stored in model and re-builds it, causing problems.
Logged
spiritovod
Global Moderator
Hero Member
*****
Posts: 1901


View Profile
« Reply #7 on: May 13, 2023, 23:17 »

@mdro: I'd suggest to revise the part with weights, because current implementation wouldn't work in most cases (literally all games using new format, checked via bNewWeightFormat). It has self-explanatory bUse16BitBoneIndex field, which is suggested to be false by respective assert, but in FSkinWeightInfo you're trying to read into int16 array by default, which is exact opposite behavior. It can be fixed by actually using bUse16BitBoneIndex value for conditional reading of weights, while removing that assert.

As for BAKE_BONE_SCALES thing, it looks like it's disabled with current definition "#if BAKE_BONE_SCALES 1", because scales are defined via "#if !BAKE_BONE_SCALES". The reason why it was not disabled by just changing 1 -> 0 is because it's still used for allowing some retargeting functions. Which means setting it to 0 enables reading scales, but disables retargeting. I guess the cases when meshes would look strange here are related to incompatible retargeted animations rather than scales.
« Last Edit: May 13, 2023, 23:43 by spiritovod » Logged
mdro
Newbie
*
Posts: 8



View Profile
« Reply #8 on: May 17, 2023, 03:41 »

@mdro: I'd suggest to revise the part with weights, because current implementation wouldn't work in most cases (literally all games using new format, checked via bNewWeightFormat). It has self-explanatory bUse16BitBoneIndex field, which is suggested to be false by respective assert, but in FSkinWeightInfo you're trying to read into int16 array by default, which is exact opposite behavior. It can be fixed by actually using bUse16BitBoneIndex value for conditional reading of weights, while removing that assert.
Good catch! Should be a quick fix.
Just to be sure: I'm currently reading into byte, you saying int16 in your post is a mistake?

As for BAKE_BONE_SCALES thing, it looks like it's disabled with current definition "#if BAKE_BONE_SCALES 1", because scales are defined via "#if !BAKE_BONE_SCALES".
Agreed, bake means that scales are integrated into translation, without it there are separate variables.
The reason why it was not disabled by just changing 1 -> 0 is because it's still used for allowing some retargeting functions. Which means setting it to 0 enables reading scales, but disables retargeting. I guess the cases when meshes would look strange here are related to incompatible retargeted animations rather than scales.
Sorry, I don't fully understand this. What do you mean exactly?
  • without baking manual override of retargeting mode won't work?
  • without baking some other animations will break, because they are specified with retargeting modes that would no longer work?
  • the animation I saw fixed without baking shouldn't work, the game is probably overriding something itself?

Thanks for helping me/us!

I have researched the tangent space/normal maps question further and I think there is a workable solution:
Both Unreal and GLTF store normals, tangents and a flag (normal.w/tangent.w) whether the binormal should be calculated for a right-handed or left-handed tangent space.
Blender and Godot ignore the stored tangents and generate a right-handed tangent space from UV directions.
Handedness of tangent space only matters for normal maps and maybe direction of anisotropy (but anisotropy isn't supported by uModel).
=> If we always set tangent.w in GLTF to 1 (right-handed) and invert the G channel of the normal map if normal.w in Unreal was negative (left-handed), all programs should interpret the model the same (as long as the tangents match UV directions).
Logged
spiritovod
Global Moderator
Hero Member
*****
Posts: 1901


View Profile
« Reply #9 on: May 17, 2023, 15:20 »

@mdro: That's weird, I just checked your github and it's all correct there with byte array, while I have slightly different code, like it was pulled from different fork. So yes, it is mistake on my side, sorry.

As for scales, I should have used better wording probably. I didn't dig into it much, and may be wrong, but from my understanding bones scales are used primarily for retargeting functionality. Current implementation is kind of trade-off, when you consider that variable scales are used for retargeting (and it's supposed to be like that for modern games in general), while ditching several other cases where scales are used differently (e.g. games on really old engines like Days Gone or asian stuff like Wuthering Waves with active usage of virtual bones). By setting BAKE_BONE_SCALES to 0 retargeting functionality would be disabled and changing retargeting mode manually wouldn't produce expected result, though it will not be very noticeable, except some cases where a game is using 2/3 skeletons for everything. And since ActorXImporter for 3ds Max supports different retargeting modes in the same way viewer does, things will not work as expected there as well. I guess cases where baked scales are breaking things are rather exceptions compared to the rest of games.

---------------------------------------------------------------------------------

Anyway, thank you very much for your contributions. Using glTF as primary format for export would allow to bypass some issues with existing psk plugins for blender (for example, currently maintained one has certain issues with multiple UVs, compared to original Befzz's plugin).
« Last Edit: May 17, 2023, 15:24 by spiritovod » Logged
mdro
Newbie
*
Posts: 8



View Profile
« Reply #10 on: July 23, 2023, 18:40 »

I've had some time for working on this again!

I have reworked the GLTF texture export:
  • Used textures are exported to a subdirectory of the mesh directory (gltf_tex/) because you can't reference textures outside the current directory in GLTF.
  • As I described earlier, models with left-handed tangent space are converted to right-handed by setting tangent.w to 1 and inverting g/y in normal maps. Fixes importers that construct their own right-handed tangents.
  • Separate occlusion maps are integrated into ORM material maps.
  • Specular power maps are inverted and integrated into ORM material maps as roughness. (From what I've read this not fully accurate, but at least qualitatively correct.)

Regarding the animation scaling issue I have taken a closer look at the deserialized data before conversions:
It looks like issue comes from an animation set with a skeleton that has a different scale on its root bone than the mesh's root bone. There are no scale keys in the animations. The retargeting mode is "animation" everywhere.
Based on comparisons to static meshes in game (assuming they are placed on the map without scaling), Unreal seems to scale the whole model with the animation skeleton's root scale. I also found the comment "MeshScale is integrated into RefSkeleton.RefBonePose[0].Scale3D" in USkeletalMesh4::ConvertMesh().
UModel instead only scales the skeleton, but doesn't adjust the mesh, leading to squished or stretched meshes when playing an animation (both in umodel and on the exported model).
Does anyone of you have an idea how to fix this? Maybe add a scale key for the root bone to each animation?
Logged
mdro
Newbie
*
Posts: 8



View Profile
« Reply #11 on: July 24, 2023, 02:29 »

I have further investigated scale in that one game: With the help of Universal Unreal Unlocker I found out the size of characters and objects in game - they are definitely not the size they are in the files.
I have also read up on animation retargeting a bit.
With that new info my conclusion is: UModel works correctly.
The game must be overriding the retargeting mode to AnimationScaled or OrientAndScale.
At least there is one useful outcome from this goose chase: I have added the remaining retargeting modes to the ctrl+r override in the viewer.
Logged
spiritovod
Global Moderator
Hero Member
*****
Posts: 1901


View Profile
« Reply #12 on: July 28, 2023, 02:53 »

@mdro: Sounds great, thank you for those adjustments and research on retargeting. I think in many modern games retargeting is usually customized via blueprints or maybe montages to meet devs needs. Could you please tell, how things work with different retargeting modes from your point of view (I mean if compared with your expectations)?
« Last Edit: July 28, 2023, 02:56 by spiritovod » Logged
mdro
Newbie
*
Posts: 8



View Profile
« Reply #13 on: August 01, 2023, 02:44 »

I mostly looked at these sources:
https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/AnimHowTo/Retargeting/
https://docs.unrealengine.com/5.1/en-US/PythonAPI/class/BoneTranslationRetargetingMode.html

For my test cases with a scaled skeletons these are the results:
  • Animation: model gets crushed/stretched to size of animation skeleton - correct, those are the positions of the bones in the animation.
  • AnimationRelative: same as Animation - correct if the description means that only differences in orientation between the skeletons are compensated.
  • AnimRotationOnly (Skeleton in UE): mostly works but looses whole model translation (usually only used in cutscenes) + minor glitches like closed eyelids - correct, most movements are bending joints, but some things are translation.
  • AnimationScaled: works - correct, compensates for scale differences between skeletons
  • OrientAndScale: works - correct, compensates for scale and should correct orientation differences (not necessary in test case)

Anyway, retargeting is (currently) only relevant for the viewer. The PSK exporter seems to have limited support for exporting the retargeting modes saved in the animation, GLTF and MD5 have nothing. We could try to bake the (possibly overridden) retargeting for each mesh, but that's better done with a full 3d modeler with finer control.
For that we need to export the animation skeleton. PSK and MD5 seem to do this already. For GLTF we only export the mesh skeleton, so we effectively force AnimationRotationOnly retargeting.
The fix for GLTF seems to be to create a separate file that only contains the animation skeleton and animations without a mesh/skin/materials. Importers should then recognise this as a pure animation library. I have manually reduced a full model that way and it works in Blender and passes GLTF validation (except for warnings for unused data left over from my rough edit).
Logged
Print 
« previous next »
Jump to:  

Powered by SMF | SMF © 2006-2009, Simple Machines LLC
Leviathan design by Bloc | XHTML | CSS