Improving Normal Map Rendering of Static Props in Unreal by Exporting Qualified Normals From 3dsmax

Alec Moody, July 2011

Background:
With this workflow we can drastically improve the quality of normal map shading in Unreal using existing 3dsmax scanline bakes. The net effect of this pipeline is improved shading quality and, in many circumstances, the ability to build assets with a lower on-card vertex count. It should be underscored that this currently only works with objects imported as static meshes.

Example:

If you are new to normal mapping in a game art pipeline and want an overview of what causes shading errors and how they are normally dealt with refer to:

http://area.autodesk.com/blogs/jean/everything_you_always_wanted_to_know_about_normal_maps_but_were_afraid_to_ask

The meat of the issue is that for best results, the math used to calculate the normal map must be the same as the math used to display the normal map. Unfortunately, Unreal’s normal map rendering doesn’t match any baking software and as a result you need to use extra vertices (in the form of supporting geometry and UV/smoothing splits) to get a clean result. With this pipeline we can significantly reduce shading errors which will ultimately allow us to make lower vertex count assets and improve image quality. Additionally, these pipeline adjustments can be adopted without any support from the engineering team and without any run-time performance cost. They can also be applied retroactively to old assets by simply re-exporting/importing the models (no model tweaking, no re-baking). Lastly, once an artist sets up their max build to export the correct data there is very little change in their pipeline.

Requirements:

The 3dsmax Viewport running Qualified Normals (requires Max 2011 w/ hotfix 4 or Max 2012)

A reasonably recent build of Unreal that supports Explicit Normals (for UDK builds from the summer of 2010 and on)

A flow chart of this process:

Step 1- Enable Qualified Normals in 3dsmax:

3dsmax 2011 hotfix 4 fixed viewport normal map rendering so it matches the scanline renderer. This functionality has been left as opt-in so you will need to add a line to your 3dsmax.ini to enable it. The 3dsmax.ini is located in "\Users\UserName\AppData\Local\Autodesk\3dsmax\3dsmax 2011/2012\enu"

In order to enable qualified normals add these lines to your 3dsmax.ini

[ViewportNormalMapping]
ViewportNormalMappingType=Qualified

Some info from Autodesk can be found in their original hotfix 4 post:
http://area.autodesk.com/blogs/shane/hotfix_4_3ds_max_2011_3ds_max_design_2011

In max 2012 you must disable the nitrous viewport. When we export our FBX file it’s going to use whatever tangent data the viewport is rendering. The nitrous viewport doesn’t currently support qualified normals so you need to run in DirectX mode.

At this point it would be best to test qualified normals to ensure they work. You can download a test scene here:

Qualified Normals Test

Your results should match this image:

While the results we will achieve in Unreal will not be this good, they will improve drastically.

Step 2- Exporting your object as FBX

FBX will allow you to package tangent data with the model, ascii will not. Export your model as FBX with the settings pictured here:

The important box is the one that says Tangents and Binormals

Step 3- Importing into Unreal

Import your model with the “Explicit Normals” box checked. Avoid any settings that change smoothing, or force vertex splitting.

If you want to see a direct comparison of default behavior with this pipeline you can re-import your mesh without this checked.

Limitations and Conclusion:

This only works with meshes imported as static props.

If a mesh needs to be re-exported from Max this asset chain must be followed again or the shading will revert to the default/worse look.

This is a significant improvement to the shading behavior but not a perfect solution. To totally avoid shading artifacts you will still need to split sharp edges on your UVs and smoothing. For this to be fixed correctly Unreal’s shader code must be changed.

It would also be possible, with a small effort from a tools programmer, to adapt this workflow to earlier versions of Max using the 3point shader and their “Quality Mode” normals. These quality mode normals are functionally the same as max’s “Qualified Normals” except that they are stored in a vertex color channel and currently cannot be exported with the FBX. If a new FBX exporter was written that used this alternate data there would be no need for max 2011/2012.