Modding:Texture formatting
This tutorial is from atmosfur
Work in Progress
This guide has several sections related to the texture formats that BG3 uses, and how to export and prepare them in several programs. This guide is mainly about maps baked from custom 3D models and does not include skin textures currently.
The texture maps this guide covers are:
_BM (Basecolor Map)
_NM (Normal Map)
_MSK (Mask Map)
_PM (Physcial Map)
Additionally, to create a PM you will need:
AO (Ambient Occlusion Map)
Roughness Map
Metallic Map
This guide also includes a section on setting up maps by hand in GIMP if you do not have Substance Painter, or edited your textures by hand.
Substance Painter Export Preset
In case the BG3 export presets for Substance Painter do not work for you, or you have a pre-Adobe version of Substance Painter, here's how you can set up your own export presets in Substance Painter. I personally divided these into two separate presets due to my Basecolor and Mask map both pulling from different layers of basecolor information.
1. In the File -> Export Textures window, select the Configuration tab.
2. Click the small + icon next to "Presets" on the left side to create a new preset.
3. Create two R+G+B+A Output maps and one R+G+B Output map for the NM, PM and MSK map respectively.
4. On the right, you will see which Input maps you will need to drag and drop onto the Output maps R,G,B and A slots.
5. Drag and drop the Input maps onto the corresponding slots in the Output maps tab:
Normal Map (Make sure to use the Converted maps - NormalDirectX):
R: Leave this empty
G: NormalDirectX, select Green channel
B: NormalDirectX, select Blue channel
A: NormalDirectX, select Red channel
Physical Map:
R: Metallic, select Gray channel
G: Roughness, select Gray channel
B: Ambient occlusion, select Gray channel
A: Leave this empty
Mask Map:
This is specifically for horn textures, you will need to check a vanilla mask of whatever you are texturing to see which channel has the mask for customizing color. In this case, horns only use the Green channel for that.
R: Leave this empty for horns
G: Base Color, select Green channel
B: Leave this empty for horns
6. As mentioned, for the Basecolor Map I create a new export preset. For this you just need one RGB+A Output map, and drag the Base Color Input map onto the RGB slot.
7. Go back to the "Export" tab of the window and select your newly created export presets from the drop down at the top, then just hit export. Double check your created textures in a program that lets you view channels (such as GIMP or Photoshop) to make sure they all got assigned correctly. You will still need to convert your textures to DDS now.
Creating the DDS files
Simple Conversion
If your textures have all their channels set up correctly already, you just need to export them with the correct DDS compression in the editing program of your choice. I'm using Paint.NET as an example.
Basecolor Map Settings:
Choose BC1 (Linear,DTX1), make sure Generate Mip Maps is checked
Mask Map Settings:
Choose BC5 (Linear, Unsigned), make sure Generate Mip Maps is checked
Normal Map Settings:
Choose BC3 (Linear, DTX5), make sure Generate Mip Maps is checked
Physical Map Settings:
Choose BC1 (Linear,DTX1), make sure Generate Mip Maps is checked
Manually Assign Channels
If you've created your textures in a program other than Substance Painter, you might need to manually assign the correct channels to your maps. Your best choice for doing this is probably GIMP, and I tried to make this part of the guide as beginner friendly as possible if you've never used the program before.
I recommend working on one map type at a time if you're following along with this guide. Again, this uses a horns texture as an example, the way other textures are set up in the game might be different, and it does not touch on the way skin textures are set up.
Normal Map
1. Open your Normal Map in GIMP, and click on the Channels tab to view the channels of your map. Depending on the program you created your normal map with, you might or might not have an Alpha channel already.
2. If you do not see an Alpha channel in your channels, switch back over to the Layers tab, right-click on your layer and select "Add Alpha Channel". This will add an empty Alpha channel.
3. Back in your Channels tab, right-click on the "Red" channel and select "Channel to Selection".
4. Swap back to your Layer tab, then in your menu bar go to "Edit -> Clear" to delete the selected areas. This basically copies the contents of the Red channel to the Alpha channel.
5. Deselect your image by going to "Select -> None" in the menu bar.
6. Check your Channels tab, your Alpha channel should now look like your Red channel.
7. In the menu bar, go to "Colors -> Components -> Decompose". This is going to create and open a new version of your file, with all the channels separated into individual layers.
8. In the pop-up window select RGBA as the Color mode, then hit OK.
9. You should now see a file that looks like your normal map, but in greyscale. In the layers tab you will see that each channel has its own layer now.
10. Before the next step, make sure that your Foreground color is set to black. If it is not, you can click the small icon of a black and white square underneath the color selector to reset it.
11. Click on the "red" layer to set it to your active layer, then in the menu bar go to "Edit -> Fill with FG Color" to fill the layer with black.
12. Go to "Colors -> Components -> Recompose" in the menu bar to recompose the layers back into channels in your original file.
13. Go back to the original file and check the Channels tab to make sure the red channel is just black (empty) now.
14. If your original file seems to still be the same as before, make sure that in the channels tab none of the channels are selected/highlighted. If any of them are, just click on them again to de-select them, then go back to the Decomposed Layers file and repeat step 12 to recompose it.
15. Your recomposed file is now ready to export. Go to "File -> Export As..."
16. Select a save location for your file, and name it to whatever you choose. Check the vanilla game textures to see what naming conventions are used for the files the textures are for. I personally also export a .png version to keep as a backup, but you will need a .dds file for the game. You can either rename your file with the .dds extension, or click "Select File Type (By Extension)" at the bottom to select the .dds file type. Once ready, just click "Export" to save your texture. A window will pop-up with export options. For .png, just leave everything at default.
17. To correctly export your Normal map as a .dds file, make sure to select "BC3/DTX5" for Compression, and "Generate mipmaps" for Mipmaps. Set the Filter under Mipmap Options to "Kaiser".
Physical Map
Basecolor Map
Mask Map
GIMP TL;DR