Compositing 3ds Max 9 Renderings in Photoshop CS3

Scott Onstott
Book & Video Author

Compositing is the art of creating a new image by combining images from different sources. In this project, I will be rendering multiple lighting components from a 3D model in Autodesk 3ds Max 9 and combining the results in a new image using Adobe Photoshop CS3 Extended.

The technique presented here is one possible solution for overcoming what is a problem with all high-end 3D packages—excessive rendering times for generating acceptable presentation imagery. I'm not saying that 3D compositing will yield higher quality imagery than what is currently possible using 3ds Max, Maya, Softimage|XSI, (or similar) and a sophisticated renderer such as mental ray. Instead, 3D compositing is a quick process for achieving an improved level of realism from relatively simple, quickly rendered image inputs.

There is no getting around the need for a well-built 3D model. Here I have opened a 3D model of the Gamble House in 3ds Max 9. I created a camera, set the target render size, turned on safe frames, and composed the scene. There is one direct light in the scene and all the 3D surfaces have materials, many with texture maps and mapping coordinates. You can download the sample files here (7 mb).

Next on the agenda is to have 3ds Max render specific components of the total lighting solution. In addition to the regular Scanline rendering, we want Diffuse, Shadow, Z Depth, and Specular components rendered out to files. These are called render elements in 3ds Max parlance; they are added on a tab of the same name in the Render Scene dialog box.

The Z Depth render element deserves special mention because it has its own rollout at the bottom of the Render Scene dialog. You must enter Z Min and Z Max distances in that rollout to get a usable rendering of this element. I suggest using a Tape helper to measure the distance from the camera to the front and back sides of the building. These distances are approximately 85' and 235' in this project. Z Depth represents distance in grayscale as measured along the axis that goes directly into the picture plane. It useful for simulating fog and depth-of-field in Photoshop.

Render all elements and save them as separate .TIF files on disk. Render and save the scanline image as well (this is what you see by default in the Rendered Frame window). The sample file is set up to do all this automatically when you render.

Another very helpful component is called ambient occlusion (called clay rendering or dirt map in some renderers), and it requires a bit more setup to render out to a file. Ambient occlusion is not classified as a render element per se, but is rendered separately by the mental ray renderer. The interesting thing about ambient occlusion is that it can give you very attractive, soft shadows in any compositing project. Ambient occlusion is calculated from the ambient light in the scene, independent of any specific light source.

In a nutshell, to generate an ambient occlusion rendering in 3ds Max or VIZ, use the following steps. Open the Render Scene dialog and open the Assign Renderer rollout at the bottom of the Common tab. Set the Production Renderer to mental ray.

Next, open the Material Editor and create a new mental ray material. Assign the Ambient/Reflective Occlusion map to the Surface component of the material. Open the Render Scene dialog alongside the Material Editor and switch to its Processing tab. Enable the Material Override feature and drag the material you just created from the Material Editor into the Material Override slot in the Render Scene dialog box.

Do a test rendering and watch as the mental ray buckets appear in the Rendered Frame window. If the rendering is coming in with coarse grain, hit Esc to stop the rendering. Go back to the Render Scene dialog and switch to the Renderer tab. Improve the sampling quality by increasing Samples per Pixel (both Min and Max) and by decreasing the Spatial Contrast values. Render again and it will take longer but the grain will be fine. Save the ambient occlusion image and close 3ds Max.

Shown below are all the separate image components which we will now composite in Photoshop.

Launch Photoshop and choose File > Scripts > Load Files into Stack, if you are using CS3 Extended. Select the component files you rendered in 3ds Max and click OK to stack the images as layers in one document. Do not use the create smart object option in the script. If you're using another version of Photoshop, open all images and drag each image with the Move tool, one at a time into one document. Hold down the Shift key while moving layers between documents to keep the layers aligned.

You'll notice that the Shadow layer is completely black. It is a special render element and must be composited manually. Delete the Shadow layer and open Shadow.tif in its own window. Switch to the Channels dock and observe that it has an Alpha 1 channel that actually reveals the shadows. This channel must be converted into a layer in order to composite it into the larger project. Select the Alpha 1 channel, then the Select All command, and Copy to the clipboard.

Switch back to the Layers dock and create a new layer. Paste the selection from the clipboard. The shadows appear white on a black background. Invert the image by pressing Ctrl+I (Mac: Command+I). Now the shadows are more properly black on a white background. Press V (Move tool), hold down Shift, and drag this layer into your project document. Rename the layer Shadow.

The Z Depth layer has the opposite problem as compared to the Shadow layer. In order to use Z Depth as a depth map for the Lens Blur filter, it must be a channel instead of a layer. Target the Z Depth layer, Select All, and Copy to the clipboard. Switch to the Channels dock, create a New channel, and Paste from the clipboard. Switch back to the Layers dock and Delete the Z Depth layer.

Compositing is really more of an art than a science. Photoshop compositing involves playing with blend modes, layer opacities, layer order, layer masks, adjustment layers and clipping masks—all the while evaluating the composite with an aesthetic eye. Once you get up to speed with Photoshop, these skills will be second nature. The following image shows what I ended up with in the Layers and Channels docks.

Probably the best way to understand the Photoshop side of this technique is to open the sample PSD file and investigate it yourself. Toggle layers and masks on and off and kick the tires. Let me give you a few pointers to help you understand my workflow. I started by creating a Gradient Fill layer at the bottom of the Layer stack. I customized the gradient to blend turf into sky in the distance.

Notice that many of the layers have masks—most of these are identical to the Alpha 1 channel. Ctrl-click the Alpha 1 channel to load it as a selection. Switch back to the Layers dock and create a Mask and there you have it.

While I was compositing, I found that some of the layers were too dark. I compensated by adding Exposure adjustment layers and taking the exposure up a few notches. Be warned, however, that adjustment layers affect everything below them in the Layer stack—and that's usually not what you want when compositing. To make an adjustment layer affect only the layer below it, create a Clipping Mask. This is done simply by Alt-clicking (Mac: Option-clicking) the border between the adjustment layer and the layer below it. I color-coded layers with their adjustments, so you can see at-a-glace what goes together.

The Shadow layer's blend mode is set to Multiply and the Specular layer's blend mode is Screen. I found that the Scanline layer looked best with Lighten as its blend mode. Notice that the Shadow layer is a smart object with Gaussian Blur assigned as a re-editable smart filter (new in CS3). I blurred the Shadow layer to soften its hard-edged raytraced shadows.

The Ambient Occlusion layer's blend mode is set to Multiply because it is a type of shadow layer. I custom-painted a mask to hide portions of the Ambient Occlusion layer's contribution to the overall composite. I thought the area under the second floor deck was too dark, so I masked out its ambient occlusion by painting in black with a large soft brush.

The top and final layer started out as a Stamp of all visible layers. This should be done only after you are satisfied with how all visible layers appear by pressing Alt+Shift+Ctrl+E (Mac: Option+Shift+Command+E). Then I fired up the Lens Blur filter and selected the Z Depth channel as its Depth Map. The lens blur is then applied to objects in the distance first, fading into sharpness as the Z Depth channel gets lighter.

Anyway, there should be a lot here to chew on. Once properly digested (read practiced), these techniques can really be applied quickly to just about any project. I hope you'll agree that the image below is more than the sum of its parts. In my opinion, it is much more realistic than what the regular Scanline rendering in 3ds Max offers, and it took me much longer to write this article than to render and composite the entire project using 3ds Max and Photoshop.

A video showing this technique is available on my free video podcast, The Digital Architect.

About the Author

Scott Onstott is a book and video author of AEC software tutorials. He has a degree in architecture from UC Berkeley and has served as an instructor there, in addition to working in several prominent engineering, architecture, and interiors firms in San Francisco. He has also worked as a technical editor and technology consultant.

Scott has contributed to over two dozen books and videos on AutoCAD, Architectural Desktop, VIZ Render, Revit, 3ds Max, VIZ, Photoshop, Illustrator, Painter, Fireworks, and Dreamweaver. He most recently co-authored AutoCAD:_Professional_Tips_and_Tricks with Lynn Allen. He can be reached via: www.ScottOnstott.com.