Tuesday, 12 January 2010

3d: mental ray_The usual suspects: Displacement and Proxy errors

The above image was produced whilst at GMJ Design ltd

Although we had meticulously covered numerous ways of optimizing the 3D scenes and rendering times in our latest book, the following tips will further help you to address other common memory errors.

With displacement and proxies,the memory is only allocated at render time. So even when your small test renders seem ok and the FG files had been cached, your computer may still run out of memory when rendering the final high resolution image.

Overcoming the problem:

1- Ensure that you have opened the mental ray message window dialog box with all its functions checked, in order to follow the rendering process and detect any problems, whilst paying attention to the buckets being rendered on frame buffer.

2-Proxy errors are often only detectable by looking at the last rendered buckets computed on the frame buffer before the crash, and the last messages from mental ray messages window dialog box.

Normally it should read as follows: “...mip binaryproxy= Loading ...bytes geometry.”

Followed by this message “... progr= scene cache flushed asynchronously ...MB for module job, now: ...MB” and the final mental error message dialog box: “The render was cancelled due to insufficient memory...”.

These errors can be rectified by first changing the ray tracing acceleration to BSP2 type.

Next, ensure that you reduce the amount of copied proxies in your scene. By that, copy instance all similar plants, grass or/and trees in the scene(or other types of geometry) . If there are too many sets of different plants and trees in the scene, then it would be worth attaching them as normal copied meshes first, and later convert them into proxies.

This exercise is to reduce the amount of copied proxies in the scene. The ideal situation would be to have mental ray load up proxies only once (i.e. multiple instanced proxies or one big proxy with different sets of plants, trees etc, originally attached together as one mesh).

Finally, reduce the density of the relevant geometry and increase the page file as previously shown. The process of creating and working with proxies is covered in detail in the 2nd edition of my book with Roger.

3-Displacement errors are easier to detect and tackle. The mental ray message should indicate the “rogue” object/s’ name and mention “...retesellation...”: “...ms from ground (geomObject(mesh00)...”.

followed by “...aborting, not possible to avoid retesellating...”; “...scene cache flushed asynchronously...”.

And the final mental ray error dialog box reading: “The render was cancelled due to insufficient memory...”.

To solve this problem, simply open the renderer tab, on shadows and displacement parameters, under displacement (global settings), change the Max. Subdiv value to 1k or lower. This should be sufficient. Note that, values lower than 256 may result in a very faceted displacement (i.e. not accurate), so it is worth gradually decreasing/test rendering the values without compromising the quality too much.

Note: In addition to the above mentioned,in very extreme and difficult cases of memory loss, one can additionally enable the "use fast rasterizer (rapid motion blur) function, from the "rendering algorithms" parameters.
This rendering method will bypass most mental ray memory issues.

It is worth noting that although very powerful, this rendering algorithm disables some of render elements. To override this, simply render the file output to an EXR file extension type, provided one has the material IDs/Object IDs,etc, originally enabled in 3Ds Max.

If facing difficulties extracting these EXR passes in Photoshop,After Effects,etc;simply switch your 3Ds Max back to standard mental ray rendering algorithm when computing your AO pass, and enable your rendered elements again; with cached FG at a very low res to render the final output in higher res.

I hope you found this post useful.



My 3D Portfolio:

More tips and Tricks:

Post-production techniques

Tips & tricks for architectural Visualisation: Part 1

Essential tips & tricks for VRay & mental ray

Photorealistic Rendering

Creating Customised IES lights

Realistic materials

Creating a velvet/suede material 







  1. Thanks man! That saved me last nite!!!! I had a lot of displacement objs and I hadn't change the default setting of max subdivisions from 16k. It grew out of memory every time I tried to do a production render (resolution 2048x1536 and up) and I couldn't find out why... I had an endline of monday morning to submit the renderings, I had reached morning time and I was about to give up until I saw your blog post... You save my @$$, man! Thanks, thanks, and thanks again!

  2. No problem manolis; you are welcome!!!

    Also, do encourage others to post their mental ray problems here.I will do my best to solve them.



  3. I'm getting this error on our render nodes. The scene has no mr proxies, or 3d displacements. I'm also running max 2010, which as I understand it, handles the bitmap paging automatically. I've saved out the FG cache files and am reading the file sequence back in. Using BSP2 as well. Is there anything else I can tweak to resolve this?

  4. Hi there,

    As suggested earlier:"...simply enable the "use fast rasterizer (rapid motion blur) function, from the "rendering algorithms" parameters. This rendering method should bypass most mental ray memory issues.
    It is worth noting that although very powerful, this rendering algorithm disables some of render elements. To override this, simply switch back on to your standard rendering algorithm when computing your AO pass, and enable your rendered elements again; with cached FG at a very low res to render at higher res".

    Also, I have some questions:

    1-Have you got too many reflective surfaces?
    If so, there is an old post about BSP values in this blog, where the following was mentioned: "...Reduce the global reflection/refractions parameters if there are too many reflective objects in the scene (i.e. max trace depth= 3; max reflections=1 or 2; max refractions=3). These values should be reduced with caution, as very small values of "depth" and "refractions" may cause artifacts(i.e. black blotches)..."

    2-How big is your 3Ds Max file? If it's bigger than 90MB, you should seriously consider converting some of your mesh to mr proxies, to keep your 3D scene under 90MB .As mentioned earlier, proxies are covered in detail in our latest book.

    Finally, also consider using the strip render function from 3Ds Max Networkrender.This subject is also discussed in detail in our latest book.

    The above mentioned should further help you address any memory errors.



  5. Moreover,the following was also mentioned in the " Decoding BSP mental ray BSP tree" post of my blog:

    "...Reduce the number of lights casting soft shadows, if many in the scene (i.e. have few key lights casting shadows, and the remaining lights should only illuminate the scene without casting shadows). This technique will reduce the rendering times by half..."

    "...Have the correct sampling quality (i.e. maximum samples per pixel value should not be higher than 16)..."



  6. Please, something to vray problems...

  7. Hi Lee Bano,

    Thanks for your feedback here!!

    Did you mean that I should also post something about common VRay problems?



  8. Hi,I´m from Brazil,I have a blog about my works in 3D,I would like that you look my works,I can see that you must know a lot of things about 3d

    it´s my blog


    I would like you writte anything about my works or post a comment, pleeaseeee (y)

  9. Hi there, great post!

    I use your book often, but unfortunately I have the first edition which doesn't deal with proxies. :)

    I have a question... I'm rendering a scene with a large building, 3 types od proxy trees all instanced around, and a huge displaced grass surface. I have a phenom X6 with 8 gb of ram and win7 64 bit, max 2010. I knew from before I should use BSP2, scanline off, placeholders and all that stuff. But the thing is, I still get out of memory errors constantly. Everything renders really fast, when it renders, but somewhere around 50% of the render it stops with an out od memory error. It's driving me crazy :\ Any help would be appreciated.

  10. Hi Hrvoje,

    Thanks for posting your query here!

    In order to accurately diagnose the problem, I need to see the relevant mental ray message errors.
    However, I am confident that you will be able to find the answer/s on this blog.
    There is only a hand full of reasons why you are being prompted with such errors:

    1-Poor usage of displacement settings

    2-Poor usage of proxies and/or mesh in your scene

    2-Frame buffer display ( one should use the strip render to correct this error)

    3-Final gather computation (one should cache the FG and geometry prior to sending the final big render)

    All the answers for above errors are listed on this post and other older posts of my blog:

    1- http://jamiecardoso-mentalray.blogspot.com/2010/04/decoding-bsp.html

    2- http://jamiecardoso-mentalray.blogspot.com/2009/11/out-of-memory-issues_14.html

    Finally, for what it's worth, from my humble experience, it is not about enabling all possible mental ray settings. It is about finding out the root of the problem at hand...and tackling it accordingly.



  11. Where can i download a good proxy breaker that will work on xp home edition? One with no viruses or errors?

    Brad Fallon

  12. Hello
    Thanks for such post. I will use it for my future project. However I'm thinking of a huge switch toward Pixar Renderman, which give fair programming capabilities.

    My Website