This is mainly due to the fact that Max cannot free up enough memory necessary to process the full render.
There is a quick fix for this however, it is strongly recommended to ultimately add more memory to the computer(i.e.4GB/8GB+) whilst running 64bit applications for Max and windows.
Also, check my New Book: 3D Photorealistic Rendering: Interiors & Exteriors with V-Ray and 3ds Max
The following “trick” may or may not work for 32bit applications however, it definitely works for the 64bit.
1-On start up windows, choose the control panel option.
2-In control panel, choose system.
3-In the system settings, go to advanced system settings.
4-The system properties dialog should open.
5-In the performance group, click on the settings button. The performance options dialog should open.
6-Click on the advanced tab; under the virtual memory group click on the change button. The virtual memory dialog should appear.
7-Uncheck the automatically manage paging file size for all drives function, to enable the paging file for each drive settings.
Choose your C drive as your volume label of choice, and check the custom size function. The initial size (mb) function should be equal to 1.5 times of your total memory (RAM) value (i.e. in my case I have 8GB, therefore 12GB).
The maximum size (mb) should be equal to at least 3 times of your total memory size(i.e. 24GB).
It is worth mentioning that some users that have huge amounts of free hard disk space(i.e. 200GB C drive space) tend to set their maximum size(mb) value to about 60GB or more.
Moreover, prior to setting the maximum size, ensure that you have at least 3 times freer disk space than the maximum size value.
Next, click the set button to set your parameters, followed by ok. The system properties warning should pop up, click ok to close it.
8-Back on performance options, click ok, followed by clicking "apply" to close it.
9-The windows restart warning dialog should pop up, choose the appropriate option.
The changes will only take effect once the computer is restarted. Also, it is recommended to empty one’s page file from time to time.
It is worth mentioning that, the above procedure will not speed up your renderings however, it will ensure that you never run out of memory when it is needed.
To speed up renders one has to ultimately add extra processors (i.e. 2.80 GHZ or higher) and more memory with 64bit applications (i.e. 4GB/8GB+).
Finally, in very extreme and difficult cases of memory loss, one can additionally enable the "use fast rasterizer (rapid motion blur) function, from the "rendering algorithms" parameters.
This rendering method will bypass most mental ray memory issues.
Note: Although very powerful, this rendering algorithm disables some of render elements. To override this, simply render the file output to an EXR file extension type, provided one has the material IDs/Object IDs,etc, originally enabled in 3Ds Max.
If facing difficulties extracting these EXR passes in Photoshop,After Effects,etc;simply switch your 3Ds Max back to standard mental ray rendering algorithm when computing your AO pass, and enable your rendered elements again; with cached FG at a very low res to render the final output in higher res.
For more information about tackling memory issues,please check my other posts in this blog:
mental ray_the usual suspects:Displacement and proxy errors
Decoding mental ray BSP tree
I hope you have found this post interesting.
My 3D Portfolio:
New Book: 3D Photorealistic Rendering: Interiors & Exteriors with V-Ray and 3ds Max
More tips and Tricks:
Tips & tricks for architectural Visualisation: Part 1
Essential tips & tricks for VRay & mental ray
Creating Customised IES lights
Creating a velvet/suede material