Friday Diversions

3D World recently published the first official "review" of Brazil r/s v2.0. The problem though, is that it wasn't a straight-up review, it was primarily a comparison between Brazil 2 and V-Ray 1.5. At SplutterFish we were really looking forward to this review -- Brazil 2 is really a great improvement over Brazil 1 -- we knew that and we didn't think that a straight review would see anything but that.... except the "review" is primarily a comparison, and the comparison wasn't very rigorous or fair.

To start, here's one thing we know: Brazil 2.0 is fast. Very fast. Much faster than Brazil 1, and, we believe, on par with V-Ray on straight GI rendering. When you take GI rendering and start combining multiple "blurry" effects and features, Brazil kicks V-Ray's butt.

The most repeated claim in the "review," based on a single comparison scene, is that Brazil is slower than V-Ray on interiors. We're calling, "bullshit." The rendertimes stated in the article immediately threw up red flags among everyone within SplutterFish and among many Brazil artists -- especially when you see how different the rendertimes were. Saying that "Brazil is slower than V-Ray" is not however, the only conclusion of the article, and I don't want to give the impression that it was a hit-piece against Brazil. The fact is that the review is actually quite positive about Brazil 2, and I'd like to thank the publisher and the authors for not only the review, but also for making the scene files available so that we could look more closely at what was going on.

If you go to 3D World's blog, you can download the scene files and high-res versions of the imagery from the article.

By looking at the imagery you can see several things right off the bat that would lead one renderer to appear faster than another. Here's a quick run-down of the more obvious things:

• The general tonal quality of the images differ greatly. If you're going to do a comparison, render everything out vanilla and apply the same tone mapping in post -- tone mapping and gamma correction can hide defects, so it's best to keep the comparisons the same.
• Area shadow size: bigger in Brazil, looks nicer, but big area shadows are obviously more costly to calculate.
• The lighting gradients are pretty smooth in the Brazil version -- very grainy in the V-Ray version. Grainy means faster -- again, the comparison isn't fair if you're going to hold the renderers to different standards.
  
• There's a lack of contrast on the ceiling of the v-ray image -- is there a bumpmap missing?... or being ignored by the renderer in the V-Ray version? I don't know the cause, but there's clearly a difference. Calculating proper normal deflections is more costly than just faking it, so knowing what's going on to cause the visual difference is important to a real apples to apples comparison.
  
• There's a round illuminated object in the Brazil scene that's not in the V-Ray image -- extra illuminators aren't going to help rendertimes.
• The floor in the Brazil image clearly has a bump texture that matches the pattern of the wood boards -- that's missing in the V-Ray image -- is the bump missing or is V-Ray ignoring the extra calculations to generate the normals for the bump? (see the comment above about the ceiling texture)
  
• In the coffee-table, the V-Ray image is missing the reflections of the bowl and the rest of the room. Things are obviously going to render faster if you allow elements to ignore other elements -- for whatever reason.

Looking into the scene files themselves we start seeing a few other odd things:

• First, the scene scale is hard to work with -- this isn't a performance issue at all, but if any of you've downloaded the files are are checking them out yourself, make sure you set your scene scale to 1 unit = 1 mm
• rendering the V-Ray scene with V-Ray results in a much darker image and the floor has issues. Not entirely sure we've got the same scene file that was used in the review.
  
• GI bounces set quite high w/ GI skylight bounce and no portals -- may be due to using a V-Ray workflow in Brazil. We heard a report about V-Ray sometimes renders with too much contrast -- the fix is that V-Ray users will crank the bounces too high to get light to soften the contrast. These setting may have been unnecessarily just "copied" to the brazil file when in Brazil they're just overkill.
• detail enhancement seems too high
• The rendercache search radius is smaller than photon radius: This isn't really a problem, but in conversation with some of our testers it sounds like this may indicate a workflow difference that might lead to results that adversely impact performance
  
• The rendercache is using an adaptive size rather than a fixed size -- adaptation is better for detail but can be costly and create unexpected noise
• jitter samples is off -- having this on is computationally free and helps hide regularities that might otherwise be perceptable, which in turn can allow you to dial down some settings.
• The raytrace accelerator is set to "large dataset" -- This is not necessary for this scene and it's clearly slower. One click and a re-render would have made this immediately obvious. The default accelerator is the fastest in this case, so this value was unnecessarily and intentionally changed.
  
• The irradiance cache sampling looks to be unnecessarily high

So we had one of our guys take a look at the scene. He has different hardware, so let's do some analysis on the rendertimes: In the review, there's 3 rendertimes of interest: the Brazil 2 rendertime, the V-Ray qmc rendertime, and a V-Ray lightcache rendertime. For those tests, the author's rendertimes, in minutes:seconds, were 46:31, 31:08, and 15:52 respectively, with the V-Ray version ending up being almost 3 times faster than the Brazil version! Our tester's rendertimes on the files from 3D world were approximately 17:00, 11:00, and 6:00 -- So the proportions are about the same, it's just rendered on a faster machine.

In short order, our tester had matched the Brazil 2 scene's rendertime to the vray lightcache render -- meaning he had sped the rendertime of the Brazil scene up by a factor of 3. Twenty minutes later, he'd knocked the Brazil rendertime down to 5:14, beating the V-Ray rendertime, without sacrificing the higher quality details seen in the Brazil image from the article.

Then we said, "ok, let's really go for it here. Let's lower some settings until the Brazil image is, subjectively, no worse than the V-Ray image, and really set the scene up to run as fast as we can get it." Here's what he came up with:

2 minutes and 44 seconds! More than twice as fast as V-Ray! For reals! And he never really could get some of the quality levels as low as what's seen in the V-Ray image. Here's a snapshot of a white plaster version of the model so you can see the quality of the lightflow, and a couple more images showing off the raw unfiltered photons and the Brazil irradiance cache. The new scene file is attached here.

Conclusions

1. Apples to apples comparisons on complex scenes are not easy, and care must be taken to ensure that you're holding both renderers to the same standard. This is why most rigorous performance tests use more controlled scenes and carefully analyze the the interaction of features and settings. Our best guess is that in this case the V-Ray setting were simply copied into the Brazil scene without regard for the quality differences or the level of overkill that some of those settings were creating.
   
2. Brazil is faster than V-Ray on interiors!

It's nice to have this file to work with, and again I thank the publisher for sharing these with us. In a future post I'd like to go into more details on the types of things that were done to get the scene faster.

Comments are open -- keep our comment policy in mind -- and yeah, I'm a big meeny :P