I have never had to do a follow up to any blog post I've ever written; but I feel like I really need to with that last one and clarify a few things.

At the time of publishing I thought I was merely lighting a firecracker, but it seems more like I set off a crate of dynamite.  I knew there was going to be some discussion about the results, but I did not anticipate the nearly 350+ comments.  People are very particular about performance and benchmarking (as it is fair to be).  Everyone is allowed to call BS if they see fit.

It's been three weeks since the article went live. I wanted to take some time for the dust to settle in order to read through what everyone wrote; and respond.  If you haven't been privy to any of the discussion, it's been on /r/cpp and Hacker News.  Along with some talk on Hackaday.

 

"I didn't understand how to use final properly"

I saw this comment pop up a few times; that I missed the point of final.  The proper use of final wasn't the thesis of my article.

There are plenty of resources explaining how to use it and its purpose in the design of a C++ application.  My concern was other articles claiming it can improve performance without a benchmark to back up their statements.  Please read the titles of these articles:

• The Performance Benefits of Final Classes (March 2020)
• Using final in C++ to improve performance (November 2022)
• All About C++ final: Boosting Performance with DeVirtualization Techniques (January 2024)

None of these have any metrics posted.  But all of these titles imply "final makes code go faster".  They all talk about how final is used, including the generated assembly and what's happening at the machine level.  That fills the "how?" and "why?" of final.  But that isn't a benchmark.  To say it improves performance but not have any proof to back it up is dangerous.

For the longest time we have been living in an environment where we skim articles (reading only a headline) and glean information to take it as fact; without actually verifying anything.  Part of my previous blog post was trying to highlight what can happen if you do this.  It is what I did initially, noticed nothing was matching those claims and decided to test it a bit further.

There was one thing I was wrong about: someone else did a benchmark of final in the past.  In their case they found a consistent performance increase.  They even re-ran their benchmark and saw the same results from 10 years ago.  In my case it was faster in some instances, slower in others.  I thank them for reaching out and have updated my older post.

I recall there being a comment about how my "improper" use of final could be a reason why clang had its performance slowdown.  My counter to that: GCC had a consistent performance increase with use of the keyword.  It was used the keyword as intended (and described by the linked articles above).  I put it on many classes that have no further subclassing, and there was a performance boost in this case.  Clang on the other hand, had a decrease with the exact same code.

 

"This isn't a good benchmark"

The previous article was written with the context that others may have read the project's README or have read some of the other prior posts.  Let me rewind a little:

PeterShirleyRayTracing, a.k.a. PSRayTracing (a.a.k.a. PSRT) didn't start out as a benchmarking tool.  I wanted to revisit a ray tracing book series I read when I was fresh out of university (2016), but this time with all the knowledge of C/C++ I had accumulated since that time.  I first went through the books but as an exercise to learn Nim.  Between then and 2020 I had seen images from the book pop up online here and there. Mr. Shirley had actually made the mini-books free to read during that time.  Reading the book's old code and the newer editions, I noticed there were a lot of areas for improvement in performance.  PSRT at first was an experiment in writing performant C++ code, but with some constraints:

1. Has to follow the book's original architecture
2. Needs to be cleaner and modern
3. Must showcase safer C++
   • E.g. The book's old code used raw pointers. std::shared_ptr was used in later versions; I know this is a bottleneck but that is something I've meant to look at later at later on.  (But some other work as been done.)
4. Must be standard and portable
5. Full support for GCC and Clang (then later MSVC)
6. Be "Vanilla C++" as possible.  I don't want to force someone to bring in a hefty library
   • There is an exception for libraries (like PCG32) that allow an increase in performance and are easily integrated like being a single header library
7. Be able to turn on and off changes from the book's original code to see the effects of rewriting parts
8. Extending is okay, but they can't violate any of the above rules
   • E.g. multithreading was added and some new scenes.  The Qt UI is a massive exception to rule no. 6 (but it's not required to run the project)

For the initial revision of PSRT, its code was 4-5x faster than the books' original code (single threaded).  I was very proud of this.

Later on a Python script was added so PSRT could be better tested via fuzzing.  Parameters could differentiate scenes, how many cores to use, how many samples per pixel, ray depth, etc.  It was both meant to check for correctness and measure performance.  The measurement of performance only is the time spent rendering.  Startup and teardown of PSRT is not measured (and it's negligible). This way if I came across some new technique a change could be made and verify it does not break anything from before.  The script has evolved since then.

To explain how the testing and analysis operates a little more simpler:

1. Each scene would be fuzz tested, say three times (real tests do way more), and their runtimes recorded.  Parameters could be wildly different
   • For this example let's say once scene resulted in the times of [3.1, 10.5, 7.4] (real testing used 30 values)
2. Then the same suite would be run, but with a change in code
   • times=[2.7, 8.8, 6.9]
3. From this a percentage difference of each test case would be computed
   • [13%, 17%, 7%]
4. A mean & median per scene could be calculated
   • mean=12.3%, median=13%
   • Each scene is different.  Sometimes radically, other times only slightly.  That's why it's important to look at the results per scene
5. From there a cumulative "how much faster or slower" for the change could be found

I hope this explains it better.

I neglected to mention what compiler flags were used.  All of the code was built with CMake in Release mode.  This uses -O3 in most cases.  This was something I should have specified first.  I know there are other flags that could have been used to eek out some other tiny gains but I do not think it was relevant.  I wanted to use the basics and what most people would do by default.  I also configured the machines to only run the testing script and PSRT.  Nothing else (other than the OS).  Networking was disabled as well so nothing could interrupt and consume any resources available.

 

Simple vs. Complex

One commenter pointed out how they didn't like this, saying that they preferred simpler tests benchmarking atomic units. For example, measuring a bubble sort algorithm and only that.  There are already a plethora of tests out there that do just this.  That isn't good enough. In the real world we're writing complex systems that interact.

Prefer integration tests; verify the whole product works.  Unit testing is good for small components but I only like to do this only when the tiny bits need testing.  E.g. if a single function had a bug and we want to double check it going forward.

 

Other Benchmarks

In all of the comments that I read, I only recall coming across one other benchmark of final; they reported a speedup.  But our methods of testing are completely different.  They were testing atomic components.  Mine was not.

In episode 381 of CppCast it was discussed that there are many practices in the C++ world that are claimed to be more performant without providing any numbers.  To anyone who doesn't think this was an adequate benchmark: Do you have an alternative?  I'm not finding any. If you don't think this was a good benchmark please explain why and tell me what should be done instead.

 

"The author provided no analysis about clang's slowdown"

This is one that I think is a more fair criticism of the article.  In my defense, this is a topic I do not know that much about.  I'm not a compiler engineer, not an expert on the subject of low level performance optimization, nor the inner workings of clang and LLVM.  For earlier development of PSRT, tools like perf, flame graphs, valgrind/cachegrind, Godbolt's Compiler Explorer, etc. were used.  But I do not feel comfortable providing a deep analysis on the issue with clang.

Time could have been spent researching the subject more and doing proper analysis, but this would have taken months.  I did reach out to a friend of mine who works at Apple who provided me with some tips.  Reading the comments on Hacker News, avenues seem to be looking at LTO, icache, inlining, etc. (Tickets have already been filed for further investigation.)

Someone did ask me to check the sizes of the generated binaires with final turned on and off. Devirtualization causing excessive inlining could be the cause. With final turned on, the binary was 8192 bytes larger; I'm not sure how significant that is to impact performance. For comparison, GCC's compiled with final was only 4096 bytes larger than no final. But GCC's binary was about 0.2 MB larger (overall) than clang's. I do not think binary size is a factor.

 

LLVM Engineer

On Hacker News there was a comment left by someone who works on the LLVM project.  Quoting them:

"As an LLVM developer, I really wish the author filed a bug report and waited for some analysis BEFORE publishing an article (that may never get amended) that recommends not using this keyword with clang for performance reasons. I suspect there's just a bug in clang."

1. I am not sure if this was a bug.  I have had performance drops with clang compared to GCC, so I didn't view this as bug worthy.  I checked the LLVM issue tracker in the week after publishing and saw that no one else had.  So I went ahead and filed a ticket.
2. I have amended articles in the past in light of new information.  A previous revision of this project added in the aforementioned Qt GUI.  When I noticed some bugs in Qt, an engineer from the company reached out to me and I updated the original article.  Last week, I thought there were no other benchmarks about final in existence.  I found out I was wrong and my previous article has been adjusted to include that new information.
   
   If there is a bug in clang/LLVM, it becomes fixed, and the slowdown from using final is reduced (or reversed), I will update the article.

 

Random Number Generator Might Be The Cause of Clang's Slowdown

The RNG was already a vector for performance improvement in the past.  Compared with the original book's code, using PCG's RNG showed improved performance over what was available in standard C++.  In the past I was wondering if there could be further improvements in this area.

One reader decided to dig a bit deeper.  That person is Ivan Zechev. He's done some amazing work already and found that the issue with clang might have been related to the RNG and std::uniform_real_distribution.  Calls to logl were not being properly inlined.  And this looks like a long standing issue in clang/LLVM that has never been fixed.

Mr. Zechev sent me a merge request for review, but I have held off on merging it because it actually changed how scenes were set up.  This can drastically alter how long it takes to render an image, because the scene is now different.  In our case, it was book2::final_scene.  At first the floor was completely changed.  Later he was able to correct for that, but other elements were not matching. The uniform distributor (in clang) was producing different numbers with his changes.  For this, I cannot merge.  I commend him for his investigation and will be looking at it in the future.  Thank you.

But this only uncovers a horrible problem: Things in std:: are not portable; which kind of means that the "standard library" really isn't ... well... standard.  In regards to std::uniform_real_distribution there is some more information here.  The C++ standard allows this but it doesn't seem right.

 

"There's no inspection of the assembly"

The other articles have talked about what assembly is generated.  I do not see why it was needed for mine.  What the other articles neglected to do was measure.  This is the gap I wanted to fill in.

I use C++ at the surface level.  I'm pretty sure most people do as well.  Part of the point of having a higher level language is to abstract away these lower level concepts.  PSRT is meant to be written this way; portable, memory safe, modern C++.  Knowing assembly definitely helps, but it should not be a requirement.  This is a C++ project.

---

 

Update May 15th, 2024:

After posting this article on /r/cpp, user /u/lgovedic provided a well thought out comment.  I'd like to repost that here for other readers:

---

 

There will be follow ups and other investigations; but not immediately.  I am willing to amend anything in light of new data.  This is not my full time job and only a hobby project.  Anyone is allowed to contribute and is welcome to do so.

If you're writing C++, there's a good reason (maybe...) as to why you are. And probably, that reason is performance. So often when reading about the language you'll find all sorts of "performance tips and tricks" or "do this instead because it's more efficient". Sometimes you get a good explanation as to why you should. But more often than not, you won't find any hard numbers to back up that claim.

I recently found a peculiar one, the final keyword. I'm a little ashamed I haven't learned about this one earlier. Multiple blog posts claim that it can improve performance^{(sorry for linking a Medium article)}. It almost seems like it's almost free, and for a very measly change. After reading you'll notice something interesting: no one posted any metrics. Zero. Nada. Zilch. It essentially is "just trust me bro." Claims of performance improvements aren't worth salt unless you have the numbers to back it up. You also need to be able to reproduce the results. I've been guilty of this in the past (see a PR for Godot I made).

Being a good little engineer with a high performance C++ pet project, I really wanted to validate this claim.

Update May 3rd, 2024: When posting on /r/cpp, someone else did mention they did some perf testing of final before and had some numbers.  Theirs was from about a decade ago.  I did not find this in my initial searches.  The comment thread and their article can be found here.

---

I keep on finding myself unable to get away from my pandemic era distraction, PSRayTracing. But I think this is actually a VERY good candidate for testing final. It has many derived classes (implementing interfaces) and they are called millions of times in normal execution.

For the (many) of you who haven't been following this project, the quick and skinny on PSRayTracing: it's a ray tracer implemented in C++, derived from Peter Shirley's ray tracing minibooks. It serves mainly an academic purpose, but is modeled after my professional experiences writing C++. The goal is to show readers how you can (re)write C++ to be more performant, clean, and well structured. It has additions and improvements from Dr. Shirley's original code. One of the big features I have in it is the ability to toggle on and off changes from the book (via CMake), as well as being able to supply other options like random seeds, multi-core rendering. It is somewhere 4-5x faster than the original book code (single threaded).

 

How This Was Done

Leveraging the build system, I added an extra option to the CMakeLists.txt:

Then in C++ we can use (ab)use the pre processor to make a FINAL macro:

And easily it can slapped onto any classes of interest:

Now, we can turn on & off the usage of final in our code base. Yes, it is very hacky and I am disgusted by this myself. I would never do this in an actual product, but it provides us a really nice way to apply the final keyword to the code and turn it on and off as we need it for the experiment.

final was placed on just about every interface. In the architecture we have things such as IHittable, IMaterial, ITexture, etc. Take a look at the final scene from book two, we've got quite a few 10K+ virtual objects in this scenario:

 

And alternatively, there are some scenes that don't have many (maybe 10):

 

Initial Concerns:

For PSRT, when testing something that can boost the performance, I first reach for the default scene book2::final. After applying final enabled the console reported:

$ ./PSRayTracing -n 100 -j 2
Scene: book2::final_scene
...
Render took 58.587 seconds

 

But then reverting the change:

$ ./PSRayTracing -n 100 -j 2
Scene: book2::final_scene
...
Render took 57.53 seconds

 

I was a tad bit perplexed? Final was slower?! After a few more runs, I saw a very minimal performance hit. Those blog posts must have lied to me...

Before just tossing this away, I thought it would be best to pull out the verification test script. In a previous revision this was made to essentially fuzz test PSRayTracing (see previous post here). The repo already contains a small set of well known test cases. That suite initially ran for about 20 minutes. But this is where it got a little interesting. The script reported using final slightly faster; wtih final it took 11m 29s. Without final it was 11m 44s. That's +2%. Actually significant.

Something seemed up; more investigation was required.

 

Big Beefy Testing

Unsatisfied with the above, I created a "large test suite" to be more intensive. On my dev machine it needed to run for 8 hours. This was done by bumping up some of the test parameters. Here are the details on what's been tweaked:

• Number of Times to Test a Scene: 10 → 30
• Image Size: [320x240, 400x400, 852x480] → [720x1280, 720x720, 1280x720]
• Ray Depth: [10, 25, 50] → [20, 35, 50]
• Samples Per Pixel: [5, 10, 25] → [25, 50, 75]

Some test cases now would render in 10 seconds, others would take up to 10 minutes to complete. I thought this was much more comprehensive. The smaller suite did around 350+ test cases in 20+ minutes. This now would do 1150+ over the course of 8+ hours.

The performance of a C++ program is also very compiler (and system) dependent as well. So to be more thorough, this was tested across three machines, three operating systems, and with three different compilers; once with final, and once without it enabled. After doing the math, the machines were chugging along for a cumulative 125+ hours. 🫠

Please look at the tables below for specifics, but the configurations were:

• AMD Ryzen 9:
  • Linux: GCC & Clang
  • Windows: GCC & MSVC
• Apple M1 Mac: GCC & Clang
• Intel i7: Linux GCC

For example, one configuration is "AMD Ryzen 9 with Ubuntu Linux using GCC" and another would be "Apple M1 Mac with macOS using Clang". Not all versions of the compilers were all the same; some were harder to get than others. And I do need to note at the time of writing this (and after gathering the data) a new version of Clang was released. Here, is the general summary of the test results:

 

This gives off some interesting findings, but tells us one thing right now: across the board, final isn't always faster; it's in fact slower in some situations. Sometimes there is a nice speedup (>1%), other times it is detrimental.

While it may be fun to compare compiler vs. compiler for this application (e.g. "Monday Night Compiler Smackdown"), I do not believe it is a fair thing to do with this data; it's only fair to compare "with final" and "without final" To compare compilers (and on different systems) a more comprehensive testing system is required. But there are some interesting observations:

• Clang on x86_64 is slow.
• Windows is less performant; Microsoft's own compiler is even lagging.
• Apple's silicon chips are absolute powerhouses.

But each scene is different, and contains a different amount of objects that are marked with final. It would be interesting to see percentage wise, how many test cases ran faster or slower with final. Tabling that data, we get this:

 

That 1% perf boost for some C++ applications is very desirable (e.g. HFT). And if we're hitting it for 50%+ of our test cases it seems like using final is something that we should consider. But on the flip side, we also need to see how the inverse looks. How much slower was it? And for how many test cases?

 

Clang on x86_64 Linux right there is an absolute "yikes". More than 90% of test cases ran at least 5% slower with final turned on!! Remember how I said a 1% increase is good for some applications? A 1% hit is also bad. Windows with MSVC isn't faring too well either.

As stated way above, this is very scene dependent. Some have only a handful of virtual objects. Others have warehouses full of them. Taking a look (on average) how much faster/slower a scene is with final turned on:

I don't know Pandas that well. I was having some issues creating a Multi-Index table (from arrays) and having the table be both styled and formatted nicely. So instead each column has a configuration number appended to the end of its name. Here is what each number means:

• 0 - GCC 13.2.0 AMD Ryzen 9 6900HX Ubuntu 23.10
• 1 - Clang 17.0.2 AMD Ryzen 9 6900HX Ubuntu 23.10
• 2 - MSVC 17 AMD Ryzen 9 6900HX Windows 11 Home (22631.3085)
• 3 - GCC 13.2.0 (w64devkit) AMD Ryzen 9 6900HX Windows 11 Home (22631.3085)
• 4 - Clang 15 M1 macOS 14.3 (23D56)
• 5 - GCC 13.2.0 (homebrew) M1 macOS 14.3 (23D56)
• 6 - GCC 12.3.0 i7-10750H Ubuntu 22.04.3

 

So this is where things are really eye popping. On some configurations and specific scenes might have a 10% perf boost. For example book1::final_scene with GCC on AMD & Linux. But other scenes (on the same configuration) have a minimal 0.5% increase such as fun::three_spheres.

But just switching the compiler over to Clang (still running on that AMD & Linux) there's a major perf hit of -5% and -17% (respectively) on those same two scenes!! MSVC (on AMD) looks to be a bit of a mixed bag where some scenes are more performant with final and others ones take a significant hit.

Apple's M1 is somewhat interesting where the gains and hits are very minimal, but GCC has a significant benefit for two scenes.

Whether there were many (or few) virtual objects had next to no correlation if final was a performance boon or hit.

 

Clang Concerns Me

PSRayTracing also runs on Android and iOS. Most likely a small fraction of apps available for these platforms are written in C++, but there are some programs that make use of language for performance reasons on the two systems. Clang is the compiler that is used for these two platforms.

I unfortunately don't have a framework in place to test performance on Android and iOS like I do with desktop systems But I can do a simple "render-scene-with-same-parameters-one-with-final-and-one-without" test as the app reports how long the process took.

Going from the data above, my hypothesis was that both platforms would be less performant with final turned on. By how much, I don't know. Here are the results:

• iPhone 12: I saw no difference; With and without final it took about 2 minutes and 36 seconds to perform the same render.
• Pixel 6 Pro: final was slower. It was 49 vs 46 seconds. A difference of three seconds might not seem like much, but that is a 6% slowdown; that is fairly significant. (clang 14 was used here BTW).

If you think I'm being a little silly with these tiny percentages, please take a look at Nicholas Ormrod's 2016 CppCon talks about optimizing std::string for Facebook. I've referenced it before and will continue to do it.

I have no idea if this is a Clang issue or an LLVM one. If it is the latter, this may have implications for other LLVM languages such as Rust and Swift.

 

For The Future (And What I Wish I Did Instead):

All in all this was a very fascinating detour; but I think I'm satisfied with what's been discovered. If I could redo some things (or be given money to work on this project):

1. Have each scene be able to report some metadata. E.g. number of objects, materials, etc. It is easily doable but didn't seem worth it for this study of final.
2. Have better knowledge of Jupyter+Pandas. I'm a C++ dev, not a data scientist. I'd like to be able to understand how to better transform the measured results and make it look prettier.
3. A way to run the automated tests on Android and iOS. These two platforms can't easily be tested right now and I feel like this is a notable blindspot
4. run_verfication_tests.py is turning more into an application (as opposed to a small script).
   • Features are being bolted on. Better architecture is needed soon.
   • Saving and loading testing state was added, but this should have been something from the start and feels like more of a hack to me
   • I wish the output of the results were in a JSON format first instead of CSV. I had to fuddle with PyExcel more than desired.
5. PNGs are starting to get kinda chunky. One time I ran out of disk space. Lossless WebP might be better as a render output.
6. Comparing more Intel chips, and with more compilers. The i7 was something I had lying around.

 

Conclusions

In case you skimmed to the end, here's the summary:

• Benefit seems to be available for GCC.
• Doesn't affect Apple's chips much at all.
• Do not use final with Clang, and maybe MSVC as well.
• It all depends on your configuration/platform; test & measure to see if it's worth it.

Personally, I'm not turning it on. And would in fact, avoid using it. It doesn't seem consistent.

For those who want to look at the raw data and the Jupyter notebook I used to process & present these findings, it's over here.

---

If you want to take a look at the project, it's up on GitHub (but the active development is done over on GitLab). Looking forward to the next time in one year when I pick up this project again. 😉

Update May 3rd, 2024: This article has generated quite a bit more buzz than I anticipated.  I will be doing a follow up soon enough.  I think there is a lot of insightful discussion on /r/cpp and Hacker News about this.  Please take a look.