Simul Support ExchangeCategory: Sky and LightingPostOpaque destroys performance
Dean Vaessen asked 4 years ago

Periodically my gpu usage goes through the roof and for a while my performance drops extremely low. 10-20fps. Eventually it recovers for a few moments and runs smoothly again. This is both while working and playing in the editor.
I checked the GPU visualiser, and noticed the issue is the PostOpaque function. I’m running .24.3 (though this has been going on for longer) and the latest ccd release, and have also updated my hardware drivers just in case.
I found a few people with the same issue on the Unreal forums, one person resolving it by getting a better GPU. In my case, I run a 2080 and the level I had loaded in the editor was extremely basic so it has to be some sort of other issue.
Is this resolvable?

Dean Vaessen replied 4 years ago

I am closing this ticket and re-filing in the proper category.

Dean Vaessen replied 4 years ago

Seems like I can’t hide this post so I won’t repost after all. I wanted to move this to the plugin category as I am using the Unreal plugin.

James Evans Staff replied 4 years ago

Thanks, this category is fine. Ill look into why you cant hide a post!

2 Answers
James - Simul Support Staff answered 4 years ago

We are aware of certain performance issues currently, and are always looking into them. We have a section in the documentation about performance profiling. If you could do this that would be very helpful. If you can attach a .txt here please do that, or email it to contact@simul.co 

https://docs.simul.co/programming/performance.html

Dean Vaessen replied 4 years ago

Hey James,

Thank you for the reply. I took some time yesterday to track this issue down and found that I could reproduce the issue with at least one of my marketplace assets. I’m sure I had periodic slowdowns before I installed this asset too, but I’m not sure whether or not that was due to TrueSky. In that light, I’ll consider the original issue as resolved with the suggestion that it might be worth looking in to for you as a way to track a source of performance issues, though you may close the question. If I have further issues with performance I’ll send in some information to the e-mail address you provided.

The asset in question is Waterline Pro (https://www.unrealengine.com/marketplace/en-US/product/waterline), and to the best of my ability to debug, it seems that using TrueSky with a Waterline high simulation accuracy value causes a bigger performance hit than with the default render setup. I thought it was *only* a TrueSky issue, but I eventually found that in some cases with a high enough simulation resolution (an option in their actor), a few extra players (play in editor on dedicated server), and some start/stops, I can sometimes get similar but not completely the same issues with the default Unreal renderer. I say sometimes because it’s a bit tricky to reproduce and it definitely seems easier to get Unreal to trip up with TrueSky active in the scene.

Generally: GPU usage went through the roof, memory consumption grew out of control, and even after stopping the play session it stayed that way (when I quit the session early on, it seemed to recover somewhat after a while). The editor became almost unresponsive and FPS stayed at 10-15fps. Perhaps the editor’s garbage collector kicked in and killed performance for a while? Unreal’s renderer does seem a lot more stable in this scenario and it it’s harder to break in combination with this asset but there have been times where it too acted up.

Upon inspecting the GPU Visualiser, I noticed that the same calls to TrueSky::PostOpaque were referenced within the GPU Visualiser for the water simulation actors that I added. I took a look at their composition and found they have a SceneCaptureComponent2D that captures every tick. When turning this off, this seems to fix the issue. It thus seems like because PostOpaque is quite heavy, it exacerbated the issue (it would run for multiple scene captures it seems like).

In unrelated scenes without Waterline, I did see about 4ms of render time for the PostOpaque call at times. That seems excessive, but perhaps more in line with general performance improvements that I should send in to contact@simul.co?

Dean Vaessen replied 4 years ago

Hi again James,

After I *thought* I had conquered it, I seem to have rejoiced too soon. I managed to make the following render snapshot. It seems to be an issue with the CloudRenderer? Performance was fine initially on Unreal startup, but it regressed after a while. Note that performance was still bad even when the plays ession was stopped in the editor. I can’t seem to add a .txt here but here’s the output:

[trueSKY Profiler]
TOTAL 91.7957
View 32: 1546×1018 1.5098 (1.645%)
RenderMixedResolution 1.3977 (92.580%)
RenderCelestialBackground 0.0025 (0.177%)
CloudRenderer::GetSunOcclusion 0.0000 (0.810%)
RenderSun 0.0024 (96.497%)
RenderPlanets 0.0000 (1.922%)
RenderPlanet 0.0000 (23.645%)
Low-Res 1.3154 (94.109%)
CloudRenderer::Render 1.3154 (100.000%)
Main Render 1.3142 (99.909%)
Lightpasses 0.0012 (0.089%)
LightspaceScatteringVolume 0.0799 (5.713%)
Godrays 0.0608 (76.192%)
Godrays Accumulation 0.0070 (8.775%)
Scattering 0.0116 (14.571%)
Composite 0.0679 (4.500%)
draw 0.0676 (99.457%)
DownscaleDepth 0.0371 (2.459%)
Initial 0.0000 (0.003%)
Effect 0.0000 (0.049%)
Stochastic 0.0163 (43.973%)
Aggregate 0.0208 (55.973%)
Render2DFades 0.0065 (0.429%)
Loss 0.0061 (93.603%)
Inscatter 0.0001 (0.940%)
Skylight 0.0003 (3.880%)
CopyTexture 0.0421 (0.046%)
PreRenderUpdate 90.1651 (98.224%)
PreRenderUpdate 90.1651 (100.000%)
1 0.0268 (0.030%)
BaseSkyRenderer::EnsureTexturesAreUpToDate 0.0132 (49.270%)
MakeLossAndInscatterTextures 0.0000 (0.001%)
LightTable 0.0132 (99.890%)
Lighting Queries 0.0136 (50.648%)
get 0.0000 (0.007%)
set 0.0136 (99.991%)
2 90.1520 (99.985%)
CloudRenderer::PreRenderUpdate 90.1520 (100.000%)
1 0.0000 (0.000%)
2 84.9385 (94.217%)
3 6.1673 (6.841%)
4 0.0260 (0.029%)
3 0.0000 (0.000%)
4 0.0000 (0.000%)
Cubemaps 0.0786 (0.086%)
UpdateSkylights 0.0786 (99.976%)
Skylight Cubemap ? 0.0000 (0.000%)
Mip 0.0000 (0.000%)
Weather Render 0.0000 (0.000%)
RenderMixedResolution 0.0000 (0.000%)
Composite 0.0000 (0.000%)
Render2DFades 0.0000 (0.000%)
Calc Spherical Harmonics 0.0000 (0.000%)
clear 0.0000 (0.000%)
jitter 0.0000 (0.000%)
encode 0.0000 (0.000%)
Read Buffer 0.0000 (0.000%)
text 0.0001 (0.000%)
Time spent waiting for queries: 0.0796ms
CPU
TOTAL 0.0000

Dean Vaessen replied 4 years ago

I will also send this in per email. Not sure if this should be in a different thread because this seems to be different from the PostOpaque calls that I noticed taking longer. Not sure if these issues are related…

Dean Vaessen replied 4 years ago

Sorry for the plethora of comments, but regarding “Not sure if these issues are related…”

To add to this, whenever “PostOpaqueExtensions” start ballooning in Unreal’s own GPU profiling tools, this seems to correspond with the increase in render time of TrueSky’s CloudRenderer::PreRenderUpdate step. So it seems this is the culprit in my case.

Roderick Kennedy Staff replied 4 years ago

That’s correct. PostOpaqueExtensions is where trueSKY hooks in. The trueSKY profiler should produce about the same numbers on average. The issue you’re seeing is a current problem with cloud lighting calculations. We expect to have it fixed in the next few days.

Gombor Gábor answered 4 years ago

What is the status about PostOpaque optimization?
I still not decided about my sky implementation. TrueSky quality is the best, but if you create a game, you cant afford it consumes 1/3 GPU speed. If you can optimize it, the answer is obvious, if not then the plugin is sadly only usable for demonstrations but not for games 🙁