Dean Vaessen replied 5 years ago

Hey James,

Thank you for the reply. I took some time yesterday to track this issue down and found that I could reproduce the issue with at least one of my marketplace assets. I’m sure I had periodic slowdowns before I installed this asset too, but I’m not sure whether or not that was due to TrueSky. In that light, I’ll consider the original issue as resolved with the suggestion that it might be worth looking in to for you as a way to track a source of performance issues, though you may close the question. If I have further issues with performance I’ll send in some information to the e-mail address you provided.

The asset in question is Waterline Pro (https://www.unrealengine.com/marketplace/en-US/product/waterline), and to the best of my ability to debug, it seems that using TrueSky with a Waterline high simulation accuracy value causes a bigger performance hit than with the default render setup. I thought it was *only* a TrueSky issue, but I eventually found that in some cases with a high enough simulation resolution (an option in their actor), a few extra players (play in editor on dedicated server), and some start/stops, I can sometimes get similar but not completely the same issues with the default Unreal renderer. I say sometimes because it’s a bit tricky to reproduce and it definitely seems easier to get Unreal to trip up with TrueSky active in the scene.

Generally: GPU usage went through the roof, memory consumption grew out of control, and even after stopping the play session it stayed that way (when I quit the session early on, it seemed to recover somewhat after a while). The editor became almost unresponsive and FPS stayed at 10-15fps. Perhaps the editor’s garbage collector kicked in and killed performance for a while? Unreal’s renderer does seem a lot more stable in this scenario and it it’s harder to break in combination with this asset but there have been times where it too acted up.

Upon inspecting the GPU Visualiser, I noticed that the same calls to TrueSky::PostOpaque were referenced within the GPU Visualiser for the water simulation actors that I added. I took a look at their composition and found they have a SceneCaptureComponent2D that captures every tick. When turning this off, this seems to fix the issue. It thus seems like because PostOpaque is quite heavy, it exacerbated the issue (it would run for multiple scene captures it seems like).

In unrelated scenes without Waterline, I did see about 4ms of render time for the PostOpaque call at times. That seems excessive, but perhaps more in line with general performance improvements that I should send in to contact@simul.co?

Dean Vaessen replied 5 years ago

Hi again James,

After I *thought* I had conquered it, I seem to have rejoiced too soon. I managed to make the following render snapshot. It seems to be an issue with the CloudRenderer? Performance was fine initially on Unreal startup, but it regressed after a while. Note that performance was still bad even when the plays ession was stopped in the editor. I can’t seem to add a .txt here but here’s the output:

[trueSKY Profiler]
TOTAL 91.7957
View 32: 1546×1018 1.5098 (1.645%)
RenderMixedResolution 1.3977 (92.580%)
RenderCelestialBackground 0.0025 (0.177%)
CloudRenderer::GetSunOcclusion 0.0000 (0.810%)
RenderSun 0.0024 (96.497%)
RenderPlanets 0.0000 (1.922%)
RenderPlanet 0.0000 (23.645%)
Low-Res 1.3154 (94.109%)
CloudRenderer::Render 1.3154 (100.000%)
Main Render 1.3142 (99.909%)
Lightpasses 0.0012 (0.089%)
LightspaceScatteringVolume 0.0799 (5.713%)
Godrays 0.0608 (76.192%)
Godrays Accumulation 0.0070 (8.775%)
Scattering 0.0116 (14.571%)
Composite 0.0679 (4.500%)
draw 0.0676 (99.457%)
DownscaleDepth 0.0371 (2.459%)
Initial 0.0000 (0.003%)
Effect 0.0000 (0.049%)
Stochastic 0.0163 (43.973%)
Aggregate 0.0208 (55.973%)
Render2DFades 0.0065 (0.429%)
Loss 0.0061 (93.603%)
Inscatter 0.0001 (0.940%)
Skylight 0.0003 (3.880%)
CopyTexture 0.0421 (0.046%)
PreRenderUpdate 90.1651 (98.224%)
PreRenderUpdate 90.1651 (100.000%)
1 0.0268 (0.030%)
BaseSkyRenderer::EnsureTexturesAreUpToDate 0.0132 (49.270%)
MakeLossAndInscatterTextures 0.0000 (0.001%)
LightTable 0.0132 (99.890%)
Lighting Queries 0.0136 (50.648%)
get 0.0000 (0.007%)
set 0.0136 (99.991%)
2 90.1520 (99.985%)
CloudRenderer::PreRenderUpdate 90.1520 (100.000%)
1 0.0000 (0.000%)
2 84.9385 (94.217%)
3 6.1673 (6.841%)
4 0.0260 (0.029%)
3 0.0000 (0.000%)
4 0.0000 (0.000%)
Cubemaps 0.0786 (0.086%)
UpdateSkylights 0.0786 (99.976%)
Skylight Cubemap ? 0.0000 (0.000%)
Mip 0.0000 (0.000%)
Weather Render 0.0000 (0.000%)
RenderMixedResolution 0.0000 (0.000%)
Composite 0.0000 (0.000%)
Render2DFades 0.0000 (0.000%)
Calc Spherical Harmonics 0.0000 (0.000%)
clear 0.0000 (0.000%)
jitter 0.0000 (0.000%)
encode 0.0000 (0.000%)
Read Buffer 0.0000 (0.000%)
text 0.0001 (0.000%)
Time spent waiting for queries: 0.0796ms
CPU
TOTAL 0.0000

Dean Vaessen replied 5 years ago

I will also send this in per email. Not sure if this should be in a different thread because this seems to be different from the PostOpaque calls that I noticed taking longer. Not sure if these issues are related…

Dean Vaessen replied 5 years ago

Sorry for the plethora of comments, but regarding “Not sure if these issues are related…”

To add to this, whenever “PostOpaqueExtensions” start ballooning in Unreal’s own GPU profiling tools, this seems to correspond with the increase in render time of TrueSky’s CloudRenderer::PreRenderUpdate step. So it seems this is the culprit in my case.

Roderick Kennedy Staff replied 5 years ago

That’s correct. PostOpaqueExtensions is where trueSKY hooks in. The trueSKY profiler should produce about the same numbers on average. The issue you’re seeing is a current problem with cloud lighting calculations. We expect to have it fixed in the next few days.