New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
3080 & 3090 coumpute capability 86 degraded performance after some updates #44116
Comments
Can you please report this on the NVIDIA developer forum? CC @nluehr We can circle back here if/when this is triaged down to an issue with the TF nightly and/or TF release builds. |
3090's performance on 20.10 tf1 ngc container is even 15-20% better than on 20.08, so I guess we can just agree that we should never use 20.08 container because it sucks and let go of this issue. I'll later also try to run some tests on 20.10 tf2 and report back. Edit: seems like might be very different for different cases, though. Got to test more. Will report later. |
Ran extensive benchmarks https://fsymbols.com/3080-3090-benchmarks/ |
(And the performance was pretty inconsistent, you better take a look.) |
Retested on 20.11 container. 3080 performance still effed up https://fsymbols.com/3080-3090-benchmarks/ It still has Cudnn 8.04 and same CUDA version as 20.10 container, though. |
Could you please test in the latest |
This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you. |
Closing as stale. Please reopen if you'd like to work on this further. |
This issue is apparent from the difference in performance in NGC containers https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow . For 20.08 se-resnext101 example training performance
(have to adapt directories)
on 3080 is around 370-400 img/sec. While on 20.09 container it's more like 115 img/sec. This is also similar for resnet-50 and most likely all other CNN benchmarks. This is not an issue with my setup, it's the same for other folks - you can view discussion at https://www.pugetsystems.com/labs/hpc/RTX3090-TensorFlow-NAMD-and-HPCG-Performance-on-Linux-Preliminary-1902/
The text was updated successfully, but these errors were encountered: