Exporting RGB image as a Geotiff

How much RAM do you have in your computer? It should be at least 8GB.
Maybe this can help you?

When you use the suggested graph you should adapt the gpt.vmoptions to increase the memory which can be used.
Add
-Xmx9G
You can find the file in the bin folder of the installation directory.