While generating a geotiff of water mask, if the size is considerably bigger (around 1 GB), its unable to export the image as geotiff.
Have you tried the bigtiff option?
I am facing exactly same issue
Can you both tell a bit more? Do you see an error dialog which tells more about the reason?
Can you provide the log file?
You can access it by Help-> Show Log Directory
Reproduce the error and then attach the messages.log file here.