The land/sea mask operator is doing the work but the flaw is the area outside the shapefile is replaced with NaN values occupying lot of space in the disk. The major purpose of clipping the image is to save the disk space and do processing for the required area.
images are always rectangular. So the outside of your irregular shaped area needs to be filled up with values.
If you want to save disk space you can use NetCDF4. The NaN data is well compressible.
You can also enable compression for BigTiff.
Add to the snap.properties file (in installdir/etc) the following line: