SNAP: Clip Raster by Shapefile

The Land/Sea Mask operator should actually do what you intend.

The error message seems a bit strange. Have you set all parameters?