HI madjidh,
actually it should work with the ‘||’ operator.
I just tried on a sentinel-3 product with this expresson:
quality_flags_land || quality_flags_coastline ? NaN : Oa08_radiance
As a result I got only water pixels.
Can you tell what is not working in your case when you use the or operator. Have you tried the expression first in SNAP Desktop?