Mark values outside specific range as NaN

You can set the valid pixel expression on the band

myBand.setValidPixelExpression(myBand < maxValue && myBand > minValue)

If you want to filter just a single value you can set the no-data value

myBand.setNoDataValue(42.24)
myBand.setNoDataValueUsed(true)

Note that you have to enable the usage of the no-data value.