Some tests are not independent

I was trying to compile the development version of snap-engine and mvn clean install was failing test ImplicitBandOpImageTest in snap-core. However, the test didn’t fail when I ran that test alone, only when I ran all snap-core tests together. After some trial and error, I found that the test only failed when it ran after test VersionCheckerTest.

It turned out that VersionCheckerTest causes EngineConfig from snap-runtime to be loaded, which in turn reads (among others) the user configuration in ~/.snap/etc/snap.properties. For some unknown reason (I did not intentionally create that file), my snap.properties file contained the following lines:

snap.dataio.reader.tileHeight=512
snap.dataio.reader.tileWidth=512

These settings, in AbstractProductReader.configurePreferredTileSize(), are used to overwrite the tile size passed when setting a product’s layout like in ImplicitBandOpImageTest.ProductFactory.readProductNodeImpl(), causing a different number of tiles to be created and therefore test assertEquals(4, rasters.length) to fail in testGeophysicalImage in the same file.

I believe there are two problems here:

  • Tests should be independent: if some test sets or uses some global state, it should reset it to default just before and just after running.
  • Tests should not read user- or system-specific configuration files, but run on just their own.

In my specific case I found a workaround, but after a few hours of work. Since other similar problems can surface at any point and can be difficult to debug (because they appear on some systems and not others, or they appear with some test configurations and not others), I would suggest to check that tests are well independent.

Thanks for pointing us to this issue. We didn’t came across this problem till now. We fixed other tests which were not independent in the past, but this one worked for us so far. I updated the VersionCheckerTest now it gets its own preferences instance. So there should not be any interefence anymore.

1 Like