no, you haven’t done anything wrong. In this case it just means that these pixels cannot be predicted by the random forest classifier in its current configuration.
You can try to increase the number of trees, maybe add more training areas or train more heterogeneous classes (e.g. forest1, forest2, forest3, no-forest1, no-forest2) so that every pixel can somehow be related to any of the classes. .