The algorithm is custom code originally developed by Helmut Schiller (deceased, unpublished work). If you are interested in solving constrained linear least squares problems, you may consult the general reference below, which might be Helmut Schiller’s original source.
Charles L. Lawson and Richard J. Hanson (1995). “Solving Least Squares Problems.” https://doi.org/10.1137/1.9781611971217.
In fact I do not quite understand why the spectral unmixing in SNAP includes particularly these constrained and fully constrained algorithms. In my opinion non-negative least squares (NNLS) is most appropriate. You may Google NNLS or simply have a look at this code
which uses and refers to the original NNLS Fortran code.