The algorithm is custom code originally developed by Helmut Schiller (deceased, unpublished work). If you are interested in solving constrained linear least squares problems, you may consult the general reference below, which might be Helmut Schillerâ€™s original source.

Charles L. Lawson and Richard J. Hanson (1995). â€śSolving Least Squares Problems.â€ť https://doi.org/10.1137/1.9781611971217.

In fact I do not quite understand why the spectral unmixing in SNAP includes particularly these constrained and fully constrained algorithms. In my opinion non-negative least squares (NNLS) is most appropriate. You may Google NNLS or simply have a look at this code

https://github.com/octoflar/amber/blob/main/src/main/fortran/nnls_mod.F90

which uses and refers to the original NNLS Fortran code.