The method aimed to fit line's general format Ax+By+C=0 with least-square algorithm.
In most of cases, people are used to fit y=kx+b that slope format. However, sometimes we need to fit line's general format Ax+By+C=0 with more than 2 points.
The Loss is as follows, the distance sqr sum from these points to this line is expected to minimum
Now take the partial derivatives of F with respect to A, B, and C, and set them equal to zero.
(1)
(2)
(3)
some coefficients is defined:
n = SUM 1, s = SUM x[i], t = SUM y[i], u = SUM x[i]^2, v = SUM x[i]*y[i], w = SUM y[i]^2,
unfold (1)(2)(3), we get -vA^2B-sA^2C+(u-w)AB^2-2tABC-nAC^2+vB^3+sB^2*C = 0, (4)
vA^3+(w-u)A^2B+tA^2C-vAB^2-2sABC-tB^2C-nB*C^2 = 0, (5)
sA+tB+n*C = 0. (6)
from (6),
we get 
(7)
replace (7) into (4) or (5), we get
(nv-st)A^2 + (s^2-t^2-nu+nw)AB - (nv-s*t)*B^2 = 0. (8)
futher simplify, A^2 + [(s^2-t^2-nu+nw)/(nv-st)]AB - B^2 = 0. (9)
known, s^2-t^2 = u-w s*t=v
so (s^2-t^2-nu+nw)/(nv-st) = w-u/v
(9) is converted to A^2 + [(w-u)/v]AB - B^2 = 0. (10)
How to solve A, B In some pro, A = w-u, B=v
because, it only compute A/B, instead A&B.