Question

how to check whether a system of equation posses nontrival solutions?

Answer 1

A system of homogenous equations (all set equal to 0) has a non trivial solution if the determinant of the matrix that represents that system of equations is zero.

One way to look at this is that if you can find an inverse matrix for the system, then it must have a trivial solution. But if the determinant of the matrix is zero, then no inverse matrix exists.