Question

find a general solution for the system of differential equations: dx/dt=-7x+10z; dy/dt=9x+2y-11z; dz/dt=-8x+11z

Answer 1

What have you tried? Did you read section 6.3 of Strang?

Answer 2

Rewriting your system:
\[\overrightarrow{x}'=\left[\begin{matrix}-7 & 0 &10 \\ 9 & 2 & -11\\-8 & 0 & 11\end{matrix}\right]\overrightarrow{x}\]
finding the eigenvalues of the matrix
\[\left|\begin{matrix}-7-\lambda & 0 & 10\\ 9 & 2-\lambda & -11\\ -8&0&11-\lambda\end{matrix}\right|=0\]
\[(-7-\lambda)((2-\lambda)(11-\lambda)-0)-(0)+10(0-(-8)(2-\lambda))=0\]
\[(\lambda-1)(\lambda-2)(\lambda-3)=0\rightarrow\lambda=1,2,3\]
\[v_1=\left(\begin{matrix}5 \\ -1\\ 4\end{matrix}\right), v_2=\left(\begin{matrix}0 \\ 1\\ 0\end{matrix}\right), v_3=\left(\begin{matrix}1 \\ -2 \\ 1\end{matrix}\right)\]

the general form then is
\[u(t)=c_1e^{1t}\left(\begin{matrix}5 \\ -1 \\ 4\end{matrix}\right)+c_2e^{2t}\left(\begin{matrix}0 \\ 1 \\ 0\end{matrix}\right)+c_3e^{3t}\left(\begin{matrix}1 \\ -2 \\ 1\end{matrix}\right)\]

and the constants can be solved for if there is information about the initial value of the system.

hope this helps!