Question

use vectors to decide whether the triangle with vertices P(1,-3,-2) Q(2,0,-4) R(6,-2,-5) is right angled

Answer 1

find vectors PQ, QR and PR. Take their scalar products pairwise. If any one scalar product is zero, it means the pair of vectors(or sides) is perpendicular.

Answer 2

What happened?

Answer 3

vector of PQ is the distance between them, right?

Answer 4

Careful, you want to take the dot product of vectors representing the sides of the triangle, not the dot product of the coordinates of the vertices.
P(1,-3,-2) Q(2,0,-4) R(6,-2,-5)
Side PQ has direction P - Q =(-1,-3,2)
Side QR has direction Q - R = (-4,2,1)
Side PR has direction P - R = (-5,-1,+3)

Now if you take PR * QR you get 4-6+2=0
So the triangle has a right angle at vertex R (the angle common to both PR and QR)