5c) If BA=I_m, then by a lemma proved in class (whose proof you should understand), A is injective and B is surjective. But neither of these statements is possible. Since A sends R^m to R^n and m>n, A cannot be injective. Since B sends R^n to R^m and m>n, B cannot be surjective. To get an example with AB=I_n, let A be an n by m matrix which looks like I_n with m-n zero columns attached on the right, and let B be an m by n matrix which looks like I_n with m-n zero rows attached on the bottom.
5d) Suppose AB=I_n. Then by the lemma mentioned above, B is injective and A is surjective. By the rank-nullity theorem, A and B both have rank n so they are both invertible. By 5(b), BA=I_n.
8) Looking at a picture, we see that T permutes the standard basis vectors, so it is either [0,1,0;0,0,1;1,0,0] or [0,0,1;1,0,0;0,1,0]. Of course, in class this week we introduced a more systematic method for attacking this sort of problem; see HW#6.