Home page for accesible maths 3 Inner products 3.D Orthogonality Exercises

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

3.E The Gram-Schmidt process

Finding coordinates with respect to a basis $\mathcal{B}$ which is orthogonal is quite easy; and if it’s orthonormal, than it’s easier still. The following theorem justifies this statement.

Theorem 3.33.

Let $V$ be an inner product space, basis $\mathcal{B}=(\vec{x_{1}},\cdots,\vec{x_{n}})$ , and $\vec{v}\in V$ .

i.

If $\mathcal{B}$ is orthogonal: $\vec{v}=\sum_{i=1}^{n}\frac{\langle\vec{v},\vec{x_{i}}\rangle}{||\vec{x_{i}}||% ^{2}}\vec{x_{i}}$ ,
ii.

if $\mathcal{B}$ is orthonormal: $\vec{v}=\sum_{i=1}^{n}\langle\vec{v},\vec{x_{i}}\rangle\vec{x_{i}}$ .

In other words, the coordinates of $\vec{v}$ with respect to $\mathcal{B}$ are $\frac{\langle\vec{v},\vec{x_{1}}\rangle}{||\vec{x_{1}}||^{2}},\cdots,\frac{% \langle\vec{v},\vec{x_{n}}\rangle}{||\vec{x_{n}}||^{2}}$ .

Proof.

Since $\mathcal{B}$ is a basis, we can find scalars $\alpha_{k}\in{\mathbb{R}}$ such that $\vec{v}=\sum_{k=1}^{n}\alpha_{k}\vec{x_{k}}$ . Take the inner product of both sides with $\vec{x_{i}}$ . If the basis is orthogonal, then $\langle\vec{x_{k}},\vec{x_{i}}\rangle=0$ for any $i\neq k$ ; so using bilinearity of the inner product:

\langle\vec{v},\vec{x_{i}}\rangle=\langle\sum_{k=1}^{n}\alpha_{k}\vec{x_{k}},% \vec{x_{i}}\rangle=\sum_{k=1}^{n}\alpha_{k}\langle\vec{x_{k}},\vec{x_{i}}% \rangle=\alpha_{i}\langle\vec{x_{i}},\vec{x_{i}}\rangle.

Solving for $\alpha_{i}$ , and the result follows. ∎

Exercise 3.34:

Let’s illustrate Theorem 3.33 for $V={\mathbb{R}}^{2}$ . Consider the basis $\mathcal{B}=(\vec{x_{1}},\vec{x_{2}})$ where $\vec{x_{1}}=(1,1)$ and $\vec{x_{2}}=(1,-1)$ . This basis is orthogonal since $\langle\vec{x_{1}},\vec{x_{2}}\rangle=0$ . Now choose your own vector in ${\mathbb{R}}^{2}$ , and call it $\vec{v}$ . For your vector, compute the expression $\sum_{i=1}^{n}\frac{\langle\vec{v},\vec{x_{i}}\rangle}{||\vec{x_{i}}||^{2}}% \vec{x_{i}}$ . According to Theorem 3.33 the result should be equal to $\vec{v}$ !

[End of Exercise]

If we are given a basis $\mathcal{B}=(\vec{x_{1}},\cdots,\vec{x_{n}})$ of an inner product space $V$ , then we may wish to construct a new orthogonal basis $\mathcal{C}=(\vec{b_{1}},\cdots,\vec{b_{n}})$ from it. We do this by the Gram-Schmidt process, as follows:

$\vec{b_{1}}:=\vec{x_{1}}$ ,
Then, inductively define: $\vec{b_{k}}:=\vec{x_{k}}-\sum_{i=1}^{k-1}\frac{\langle\vec{x_{k}},\vec{b_{i}}% \rangle}{||\vec{b_{i}}||^{2}}\vec{b_{i}}$ , for each $k=2,\cdots,n$ .

The above formula is commonly called the Gram-Schmidt formula.

Exercise 3.35:

For each of the following sequences of vectors $\vec{x_{1}},\vec{x_{2}}$ , apply the Gram-Schmidt process, and compute $\vec{b_{1}},\vec{b_{2}}$ . In each case, draw the four resulting vectors on the same axis.

i.

$\vec{x_{1}}=(1,0)$ and $\vec{x_{2}}=(2,2)$ .
ii.

$\vec{x_{1}}=(2,2)$ and $\vec{x_{2}}=(1,0)$ .

[End of Exercise]

This construction has the following properties:

Theorem 3.36.

Let $\mathcal{B}=(\vec{x_{1}},\cdots,\vec{x_{n}})$ be a basis of an inner product space, and $\mathcal{C}=(\vec{b_{1}},\cdots,\vec{b_{n}})$ the sequence of vectors obtained by the Gram-Schmidt process (defined above). Then for each $k=1,\cdots,n$ the following are true.

i.

$\vec{b_{k}}\neq\vec{0}$ ,
ii.

$(\vec{b_{1}},\cdots,\vec{b_{k}})$ is an orthogonal sequence of vectors,
iii.

$\operatorname{span}\{\vec{b_{1}},\cdots,\vec{b_{k}}\}=\operatorname{span}\{% \vec{x_{1}},\cdots,\vec{x_{k}}\}$ .

Proof.

The proof is by induction on $k$ . When $k=1$ , then $\vec{b_{1}}=\vec{x_{1}}\neq 0$ , and the other statements are obvious. Let $r>1$ , then our inductive assumption is that all three statements are true for values of $k$ strictly less than $r$ ; i.e. for $k<r$ . With that assumption, we want to prove all three statements for $k=r$ .

If $\vec{b_{r}}=\vec{0}$ , then $\vec{x_{r}}\in\operatorname{span}\{\vec{b_{1}},\cdots,\vec{b_{r-1}}\}=% \operatorname{span}\{\vec{x_{1}},\cdots,\vec{x_{r-1}}\}$ , by the Gram-Schmidt formula together with the assumption (iii) for $k=r-1$ . This contradicts the assumption that $\mathcal{B}$ is linearly independent. So (i) is true for $k=r$ .

Since we have assumed (ii) for $k=r-1$ , to prove it for $k=r$ we just need to check that $\langle\vec{b_{r}},\vec{b_{j}}\rangle=0$ for any $j=1,\cdots,r-1$ , which is Exercise 3.38.

Finally, since we have assumed (iii) for $k=r-1$ , we see by the Gram-Schmidt formula that $\vec{b_{r}}$ is a linear combination of elements in $(\vec{x_{1}},\cdots,\vec{x_{r}})$ , and thus $\operatorname{span}\{\vec{b_{1}},\cdots,\vec{b_{r}}\}\subset\operatorname{span% }\{\vec{x_{1}},\cdots,\vec{x_{r}}\}$ . Equality follows because they are both subspaces of the same dimension (by (i), (ii), and Exercise 3.44). So, by induction, the result it true for all $k$ . ∎

Exercise 3.37:

Choose your own basis $\vec{x_{1}},\vec{x_{2}},\vec{x_{3}}$ of ${\mathbb{R}}^{3}$ which is not orthogonal. Apply the Gram-Schmidt process to it to obtain a new basis $\vec{b_{1}},\vec{b_{2}},\vec{b_{3}}$ . Verify that your new basis is orthogonal. Is it orthonormal?

[End of Exercise]

Exercise 3.38:

In the proof of Theorem 3.36, show that $\langle\vec{b_{r}},\vec{b_{j}}\rangle=0$ .

[End of Exercise]

Corollary 3.39.

Let $W\subset{\mathbb{R}}^{n}$ be a subspace. There is an orthonormal basis of $W$ . Furthermore, that basis can be extended to an orthonormal basis of ${\mathbb{R}}^{n}$ .

Proof.

We omit this proof from the module. Here is a sketch proof: Choose a basis of $W$ (by Theorem 2.36), apply the Gram-Schmidt process to obtain an orthogonal basis of $W$ , then scale to make it orthonormal.

Next, extend to a basis to ${\mathbb{R}}^{n}$ (Corollary 2.37), apply the Gram-Schmidt process (the first $r$ vectors are unchanged), and scale to get an orthonormal basis of ${\mathbb{R}}^{n}$ . ∎