Orthonormal Bases and Orthogonal Projections

In this article, I begin with the fact that orthonormal vectors are linearly independent and thus form a basis for the subspaces generated by them. After that, I explore the orthogonal projection and properties of the orthogonal complement. Towards the end, I detail the Pythagorean Theorem, the Cauchy-Schwarz Theorem, the Law of Cosines, and the Triangular Inequality Theorem.

Orthonormal Bases and Orthogonal Projections

The norm of a vector $v$ in $\mathbb{R}^n$ is $$ \left|\left| v \right|\right| = \sqrt{v \cdot v}. $$ A vector $u$ in $\mathbb{R}^n$ is called a unit vector if $\left|\left|u\right|\right|=1.$

Example. If $v\in \mathbb{R}^n$ and $k$ is a scalar, then $\left|\left| k v\right|\right| =|k| \left|\left| v\right|\right| $, and if $v$ is nonzero then $u=\frac{1}{\left|\left| v\right|\right| } v$ is a unit vector. Since $\left|\left| k v\right|\right| ^2=(k v)\cdot (k v)=k^2(v \cdot v)=k^2\left|\left| v\right|\right| ^2$, taking square roots provides $\left|\left| k v\right|\right| =|k| \left|\left| v\right|\right| .$ If $v$ is nonzero, then $\frac{1}{\left|\left| v\right|\right| }$ is defined and so $\left|\left| u\right|\right| = \frac{1}{\left|\left|v\right|\right| } \left|\left| v\right|\right| =1$ which follows by the first part.

Two vectors $v$ and $w$ in $\mathbb{R}^n$ are called perpendicular or orthogonal if $v \cdot w=0.$ The vectors $u_1,\ldots,u_m$ in $\mathbb{R}^n$ are called orthonormal if they are all unit vectors and orthogonal to one another. A basis of $\mathbb{R}^n$ consisting only of orthonormal vectors is called an orthonormal basis.

Theorem. (Orthonormal Vectors) Orthonormal vectors are linearly independent, and thus orthonormal vectors $u_1,\ldots,u_n\in \mathbb{R}^n$ form a basis for $\mathbb{R}^n.$

Proof. Suppose $(u_1,\ldots,u_m)$ are orthonormal vectors in $\mathbb{R}^n.$ To show linear independence suppose, $ c_1 u_1+\cdots + c_m u_m=0 $ for some scalars $c_1,\ldots,c_m$ in $\mathbb{R}.$ Applying the dot product with $u_i$, $ \left ( c_1 u_1 + \cdots +c_m u_m\right ) \cdot u_i =0 \cdot u_i=0. $ Because the dot product is distributive, $c_1(u_1 \cdot u_i )+\cdots + c_m (u_m \cdot u_i)=0.$ We know that $u_i\cdot u_i=1$ and all other dot products are zero. Therefore, $c_i=0.$ Since this holds for all $i=1,\ldots,m$, it follows that $c_1=\cdots =c_m=0$, and therefore, $(u_1,\ldots,u_m)$ are linearly independent. The second part follows since $n$ linearly independent vectors in $\mathbb{R}^n$ always forms a basis.

Example. Find three examples of an orthonormal basis for a subspace they span. The vectors $e_1,\ldots, e_m$ in $\mathbb{R}^n$ form an orthonormal basis of the subspace they span. For any scalar $\theta$, the vectors $\begin{bmatrix}\cos \theta \\ \sin \theta\end{bmatrix}$, $\begin{bmatrix}-\sin\theta \\ \cos \theta\end{bmatrix}$ form an orthonormal basis of $\mathbb{R}^2.$ The vectors $$ \begin{array}{ccc} u_1=\begin{bmatrix} 1/2 \\ 1/2 \\ 1/2 \\ 1/2\end{bmatrix}, & u_2= \begin{bmatrix} 1/2 \\ 1/2 \\ -1/2 \\ -1/2\end{bmatrix}, & u_3=\begin{bmatrix} 1/2\\ -1/2\\ 1/2 \\ -1/2\end{bmatrix} \end{array} $$ in $\mathbb{R}^4$ form an orthonormal basis of the subspace they span.

Let $V$ be a subspace of $\mathbb{R}^n.$ The orthogonal complement $V^\perp$ of $V$ is the set of those vectors $x$ in $\mathbb{R}^n$ that are orthogonal to all vectors in $V$ namely,

\begin{equation*} V^{\perp} = \{ x \in \mathbb{R}^n \mid v \cdot x =0, \text{ for all } v \text{ in } V\}. \end{equation*}

It is easy to verify that $V^\perp$ is always a subspace and that $(\mathbb{R}^n)^\perp={0}$ and ${0}^\perp=\mathbb{R}^n.$ Also notice that if $U_1\subseteq U_2$ then $U_2^\perp \subseteq U_1^\perp.$ If $x\in V^{\perp}$ then $x$ is said to be perpendicular to $V.$ The vector $x^{\parallel}$ in the following theorem is called the orthogonal projection of $x$ on a subspace $V$ of $\mathbb{R}^n$ and is denoted by $\mathop{proj}_V (x).$

Theorem. (Orthogonal Projection) (a) If $V$ is a subspace of $\mathbb{R}^n$ and $x\in \mathbb{R}^n$, then $x=x^\parallel + x^\perp$ where $x^\perp$ is perpendicular to $V$, and this representation is unique. (b) If $V$ is a subspace of $\mathbb{R}^n$ with an orthonormal basis $u_1 ,\ldots, u_m$, then \begin{equation} \mathrm{proj}_V (x):=x^\parallel = (u_1 \cdot x) u_1 + \cdots + (u_m \cdot x) u_m \end{equation} for all $x$ in $\mathbb{R}^n.$ (c) Let $u_1 , \ldots, u_n$ be an orthonormal basis in $\mathbb{R}^n$, then \begin{equation} x = (u_1 \cdot x) u_1 + \cdots + (u_n \cdot x) u_n \end{equation} for all $x$ in $\mathbb{R}^n.$

Example. Find the orthogonal projection of $\begin{bmatrix}49 \\ 49 \\ 49\end{bmatrix}$ onto the subspace of $\mathbb{R}^3$ spanned by $\begin{bmatrix}2 \\ 3 \\ 6\end{bmatrix}$ and $\begin{bmatrix} 3\\ -6\\ 2\end{bmatrix}.$ The two given vectors spanning the subspace are orthogonal since $2(3)+3(-6)+6(2)=0$, but they are not unit vectors since both have length 7. To obtain an orthonormal basis $u_1, u_2$ of the subspace, we divide by 7: $u_1=\frac{1}{7}\begin{bmatrix} 2 \\ 3\\ 6\end{bmatrix}$ and $u_2=\frac{1}{7}\begin{bmatrix}3\\ -6\\ 2\end{bmatrix}.$ Now we can use the Orthogonal Projection with $x=\begin{bmatrix}49\\ 49\\ 49\end{bmatrix}.$ Then $$ \mathop{proj}_V(x)=(u_1 \cdot x)u_1+(u_2\cdot x) u_2= 11 \begin{bmatrix}2 \\ 3 \\ 6\end{bmatrix}+(-1)\begin{bmatrix}3 \\ -6 \\ 2\end{bmatrix}= \begin{bmatrix}19 \\ 39 \\ 64\end{bmatrix} $$ as needed.

Example. Find the coordinates of the vector $x=\begin{bmatrix}4\ 5\ 6\ 7\end{bmatrix}$ with respect to the orthonormal basis $$ \begin{array}{cccc} u_1=\begin{bmatrix}1/2\\ 1/2\\ 1/2 \\1/2\end{bmatrix}, & u_2=\begin{bmatrix}1/2\\ 1/2\\ -1/2\\ -1/2\end{bmatrix}, & u_3=\begin{bmatrix}1/2\\ -1/2\\ 1/2\\ -1/2\end{bmatrix}, & u_4=\begin{bmatrix}1/2 \\ -1/2 \\ -1/2\\ 1/2\end{bmatrix}. \end{array} $$ Normally to find the coordinates of $x$ we would solve the system $$ \begin{bmatrix}4 \\ 5\\ 6 \\ 7\end{bmatrix}= c_1 \begin{bmatrix}1/2 \\ 1/2 \\ 1/2 \\ 1/2\end{bmatrix}+ c_2 \begin{bmatrix}1/2 \\ 1/2 \\ -1/2 \\ -1/2\end{bmatrix}+ c_3 \begin{bmatrix}1/2 \\\ -1/2 \ 1/2 \\ -1/2\end{bmatrix} + c_4 \begin{bmatrix}1/2 \\\ -1/2 \ -1/2 \\ 1/2\end{bmatrix} $$ for $c_1, c_2, c_3, c_4.$ However we can use Orthogonal Projection instead: $$ c_1=u_1\cdot x=\begin{bmatrix}1/2 \\ 1/2 \\ 1/2 \\ 1/2\end{bmatrix} \cdot \begin{bmatrix}4 \\ 5 \\ 6 \\ 7\end{bmatrix}=11, \hspace{1cm} c_2=u_2\cdot x=\begin{bmatrix}1/2 \\ 1/2 \\ -1/2 \\ -1/2\end{bmatrix} \cdot \begin{bmatrix}4 \\ 5 \ 6 \\ 7\end{bmatrix}=-2, $$ $$ c_3=u_3\cdot x=\begin{bmatrix}1/2 \\ -1/2 \\ 1/2 \\ -1/2\end{bmatrix} \cdot \begin{bmatrix}4 \\ 5 \\ 6 \\ 7\end{bmatrix}=-1, \hspace{1cm} c_4=u_4\cdot x=\begin{bmatrix}1/2 \\ -1/2 \\ -1/2 \\ 1/2\end{bmatrix} \cdot \begin{bmatrix}4 \\ 5 \\ 6 \\ 7\end{bmatrix}=0. $$ Therefore the $\mathcal{B}$-coordinate vector of $x$ is $\begin{bmatrix}11 \\ -2 \\ -1 \ 0\end{bmatrix}.$

Theorem. (Properties of the Orthogonal Complement) Let $V$ be a subspace of $\mathbb{R}^n$, then

(1) $\mathrm{proj}_V(x)$ is a linear transformation $\mathbb{R}^n\to V$ with kernel $V^\perp$,

(2) $V \cap V^\perp = {0}$,

(3) $\mathop{dim} V +\mathop{dim} V^\perp=n$, and

(4) $(V^\perp)^\perp = V.$

Proof. The proof of each part follows.

  • To prove the linearity of $T(x):=\mathrm{proj}_V(x)$ we will use the definition of a projection: $T(x)$ is in $V$, and $x-T(x)$ is in $V^\perp.$ To show $T(x+y)=T(x)+T(y)$, notice $T(x)+T(y)$ is in $V$ (since $V$ is a subspace), and $x+y-(T(x)+T(y))=(x-T(x))+(y-T(y))$ is in $V^\perp$ (since $V^\perp$ is a subspace). To show that $T(k x)=k T(x)$, note that $k T(x)$ is in $V$ (since $V$ is a subspace) and $k x-k T(x)=k(x-T(x))$ is in $V^\perp$ (since $V^\perp$ is a subspace).
  • Since ${0} \subseteq V$ and ${0} \subseteq V^\perp$, ${0} \subseteq V\cap V^\perp.$ If a vector $x$ is in $V$ as well as in $V^\perp$, then $x$ is orthogonal to itself: $x \cdot x =\left|\left |x\right|\right |^2=0$, so that $x$ must equal $0$ which shows $V \subseteq {0}.$ Therefore, $V\cap V^\perp={0}.$
  • Apply the Rank-Nullity Theorem to the linear transformation $T(x)=\mathop{proj}_V(x)$ yielding $$n=\mathop{dim} \mathbb{R}^n =\mathop{dim} \mathop{im} T+\mathop{dim} \ker T=\mathop{dim} V+\mathop{dim} V^\perp.$$
  • Let $v\in V.$ Then $v\cdot x=0$ for all $x$ in $V^\perp.$ Since $(V^\perp)^\perp$ contains all vectors $y$ such that $y \cdot x =0$, $v$ is in $(V^\perp)^\perp.$ So $V$ is a subspace of $(V^\perp)^\perp.$ Using (iii) with $V$ $(n=\mathop{dim} V+\mathop{dim} V^\perp)$ and again with $V^\perp$ namely $(\mathop{dim} V^\perp+\mathop{dim} (V^\perp)^\perp)$ yielding $\mathop{dim} V=\mathop{dim} (V^\perp)^\perp$; and since $V$ is a subspace of $(V^\perp)^\perp$ it follows that $V=(V^\perp)^\perp.$

Example. Find a basis for $W^\perp$, where $W=\mathop{span} \left( \begin{bmatrix}1 \\ 2 \\ 3 \\ 4\end{bmatrix}, \begin{bmatrix}5 \\ 6 \\ 7 \\ 8\end{bmatrix} \right).$ The orthogonal complement $W^\perp$ of $W$ consists of the vectors $x$ in $\mathbb{R}^4$ such that$$
\begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ x_4\end{bmatrix} \cdot \begin{bmatrix}1 \\ 2 \\ 3 \\ 4 \end{bmatrix}=0 \hspace{1cm}\text{and} \hspace{1cm} \begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ x_4\end{bmatrix} \cdot \begin{bmatrix}5 \\ 6 \\ 7 \\ 8\end{bmatrix}=0. $$ Finding these vectors amounts to solving the system $$ \begin{cases} x_1+2x_2+3x_3+4x_4&=0 \\ 5x_1+6x_2+7x_3+8x_4 & =0 \end{cases} $$ The solutions are of the form $$ \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4\end{bmatrix} =\begin{bmatrix}s+2t \\ -2s-3t \\ s \\ t\end{bmatrix} =s\begin{bmatrix}1 \\ -2 \\ 1 \\ 0\end{bmatrix} + t\begin{bmatrix}2 \\ -3 \\ 0 \\ 1\end{bmatrix}. $$ The two vectors on the right form a basis of $W^\perp.$

Theorem. Let $x, y \in \mathbb{R}^n.$ Then

(Pythagorean Theorem) \begin{equation} \label{pythagorean-theorem} ||x + y || ^2 = || x|| ^2 + || y || ^2 \end{equation} holds if and only if $x \perp y$

(Cauchy-Schwarz) \begin{equation} \label{cauchy-schwarz} | x \cdot y | \leq || x || \, || y || \end{equation} where equality holds if and only if $x $ and $y$ are parallel,

(Law of Cosines) the angle $\theta$ between $x$ and $y$ is defined as \begin{equation} \theta = \arccos \frac{x \cdot y}{|| x || \, || y||}, \end{equation}

(Triangular Inequality) \begin{equation} \label{triangular-inequality} \left|\left| x+y\right|\right| \leq \left|\left| x\right|\right| +\left|\left| y\right|\right| . \end{equation}

Proof. The proof of each part follows.

  • The verification is straightforward: \begin{align*} \left|\left| x+y \right|\right| ^2 & =(x + y)\cdot (x + y) =x \cdot x+2(x \cdot y) +y \cdot y \\ & =\left|\left| x\right|\right| ^2+2(x\cdot y)+\left|\left| y\right|\right| ^2 =\left|\left| x\right|\right| ^2+\left|\left| y\right|\right| ^2 \end{align*} where the last equality holds if and only if $x\cdot y=0.$
  • Let $V$ be a one-dimensional subspace of $\mathbb{R}^n$ spanned by a nonzero vector $y.$ Let $u=\frac{1}{\left|\left| y\right|\right| } y.$ Then $$ \left|\left| x\right|\right| \geq \left|\left| \mathop{proj}_V(x)\right|\right| =\left|\left| (x\cdot u)u\right|\right| =|x\cdot u| =\left| x \cdot \left( \frac{1}{\left|\left| y\right|\right| }y \right)\right| = \frac{1}{\left|\left| y\right|\right| }|x \cdot y| $$ multiplying by $\left|\left| y\right|\right| $, yields $| x \cdot y | \leq || x || \, || y ||.$ Notice that $\left|\left| x\right|\right| \geq \left|\left| \mathop{proj}_V(x)\right|\right| $ holds by applying the Pythagorean theorem to $x=x^\parallel +x^\perp$ with $x^\perp \cdot x^\parallel =0$ so that $\left|\left| x\right|\right| ^2=\left|\left| \mathop{proj}_V(x)\right|\right| ^2+\left|\left| x^\perp\right|\right| ^2$ which leads to $\left|\left| \mathop{proj}_V x\right|\right| \leq \left|\left| x\right|\right| .$
  • We have to make sure that $\theta$ is defined, that is $\theta$ is between $-1$ and 1, or equivalently,$$ \left | \frac{x \cdot y }{\left|\left |x\right|\right | \, \left|\left| y\right|\right| } \right | \leq 1. $$ But this follows from the Cauchy-Schwarz inequality.
  • Using the Cauchy-Schwarz inequality, the verification is straightforward, $$ \left|\left| x+y\right|\right| ^2=(x + y)\cdot (x + y)=\left|\left| x\right|\right| ^2+\left|\left| y\right|\right| ^2+2(x\cdot y) $$ $$ \leq \left|\left| x\right|\right| ^2+\left|\left| y\right|\right| ^2+2\left|\left| x\right|\right| \left|\left| y\right|\right| =(\left|\left| x\right|\right| +\left|\left| y\right|\right| )^2 $$ Taking the square root of both sides yields $\left|\left| x+y\right|\right| \leq \left|\left| x\right|\right| +\left|\left| y\right|\right| .$

Example. Determine whether the angle between the vectors $u=\begin{bmatrix}2 \ 3 \ 4\end{bmatrix}$, $v=\begin{bmatrix}2 \ -8 \ 5\end{bmatrix}$ is a right angle using the Pythagorean Theorem. Since $\left|\left| u\right|\right| =\sqrt{2^2+3^2+4^2}=\sqrt{29}$ and $\left|\left| v\right|\right| =\sqrt{2^2+(-8)^2+5^2}=\sqrt{93}.$ Then $$ \left|\left| u+v\right|\right| ^2=\left|\left| \, \, \begin{bmatrix}4 \ -5 \ 9\end{bmatrix} \, \, \right| \right|^2 =122 = 29+93= || u|| ^2 + || v || ^2 $$ shows $u \perp v.$

Example. Consider the vectors $u =\begin{bmatrix}1 \\ 1 \\ \vdots \\ 1\end{bmatrix}$ and $v=\begin{bmatrix}1 \\ 0 \\ \vdots \\ 0\end{bmatrix}$ in $\mathbb{R}^n.$ For $n=2,3,4$, find the angle $\theta$ between $u$ and $v.$ Then find the limit of $\theta$ as $n$ approaches infinity. For any possible value of $n$, $ \theta_n=\arccos \frac{u \cdot v}{\left|\left| u\right|\right| \left|\left| v\right|\right| }=\arccos \frac{1}{\sqrt{n}}. $ Then $$ \begin{bmatrix} \theta_2=\arccos \frac{1}{\sqrt{2}}=\frac{\pi}{4}, & \qquad \theta_3=\arccos \frac{1}{\sqrt{3}}=\frac{\pi}{4}\sim 0.955 \text{ rads}, & \qquad \theta_4=\arccos \frac{1}{\sqrt{4}}=\frac{\pi}{3}. \end{bmatrix} $$ Since $y=\arccos(x)$ is a continuous function, $$ \lim_{x\mapsto \infty} \theta_n = \lim_{x\mapsto \infty} \arccos\left( \frac{1}{\sqrt{n}} \right) = \arccos \left( \lim_{x\mapsto \infty} \frac{1}{\sqrt{n}} \right) = \arccos(0) =\frac{\pi}{2}. $$

David A. Smith at Dave4Math

David Smith (Dave) has a B.S. and M.S. in Mathematics and has enjoyed teaching precalculus, calculus, linear algebra, and number theory at both the junior college and university levels for over 20 years. David is the founder and CEO of Dave4Math.

Leave a Comment