Appendix B

Linear Algebra Review

You have seen linear algebra before, but perhaps it has been a while since you last found a basis or computed a kernel. This appendix is a chance to recover those ideas and put them to work in a setting where the vectors may be functions. We will move back and forth between the familiar arithmetic of lists of numbers and the arithmetic of polynomials and other functions. The point is to recognize the same mathematics in both places.

Work through the calculations with a pencil, and pause at the short review questions before reading their answers at the end. We use real numbers as scalars until the final section, where complex eigenvalues will give us a reason to allow complex coordinates too.

B.1Vectors and Vector Spaces

What can we do with vectors? In $ℝ 2$ , we can add them and multiply them by numbers. If $𝑢 = (1, 2)$ and $𝑣 = (3, - 1)$ , then

2 𝑢 - 𝑣 = 2 (1, 2) - (3, - 1) = (- 1, 5) .

Each coordinate follows ordinary arithmetic. Nothing about this calculation depends on drawing arrows, and that is useful: we would like the same arithmetic to work for objects we cannot conveniently draw as arrows.

For instance, take the functions $𝑝 (𝑡) = 1 + 2 𝑡$ and $𝑞 (𝑡) = 3 - 𝑡$ . We can make exactly the same calculation:

(2 𝑝 - 𝑞) (𝑡) = 2 (1 + 2 𝑡) - (3 - 𝑡) = - 1 + 5 𝑡 .

The constant coefficients combine just as the first coordinates did, and the coefficients of $𝑡$ combine just as the second coordinates did. The functions are different objects from their coefficient pairs, but their arithmetic has the same structure.

More generally, if $𝑓$ and $𝑔$ are real-valued functions on the same interval $𝐼$ , define

(𝑓 + 𝑔) (𝑡) = 𝑓 (𝑡) + 𝑔 (𝑡), (𝑐 𝑓) (𝑡) = 𝑐 𝑓 (𝑡) f o r e v e r y 𝑡 \in 𝐼 .

These operations are called pointwise addition and scalar multiplication: we perform the numerical operation separately at each input. The result is another whole function. In particular, the zero vector in this setting is the function which is zero at every input, not merely a function whose graph passes through the origin.

Matrices admit the same kind of arithmetic. Two real $2 \times 2$ matrices can be added entry by entry, and each entry can be multiplied by a scalar. They therefore give us another collection of objects with vector arithmetic. Matrix multiplication is an additional operation; it is not needed to make matrices into vectors.

What must this arithmetic satisfy? We need sums and scalar multiples to stay inside the collection, and we need the familiar algebraic rules to continue working.

Definition B.1 (Real vector space). A real vector space is a set $𝑉$ with an addition operation and an operation of multiplication by real scalars. For $𝑢, 𝑣, 𝑤 \in 𝑉$ and $𝑎, 𝑏 \in ℝ$ , the results $𝑢 + 𝑣$ and $𝑎 𝑢$ belong to $𝑉$ , and the following rules hold:

𝑢 + 𝑣 = 𝑣 + 𝑢, (𝑢 + 𝑣) + 𝑤 = 𝑢 + (𝑣 + 𝑤), 𝑎 (𝑢 + 𝑣) = 𝑎 𝑢 + 𝑎 𝑣, (𝑎 + 𝑏) 𝑢 = 𝑎 𝑢 + 𝑏 𝑢, 𝑎 (𝑏 𝑢) = (𝑎 𝑏) 𝑢, 1 𝑢 = 𝑢 .

There is a zero vector $0 \in 𝑉$ with $𝑢 + 0 = 𝑢$ , and every vector $𝑢$ has an additive inverse $- 𝑢$ with $𝑢 + (- 𝑢) = 0$ .

For functions with pointwise operations, these rules come from the rules for real numbers. For example, at every $𝑡$ ,

(𝑎 (𝑓 + 𝑔)) (𝑡) = 𝑎 (𝑓 (𝑡) + 𝑔 (𝑡)) = 𝑎 𝑓 (𝑡) + 𝑎 𝑔 (𝑡) = (𝑎 𝑓 + 𝑎 𝑔) (𝑡) .

Since the two functions agree at every input, they are the same function. This proves the distributive rule for functions.

We will often restrict the functions we allow. Write $𝑃 𝑛$ for the space of real polynomials of degree at most $𝑛$ , including the zero polynomial. The phrase "at most" matters: two quadratic polynomials can add to a constant or to zero, so polynomials of degree exactly two would not form a vector space. We also use $𝐶 (𝐼)$ for continuous real-valued functions on $𝐼$ , and $𝐶 𝑘 (𝐼)$ for functions with continuous derivatives through order $𝑘$ . Adding or scaling such functions preserves the required continuity and differentiability, so these are vector spaces too.

These function spaces can be much larger than $ℝ 𝑛$ . No finite list of polynomials can span all polynomials: if the largest degree in the list is $𝑚$ , every linear combination still has degree at most $𝑚$ , and misses $𝑡 𝑚 + 1$ . We call a vector space infinite-dimensional when it has no finite basis. We will review basis and dimension shortly; for now, the point is that no fixed finite list of coefficients describes every function in a general function space. Our calculations will take place in smaller finite-dimensional spaces inside it.

Try this 1. For $𝑓 (𝑡) = 1 - 𝑡$ and $𝑔 (𝑡) = 𝑡 + 𝑡 2$ , compute $3 𝑓 - 2 𝑔$ . What is the additive inverse of $𝑓$ ? Is $𝑓$ the zero function because $𝑓 (1) = 0$ ?

Try this 2. Explain why real $2 \times 3$ matrices form a vector space under entrywise operations. Does their rectangular shape cause a problem for any vector-space rule?

B.2Subspaces and Nonlinear Subsets

Once we have a vector space, we can ask whether a smaller collection inside it still supports the same arithmetic. Consider the line

𝑈 = {(𝑥, 𝑦) \in ℝ 2 : 𝑦 = 2 𝑥} .

Its vectors have the form $(𝑎, 2 𝑎)$ . Adding two gives $(𝑎, 2 𝑎) + (𝑏, 2 𝑏) = (𝑎 + 𝑏, 2 (𝑎 + 𝑏))$ , which stays on the line. Scaling gives $𝑐 (𝑎, 2 𝑎) = (𝑐 𝑎, 2 𝑐 𝑎)$ , which stays there too. The zero vector lies on the line, so we have a vector space inside $ℝ 2$ .

Definition B.2 (Subspace). A subspace of a vector space $𝑉$ is a subset $𝑈 \subseteq 𝑉$ which is itself a vector space using the addition and scalar multiplication from $𝑉$ .

We do not need to check every vector-space rule again. Commutativity and distributivity already hold in $𝑉$ , so they hold for members of $𝑈$ . What could fail is that a required result leaves the smaller set.

Proposition B.3 (Subspace test). A subset $𝑈$ of a real vector space $𝑉$ is a subspace if and only if it contains $0$ and, for every $𝑢, 𝑣 \in 𝑈$ and $𝑎, 𝑏 \in ℝ$ , it contains $𝑎 𝑢 + 𝑏 𝑣$ .

The condition includes sums, scalar multiples, and additive inverses. It is often convenient to check addition and scalar multiplication separately. To show that a set fails the test, however, just one counterexample is enough.

Move our line upward by one unit:

𝐻 = {(𝑥, 𝑦) \in ℝ 2 : 𝑦 = 2 𝑥 + 1} .

This set does not contain $(0, 0)$ , so it cannot be a subspace. Adding its point $(0, 1)$ to itself gives $(0, 2)$ , which is not on the line. Yet the line still has a simple relationship to $𝑈$ : every member is $(0, 1) + 𝑢$ for some $𝑢 \in 𝑈$ . It is a translated subspace, an affine space. We will return to that structure in section B.6.

Now consider a different subset:

𝑄 = {(𝑥, 𝑦) \in ℝ 2 : 𝑦 = 𝑥 2} .

This parabola does contain zero. Does that make it a subspace? The point $(1, 1)$ belongs to $𝑄$ , but its double $(2, 2)$ does not: the parabola requires the second coordinate to be $22 = 4$ . Knowing one point has not given us its scalar multiples. Addition fails for the same pair of points.

The parabola is not a translated line either. Any translated subspace contains the midpoint of each pair of its points: translating the points back to the subspace, averaging, and translating forward stays in the set. But the midpoint of $(0, 0)$ and $(1, 1)$ is $(1 / 2, 1 / 2)$ , and $1 / 2 \neq (1 / 2) 2$ . Thus $𝑄$ is neither a vector subspace nor an affine space.

We can see all three possibilities among functions. In $𝐶 (ℝ)$ , the functions satisfying $𝑓 (0) = 0$ form a subspace, since

(𝑎 𝑓 + 𝑏 𝑔) (0) = 𝑎 𝑓 (0) + 𝑏 𝑔 (0) = 0 .

The functions satisfying $𝑓 (0) = 1$ form a translate of that subspace: subtract the constant function $1$ and the value at zero becomes zero. They do not form a subspace themselves.

For a nonlinear counterpart, consider the following family inside $𝑃 1$ :

Q = {𝑓 𝑎 (𝑡) = 𝑎 + 𝑎 2 𝑡 : 𝑎 \in ℝ} .

Its coefficient pairs lie on exactly the parabola we just studied. The zero function belongs to this family, as does $1 + 𝑡$ , but $2 + 2 𝑡$ does not: its constant coefficient would force $𝑎 = 2$ , while its coefficient of $𝑡$ would then need to be $4$ . The midpoint function $(1 + 𝑡) / 2$ fails for the same reason. These functions live in a vector space, but this particular family is neither a subspace nor an affine space.

This is one difficulty we should expect when studying nonlinear solution sets. Even if we know several members, their linear combinations need not belong to the set. A curve may have a simple parameterization, as our parabola does, without allowing the arithmetic of a vector space. Linear algebra alone no longer reconstructs the family from a basis of solutions.

We should test the set rather than judge only the appearance of its defining equation. The nonlinear-looking condition $𝑦 2 = 0$ still describes the subspace $𝑦 = 0$ . Nonlinear equations do not automatically have solution sets of one particular shape.

Try this 3. Is the set of even continuous functions a subspace of $𝐶 (ℝ)$ ? Recall that even means $𝑓 (- 𝑡) = 𝑓 (𝑡)$ for every $𝑡$ . What about the set of functions with $𝑓 (0) \geq 0$ ?

Try this 4. The circle $𝑥 2 + 𝑦 2 = 1$ is a subset of $ℝ 2$ . Give one reason it is not a subspace and a midpoint counterexample showing that it is not an affine space either.

B.3Bases and Dimension

Suppose we know a few vectors and want to generate others from them. With $𝑢 = (1, 1)$ and $𝑣 = (1, - 1)$ , can we construct $(3, 1)$ ? Writing

𝑎 𝑢 + 𝑏 𝑣 = (𝑎 + 𝑏, 𝑎 - 𝑏) = (3, 1)

gives $𝑎 + 𝑏 = 3$ and $𝑎 - 𝑏 = 1$ . Adding the equations gives $𝑎 = 2$ , and then $𝑏 = 1$ . Thus $(3, 1) = 2 𝑢 + 𝑣$ .

An expression $𝑎 1 𝑣 1 + \dots + 𝑎 𝑘 𝑣 𝑘$ is a linear combination of $𝑣 1, \dots, 𝑣 𝑘$ . The span of these vectors is the set of all their linear combinations, with arbitrary real coefficients. Their span is always a subspace: adding or scaling combinations just changes the coefficients. It is the smallest subspace containing the given vectors.

Our two vectors span all of $ℝ 2$ . Indeed, the equations $𝑎 + 𝑏 = 𝑥$ and $𝑎 - 𝑏 = 𝑦$ have the solution

𝑎 = 𝑥 + 𝑦 2, 𝑏 = 𝑥 - 𝑦 2

for every $(𝑥, 𝑦)$ . The representation is also unique. To understand that second feature, suppose the same vector has two representations. Subtracting them produces a linear combination of $𝑢$ and $𝑣$ equal to zero. The question of uniqueness is therefore a question about combinations producing zero.

Definition B.4 (Linear independence). The vectors $𝑣 1, \dots, 𝑣 𝑘$ are linearly independent if

𝑎 1 𝑣 1 + \dots + 𝑎 𝑘 𝑣 𝑘 = 0

forces $𝑎 1 = \dots = 𝑎 𝑘 = 0$ . Otherwise they are linearly dependent.

For $𝑢$ and $𝑣$ above, a zero combination gives $𝑎 + 𝑏 = 0$ and $𝑎 - 𝑏 = 0$ , so both coefficients vanish. They are independent. If we added the third vector $𝑤 = (2, 0)$ to our list, independence would fail because $𝑢 + 𝑣 - 𝑤 = 0$ . The third vector contributes no new direction to the span.

Definition B.5 (Basis and dimension). A basis of a vector space is a linearly independent collection which spans the space. Every vector then has a unique expression as a linear combination of the basis vectors. In a finite-dimensional space, every basis has the same number of vectors; that number is the dimension.

Spanning supplies existence of the representation; independence supplies uniqueness. These are different jobs, and both are necessary.

For example, $1, 𝑡, 𝑡 2$ form a basis of $𝑃 2$ . They span it because every member has the form $𝑎 + 𝑏 𝑡 + 𝑐 𝑡 2$ . To check independence, suppose

𝑎 + 𝑏 𝑡 + 𝑐 𝑡 2 = 0 f o r e v e r y 𝑡 \in ℝ .

Evaluating at $0$ gives $𝑎 = 0$ . Differentiating the identity and evaluating at $0$ gives $𝑏 = 0$ , and differentiating twice gives $2 𝑐 = 0$ . Thus all three coefficients vanish, so $d i m 𝑃 2 = 3$ .

The words "for every $𝑡$ " are essential. Independence concerns the functions, not their values at one chosen input. All three functions $𝑡, 𝑡 2, 𝑡 3$ vanish at zero, yet they are independent as functions. One evaluation discards most of the information in a function.

In the ordered basis $B = (1, 𝑡, 𝑡 2)$ , the polynomial $𝑝 (𝑡) = 2 + 3 𝑡 + 𝑡 2$ has coordinate vector

[𝑝] B = ⎛ ⎜ ⎜ ⎜ ⎝ 231 ⎞ ⎟ ⎟ ⎟ ⎠ .

Now use $C = (1, 1 + 𝑡, (1 + 𝑡) 2)$ . To find the new coordinates, write

𝑎 + 𝑏 (1 + 𝑡) + 𝑐 (1 + 𝑡) 2 = (𝑎 + 𝑏 + 𝑐) + (𝑏 + 2 𝑐) 𝑡 + 𝑐 𝑡 2 .

Matching coefficients with $2 + 3 𝑡 + 𝑡 2$ gives $𝑐 = 1$ , $𝑏 = 1$ , and $𝑎 = 0$ . Hence

[𝑝] C = ⎛ ⎜ ⎜ ⎜ ⎝ 011 ⎞ ⎟ ⎟ ⎟ ⎠ .

The same coefficient matching works uniquely for every polynomial in $𝑃 2$ , so $C$ is another basis. The polynomial did not change; the numbers describing it changed because we changed the basis.

A few standard finite-dimensional facts are useful to remember. In a space of dimension $𝑛$ , an independent list has at most $𝑛$ vectors, and a spanning list has at least $𝑛$ . An independent list of exactly $𝑛$ vectors is automatically a basis, as is a spanning list of exactly $𝑛$ vectors. An independent list can be extended to a basis; a spanning list can have redundant vectors removed until it becomes a basis. These facts formalize the idea that dimension counts independent directions, not the number of entries used to write a particular object.

Try this 5. Do $1 + 𝑡$ , $1 - 𝑡$ , and $𝑡 2$ form a basis of $𝑃 2$ ? If so, find the coordinates of $2 + 3 𝑡 + 𝑡 2$ in this ordered basis.

Try this 6. Let $𝑈$ be the subspace of $ℝ 3$ satisfying $𝑥 + 𝑦 + 𝑧 = 0$ . Find a basis and its dimension. Why does writing its vectors with three entries not make its dimension three?

B.4Linear Maps, Kernels, and Images

We have been combining vectors inside a space. What happens when we apply an operation to them? A matrix sends an input vector to an output vector; differentiation sends an input function to another function. Both respect the arithmetic we have been reviewing:

𝐴 (𝑎 𝑢 + 𝑏 𝑣) = 𝑎 𝐴 𝑢 + 𝑏 𝐴 𝑣, (𝑎 𝑓 + 𝑏 𝑔)' = 𝑎 𝑓' + 𝑏 𝑔' .

Definition B.6 (Linear map). A map $𝑇 : 𝑉 \to 𝑊$ between real vector spaces is linear if

𝑇 (𝑎 𝑢 + 𝑏 𝑣) = 𝑎 𝑇 (𝑢) + 𝑏 𝑇 (𝑣)

for every $𝑢, 𝑣 \in 𝑉$ and $𝑎, 𝑏 \in ℝ$ .

A linear map necessarily sends zero to zero. A map which adds a fixed nonzero vector, such as $𝑥 \mapsto 𝑥 + 1$ on $ℝ$ , fails this test. Sending zero to zero is necessary, but not sufficient: $𝑥 \mapsto 𝑥 2$ does that and still fails linearity.

The domain and codomain are part of a map's specification. Consider

𝐷 : 𝑃 2 ⟶ 𝑃 1, 𝐷 𝑝 = 𝑝' .

Differentiation sends a polynomial of degree at most two to one of degree at most one. If $𝑝 = 𝑎 + 𝑏 𝑡 + 𝑐 𝑡 2$ , then $𝐷 𝑝 = 𝑏 + 2 𝑐 𝑡$ . Which inputs disappear, and which outputs can occur?

Definition B.7 (Kernel and image). For a linear map $𝑇 : 𝑉 \to 𝑊$ , its kernel and image are

k e r 𝑇 = {𝑣 \in 𝑉 : 𝑇 (𝑣) = 0}, i m 𝑇 = {𝑇 (𝑣) : 𝑣 \in 𝑉} .

The kernel is a subspace of $𝑉$ , and the image is a subspace of $𝑊$ .

For the kernel, the subspace test follows from $𝑇 (𝑎 𝑢 + 𝑏 𝑣) = 𝑎 0 + 𝑏 0 = 0$ whenever $𝑢, 𝑣$ lie in the kernel. For the image, a combination of outputs $𝑎 𝑇 (𝑢) + 𝑏 𝑇 (𝑣)$ is the output $𝑇 (𝑎 𝑢 + 𝑏 𝑣)$ . Both contain zero.

In our differentiation example, $𝐷 𝑝 = 0$ forces $𝑏 = 𝑐 = 0$ , so

k e r 𝐷 = s p a n {1} .

Every polynomial $𝑟 + 𝑠 𝑡$ in $𝑃 1$ is the derivative of $𝑟 𝑡 + (𝑠 / 2) 𝑡 2$ . Therefore $i m 𝐷 = 𝑃 1$ . Differentiation loses the constant coefficient, but it reaches every allowed output.

A map is injective when distinct inputs have distinct outputs, and surjective when every member of its codomain occurs as an output. For a linear map, injectivity is equivalent to having kernel ${0}$ . Indeed, $𝑇 (𝑢) = 𝑇 (𝑣)$ is equivalent to $𝑇 (𝑢 - 𝑣) = 0$ . A nonzero kernel vector is exactly an ambiguity in recovering the input from its output.

Thus $𝐷 : 𝑃 2 \to 𝑃 1$ is surjective but not injective. If we instead declare the codomain to be $𝑃 2$ , the same differentiation rule is no longer surjective: no derivative of a member of $𝑃 2$ equals $𝑡 2$ . If we restrict the domain to the polynomials in $𝑃 2$ with zero constant coefficient, then differentiation becomes both injective and surjective onto $𝑃 1$ . The spaces named on either side of the arrow matter!

A linear map which is both injective and surjective is a linear isomorphism. Its inverse is linear too: recovering inputs from a combination of outputs gives the corresponding combination of inputs. The coordinate map $𝑝 \mapsto [𝑝] B$ is an isomorphism from $𝑃 2$ to $ℝ 3$ . This gives a precise meaning to saying that their linear algebra is the same.

Evaluation is another useful example:

𝐸 0 : 𝐶 (ℝ) ⟶ ℝ, 𝐸 0 (𝑓) = 𝑓 (0) .

This map is linear because $(𝑎 𝑓 + 𝑏 𝑔) (0) = 𝑎 𝑓 (0) + 𝑏 𝑔 (0)$ . Its kernel is the subspace of functions vanishing at zero. It is surjective because constant functions attain any requested output. It is far from injective: many different functions have the same value at zero.

For finite-dimensional domains, there is a useful accounting rule.

Theorem B.8 (Rank–nullity). If $𝑇 : 𝑉 \to 𝑊$ is linear and $𝑉$ is finite-dimensional, then

d i m 𝑉 = d i m k e r 𝑇 + d i m i m 𝑇 .

The two dimensions on the right are called the nullity and rank.

Here is why. Choose a basis $𝑘 1, \dots, 𝑘 𝑟$ of the kernel and extend it to a basis $𝑘 1, \dots, 𝑘 𝑟, 𝑣 1, \dots, 𝑣 𝑠$ of $𝑉$ . Applying $𝑇$ to a linear combination erases the $𝑘 𝑖$ terms, so $𝑇 (𝑣 1), \dots, 𝑇 (𝑣 𝑠)$ span the image. They are also independent. If $\sum 𝑗 𝑎 𝑗 𝑇 (𝑣 𝑗) = 0$ , then $\sum 𝑗 𝑎 𝑗 𝑣 𝑗$ lies in the kernel and can be written as a combination of the $𝑘 𝑖$ . Independence of the extended basis forces every $𝑎 𝑗$ to be zero. Thus the image has dimension $𝑠$ , and $d i m 𝑉 = 𝑟 + 𝑠$ as claimed.

For polynomial differentiation, the count is $3 = 1 + 2$ : one independent constant is lost and two independent output coefficients remain. We will use this numerical dimension formula only in finite-dimensional spaces.

Try this 7. For $𝑇 : ℝ 3 \to ℝ 2$ given by $𝑇 (𝑥, 𝑦, 𝑧) = (𝑥 + 𝑦, 𝑦 + 𝑧)$ , find a basis of the kernel and determine the image. Check rank–nullity. Is $𝑇$ injective? Surjective?

Try this 8. Let $𝐽 : 𝑃 1 \to 𝑃 2$ send $𝑞$ to $𝐽 𝑞 (𝑡) = \int 𝑡 0 𝑞 (𝑠) 𝑑 𝑠$ . Find its kernel and image. Explain why it is an isomorphism onto the subspace of polynomials with zero constant coefficient, but not onto all of $𝑃 2$ .

B.5Matrices and Coordinates

How much of a linear map must we know to determine all of it? Suppose $𝑣 1, \dots, 𝑣 𝑛$ is a basis of its domain. Every input has a unique expression $𝑣 = 𝑐 1 𝑣 1 + \dots + 𝑐 𝑛 𝑣 𝑛$ , and linearity forces

𝑇 (𝑣) = 𝑐 1 𝑇 (𝑣 1) + \dots + 𝑐 𝑛 𝑇 (𝑣 𝑛) .

We only need the images of the basis vectors. Once those are known, every other output is a linear combination with the same coefficients.

Choose an ordered basis $B = (𝑣 1, \dots, 𝑣 𝑛)$ of the domain and $C = (𝑤 1, \dots, 𝑤 𝑚)$ of the codomain. Place the coordinates of $𝑇 (𝑣 𝑗)$ into column $𝑗$ of a matrix $𝑀$ . The formula above becomes

[𝑇 (𝑣)] C = 𝑀 [𝑣] B .

This is why matrix multiplication takes a linear combination of columns: the columns describe what happens to basis vectors, and the input coordinates tell us how much of each image to use. A map from an $𝑛$ -dimensional space to an $𝑚$ -dimensional space has an $𝑚 \times 𝑛$ matrix in the chosen bases.

For $𝐷 : 𝑃 2 \to 𝑃 1$ , use the bases $(1, 𝑡, 𝑡 2)$ and $(1, 𝑡)$ . The basis images are

𝐷 (1) = 0, 𝐷 (𝑡) = 1, 𝐷 (𝑡 2) = 2 𝑡 .

Their coordinate columns give

𝑀 = (010002) .

Multiplying confirms the meaning:

𝑀 ⎛ ⎜ ⎜ ⎜ ⎝ 𝑎 𝑏 𝑐 ⎞ ⎟ ⎟ ⎟ ⎠ = (𝑏 2 𝑐) .

The output column records the polynomial $𝑏 + 2 𝑐 𝑡$ , the derivative of $𝑎 + 𝑏 𝑡 + 𝑐 𝑡 2$ . The zero first column also makes the lost constant visible.

Composition explains matrix multiplication. If $𝑇 : 𝑈 \to 𝑉$ has matrix $𝐴$ and $𝑆 : 𝑉 \to 𝑊$ has matrix $𝐵$ , using the same basis of $𝑉$ for the intermediate coordinates, then

[𝑆 (𝑇 (𝑢))] = 𝐵 (𝐴 [𝑢]) = (𝐵 𝐴) [𝑢] .

The rightmost map acts first. For example, differentiation from $𝑃 1$ to $𝑃 0$ has matrix $𝑁 = (01)$ . Differentiating twice from $𝑃 2$ to $𝑃 0$ therefore has matrix

𝑁 𝑀 = (002),

which sends $(𝑎, 𝑏, 𝑐)$ to $2 𝑐$ . Order matters, and in this example the reverse matrix product is not even defined because the dimensions do not match. Even for square matrices, the products $𝐴 𝐵$ and $𝐵 𝐴$ need not agree.

An isomorphism between finite-dimensional spaces has an invertible square matrix. Its inverse matrix records the inverse map. In particular, changing basis is invertible: we have changed how we describe a vector, not discarded any information about it.

Return to the two bases $B = (1, 𝑡, 𝑡 2)$ and $C = (1, 1 + 𝑡, (1 + 𝑡) 2)$ of $𝑃 2$ . Put the $B$ -coordinates of the $C$ basis vectors into columns:

𝑃 = ⎛ ⎜ ⎜ ⎜ ⎝ 111012001 ⎞ ⎟ ⎟ ⎟ ⎠ .

Then $[𝑝] B = 𝑃 [𝑝] C$ . Solving the triangular system gives

𝑃 - 1 = ⎛ ⎜ ⎜ ⎜ ⎝ 1 - 1 1 01 - 2 001 ⎞ ⎟ ⎟ ⎟ ⎠ .

Applying $𝑃 - 1$ to $(2, 3, 1)$ recovers $(0, 1, 1)$ , as in our earlier calculation. For a map $𝑇 : 𝑉 \to 𝑉$ with matrix $𝐴$ in the old basis, the matrix in the new basis is

𝐴 n e w = 𝑃 - 1 𝐴 𝑃 .

Read this from right to left: convert the input to old coordinates, apply the old matrix, then convert the output to new coordinates. This explains the formula without memorizing where the inverse goes.

Try this 9. Find the matrix of $𝐽 : 𝑃 1 \to 𝑃 2$ from Try this 8 in the standard polynomial bases. Multiply the matrices to check that $𝐷 \circ 𝐽$ is the identity on $𝑃 1$ . What does $𝐽 \circ 𝐷$ do to a polynomial in $𝑃 2$ ?

Try this 10. In $ℝ 2$ , let $C = ((1, 1), (1, - 1))$ . Write the matrix $𝑃$ which converts $C$ -coordinates to standard coordinates. Find the $C$ -coordinates of $(4, 2)$ , and check by reconstructing the vector.

B.6Linear Equations and Affine Spaces

We now have the language to describe an entire solution set, not just calculate one answer. Begin with

𝑥 + 2 𝑦 = 3, 2 𝑥 + 4 𝑦 = 6 .

Subtracting twice the first equation from the second leaves $0 = 0$ . There is only one independent constraint. Taking $𝑦 = 𝑠$ gives

(𝑥 𝑦) = (3 - 2 𝑠 𝑠) = (30) + 𝑠 (- 2 1), 𝑠 \in ℝ .

One particular solution locates the line, while a kernel vector gives its direction. For

𝐴 = (1224),

the kernel is $s p a n {(- 2, 1)}$ . Moving in this direction changes the input without changing its output under $𝐴$ .

Why must this description work for every consistent linear equation? Let $𝑇 : 𝑉 \to 𝑊$ be linear, and suppose $𝑣 𝑝$ is one solution of $𝑇 (𝑣) = 𝑏$ . If $𝑣$ is any other solution, then

𝑇 (𝑣 - 𝑣 𝑝) = 𝑇 (𝑣) - 𝑇 (𝑣 𝑝) = 𝑏 - 𝑏 = 0 .

Thus $𝑣 - 𝑣 𝑝 \in k e r 𝑇$ . Conversely, if $ℎ \in k e r 𝑇$ , then $𝑇 (𝑣 𝑝 + ℎ) = 𝑏 + 0 = 𝑏$ . We have proved both directions of the statement:

Theorem B.9 (Solutions of a linear equation). For a linear map $𝑇 : 𝑉 \to 𝑊$ , the equation $𝑇 (𝑣) = 𝑏$ has a solution exactly when $𝑏 \in i m 𝑇$ . If $𝑣 𝑝$ is one solution, the full solution set is

{𝑣 : 𝑇 (𝑣) = 𝑏} = 𝑣 𝑝 + k e r 𝑇 = {𝑣 𝑝 + ℎ : ℎ \in k e r 𝑇} .

The notation $𝑇 - 1 ({𝑏})$ is also used for this set, the preimage of $𝑏$ . It does not assert that $𝑇$ has an inverse map. Our matrix $𝐴$ does not: each attainable output has a whole line of inputs.

An equation can also be inconsistent. If we change the second right-hand side in the matrix system to $7$ , subtraction leaves $0 = 1$ . Equivalently, $(3, 7)$ is not in the image of $𝐴$ . There is no particular solution from which to start a translation.

Definition B.10 (Affine subspace). An affine subspace of a vector space $𝑉$ is a nonempty set of the form $𝑣 𝑝 + 𝑈$ , where $𝑈$ is a vector subspace. We also call this an affine space when the surrounding vector space is understood. If $𝑈$ is finite-dimensional, the affine space has dimension $d i m 𝑈$ .

Every vector subspace is also an affine space: choose $𝑣 𝑝 = 0$ . An affine space contains zero exactly when it is a vector subspace. Indeed, if $0 = 𝑣 𝑝 + 𝑢$ for some $𝑢 \in 𝑈$ , then $𝑣 𝑝 = - 𝑢 \in 𝑈$ , and translation by $𝑣 𝑝$ leaves $𝑈$ unchanged. When distinguishing solution sets, we can therefore say "a vector space," "affine but not a vector space," or "neither." The empty set counts as neither under our definition.

What arithmetic survives in a translated space? Let $𝑣 = 𝑣 𝑝 + 𝑢$ and $𝑤 = 𝑣 𝑝 + 𝑧$ . For any real $𝑎$ ,

𝑎 𝑣 + (1 - 𝑎) 𝑤 = 𝑣 𝑝 + 𝑎 𝑢 + (1 - 𝑎) 𝑧

still belongs to $𝑣 𝑝 + 𝑈$ . More generally, linear combinations whose coefficients sum to $1$ stay inside the affine space. These are affine combinations. Arbitrary sums and scalar multiples can fail, but midpoints and entire lines through pairs of points remain available. Differences of points lie in the direction subspace $𝑈$ .

The distinction is just as useful for functions. Prescribe a polynomial derivative:

𝐷 𝑝 = 2 + 6 𝑡, 𝑝 \in 𝑃 2 .

We can find one answer by antidifferentiating: $𝑝 𝑝 (𝑡) = 2 𝑡 + 3 𝑡 2$ . Since $k e r 𝐷$ consists of constants, every answer is

𝑝 (𝑡) = 2 𝑡 + 3 𝑡 2 + 𝑐, 𝑐 \in ℝ .

The familiar constant of integration is the kernel freedom! This is a one-dimensional affine space inside the three-dimensional vector space $𝑃 2$ . It is not a vector subspace, since the zero polynomial does not have the prescribed derivative. Subtracting any two answers gives a constant, while averaging two answers gives another answer.

Compare this with the family $𝑎 + 𝑎 2 𝑡$ from section B.2. Its failure to contain the midpoint of two of its members already rules out an affine description. There is no fixed subspace whose translate gives that parabola of functions. The ambient function space still lets us add and subtract functions, but the constraint need not respect those operations. That is the structural question we will ask of differential equations: which operations on known solutions produce new solutions of the same equation?

Try this 11. Find all $(𝑥, 𝑦, 𝑧)$ with $𝑥 + 𝑦 + 𝑧 = 1$ . Write the answer as a particular vector plus a subspace, using your basis from Try this 6. Find its affine dimension. Does adding two solutions preserve the equation? Does taking their midpoint?

Try this 12. Describe all $𝑝 \in 𝑃 2$ with $𝑝' = 1 + 2 𝑡$ . Then impose $𝑝 (0) = 3$ . Describe how the solution set changes. Is the final singleton a vector subspace, an affine space, or neither?

B.7Eigenvalues and Eigenvectors

A linear map becomes a matrix once we choose a basis, and the preceding sections showed that we can choose different bases. Could a good choice make the map particularly simple? The best possible outcome would be for each basis vector to map to a multiple of itself. Then applying the map would scale each coordinate separately, without mixing them.

Consider

𝐴 = (2112) .

The standard basis vectors do not have that property: $𝐴 (1, 0) = (2, 1)$ and $𝐴 (0, 1) = (1, 2)$ . But try the vectors $𝑣 1 = (1, 1)$ and $𝑣 2 = (1, - 1)$ :

𝐴 𝑣 1 = (3, 3) = 3 𝑣 1, 𝐴 𝑣 2 = (1, - 1) = 𝑣 2 .

Along one line the map triples vectors; along the other it leaves them unchanged. These vectors form a basis, so their two simple behaviors determine the action of $𝐴$ on every vector.

Definition B.11 (Eigenvalues and eigenvectors). Let $𝑇 : 𝑉 \to 𝑉$ be linear. A nonzero vector $𝑣$ is an eigenvector with eigenvalue $𝜆$ if

𝑇 (𝑣) = 𝜆 𝑣 .

The eigenspace for an eigenvalue $𝜆$ is $k e r (𝑇 - 𝜆 𝐼)$ , where $𝐼$ is the identity map.

Zero is excluded from being an eigenvector because $𝑇 (0) = 𝜆 0$ for every $𝜆$ ; it cannot identify a special scaling factor. Zero does belong to every eigenspace, together with the eigenvectors for that eigenvalue. The eigenvalue itself can be zero: a nonzero vector in the kernel is an eigenvector with eigenvalue zero.

How do we find these vectors without guessing them? For a square matrix, rewrite $𝐴 𝑣 = 𝜆 𝑣$ as

(𝐴 - 𝜆 𝐼) 𝑣 = 0 .

We need this equation to have a nonzero solution. A square matrix has a nonzero kernel exactly when it is singular, which is equivalent to its determinant being zero. Thus the possible eigenvalues satisfy

d e t (𝐴 - 𝜆 𝐼) = 0 .

Recall that the determinant of $(𝑎 𝑏 𝑐 𝑑)$ is $𝑎 𝑑 - 𝑏 𝑐$ . For our matrix, the eigenvalue equation is

(2 - 𝜆) 2 - 1 = (𝜆 - 1) (𝜆 - 3) = 0 .

Once $𝜆 = 3$ is known, solving $(𝐴 - 3 𝐼) 𝑣 = 0$ gives $𝑦 = 𝑥$ ; the eigenspace is $s p a n {(1, 1)}$ . For $𝜆 = 1$ , the equations give $𝑦 = - 𝑥$ , so the eigenspace is $s p a n {(1, - 1)}$ . Finding eigenvalues supplies the scalars; finding the corresponding kernels supplies the vectors.

Place these eigenvectors into columns:

𝑃 = (11 1 - 1), Λ = (3001) .

The two eigenvector equations together say $𝐴 𝑃 = 𝑃 Λ$ . Because the columns of $𝑃$ are a basis, $𝑃$ is invertible, and we obtain

𝑃 - 1 𝐴 𝑃 = Λ, 𝐴 = 𝑃 Λ 𝑃 - 1 .

This is diagonalization: writing a linear map in a basis of eigenvectors makes its matrix diagonal. Conversely, any basis which makes the matrix diagonal must consist of eigenvectors, since each column then has only its own coordinate possibly nonzero. Thus a matrix is diagonalizable exactly when it has a basis of eigenvectors.

Eigenvectors for distinct eigenvalues are linearly independent. In particular, an $𝑛 \times 𝑛$ matrix with $𝑛$ distinct eigenvalues over the chosen scalar field has an eigenbasis. Repeated eigenvalues require us to check the dimensions of their eigenspaces; repetition alone does not decide whether diagonalization is possible.

Why is diagonalization useful? When we multiply powers, adjacent factors $𝑃 - 1 𝑃$ cancel, giving

𝐴 𝑘 = 𝑃 Λ 𝑘 𝑃 - 1, Λ 𝑘 = (3 𝑘 0 01) (𝑘 = 0, 1, 2, \dots) .

In eigenvector coordinates, repeated matrix multiplication is ordinary scalar exponentiation. For example, $(3, 1) = 2 𝑣 1 + 𝑣 2$ , so $𝐴 𝑘 (3, 1) = 2 \cdot 3 𝑘 𝑣 1 + 𝑣 2$ without multiplying out $𝐴 𝑘$ .

This is the algebra we will need when we study the matrix exponential in the main text: an eigenbasis makes all powers of a matrix act coordinate by coordinate.

We cannot assume such a basis exists. For

𝐵 = (1101),

the only eigenvalue is $1$ , but $(𝐵 - 𝐼) (𝑥, 𝑦) = (𝑦, 0)$ , so its eigenspace is only the line $𝑦 = 0$ . There are not two independent eigenvectors, and $𝐵$ cannot be diagonalized. By contrast, the identity matrix also has only eigenvalue $1$ , but every nonzero vector is an eigenvector, so it already is diagonal.

The scalar field matters too. The real rotation matrix

𝑅 = (0 - 1 10)

turns each nonzero real vector through a right angle, so it has no real eigenvector. Its characteristic equation is $𝜆 2 + 1 = 0$ . If we allow complex coordinates, with $𝑖 2 = - 1$ , the eigenvalues are $𝑖$ and $- 𝑖$ , with eigenvectors $(1, - 𝑖)$ and $(1, 𝑖)$ respectively. They form a basis of $ℂ 2$ , so $𝑅$ is diagonalizable over the complex numbers even though it is not diagonalizable over the reals. This is why complex numbers will appear when we study real oscillations.

Finally, an eigenvector can be a function. Differentiation on the space of smooth real-valued functions on $ℝ$ sends $𝑒 𝜆 𝑡$ to $𝜆 𝑒 𝜆 𝑡$ for every real $𝜆$ . The familiar exponential is therefore an eigenvector of differentiation. This does not assert that these functions form a basis of that infinite-dimensional space; the eigenvector equation alone is enough to explain why exponentials are natural functions to try in linear differential equations.

Try this 13. Find the eigenvalues and eigenspaces of $𝐴 = (4102)$ . Write $𝐴 = 𝑃 Λ 𝑃 - 1$ using an eigenbasis, and use the basis to compute $𝐴 𝑘 (3, 2)$ for nonnegative integers $𝑘$ .

Try this 14. Regard differentiation as a map $𝐷 : 𝑃 2 \to 𝑃 2$ . Show that its only eigenvalue is zero and find the eigenspace. Is this map diagonalizable? Explain why the exponential eigenfunctions from the preceding paragraph do not contradict your answer.

Answers to the Review Questions

1. $3 𝑓 - 2 𝑔 = 3 - 5 𝑡 - 2 𝑡 2$ , and $- 𝑓 = - 1 + 𝑡$ . The function $𝑓$ is not zero: for example, $𝑓 (0) = 1$ . Being the zero function requires vanishing at every input.

2. Addition and scalar multiplication stay among $2 \times 3$ matrices and satisfy the vector-space rules entry by entry. The zero matrix and entrywise negatives supply zero and additive inverses. Rectangular shape causes no difficulty; multiplication of two such matrices is not required.

3. Even functions contain the zero function, and $(𝑎 𝑓 + 𝑏 𝑔) (- 𝑡) = 𝑎 𝑓 (𝑡) + 𝑏 𝑔 (𝑡) = (𝑎 𝑓 + 𝑏 𝑔) (𝑡)$ , so they form a subspace. The condition $𝑓 (0) \geq 0$ fails closure under negative scaling: the constant function $1$ belongs, while its negative does not.

4. The circle does not contain zero. Its points $(1, 0)$ and $(- 1, 0)$ have midpoint $(0, 0)$ outside the circle, so it is not affine either.

5. In the identity $𝑎 (1 + 𝑡) + 𝑏 (1 - 𝑡) + 𝑐 𝑡 2 = 0$ , coefficient matching gives $𝑎 + 𝑏 = 0$ , $𝑎 - 𝑏 = 0$ , and $𝑐 = 0$ , hence all coefficients vanish. Three independent vectors in the three-dimensional space $𝑃 2$ form a basis. For the requested polynomial, $𝑎 + 𝑏 = 2$ , $𝑎 - 𝑏 = 3$ , and $𝑐 = 1$ , so the coordinates are $(5 / 2, - 1 / 2, 1)$ .

6. Write $(𝑥, 𝑦, 𝑧) = (𝑥, 𝑦, - 𝑥 - 𝑦) = 𝑥 (1, 0, - 1) + 𝑦 (0, 1, - 1)$ . The two displayed vectors span $𝑈$ and are independent by looking at their first two entries. Thus $d i m 𝑈 = 2$ . Three entries record the vector, but the constraint leaves only two independent choices.

7. Kernel vectors satisfy $𝑥 = - 𝑦$ and $𝑧 = - 𝑦$ , so $k e r 𝑇 = s p a n {(- 1, 1, - 1)}$ . Every $(𝑟, 𝑠)$ is the image of $(𝑟, 0, 𝑠)$ , hence the image is $ℝ 2$ . The dimension count is $3 = 1 + 2$ . The map is surjective and not injective.

8. $𝐽 (𝑎 + 𝑏 𝑡) = 𝑎 𝑡 + (𝑏 / 2) 𝑡 2$ . Its kernel is ${0}$ and its image is $s p a n {𝑡, 𝑡 2}$ . It is injective and maps onto this subspace, but cannot produce a polynomial with nonzero constant term.

9. The matrix of $𝐽$ is

𝐾 = ⎛ ⎜ ⎜ ⎜ ⎝ 0010 0 1 / 2 ⎞ ⎟ ⎟ ⎟ ⎠ .

With $𝑀$ the matrix of $𝐷$ from section B.5,

𝑀 𝐾 = (1001), 𝐾 𝑀 = ⎛ ⎜ ⎜ ⎜ ⎝ 000010001 ⎞ ⎟ ⎟ ⎟ ⎠ .

Thus $𝐷 \circ 𝐽$ is the identity on $𝑃 1$ , while $(𝐽 \circ 𝐷) 𝑝 = 𝑝 - 𝑝 (0)$ removes the constant coefficient.

10. $𝑃 = (11 1 - 1)$ . Solving $𝑎 + 𝑏 = 4$ , $𝑎 - 𝑏 = 2$ gives coordinates $(3, 1)$ . Indeed, $3 (1, 1) + (1, - 1) = (4, 2)$ .

11. The full set is

(0, 0, 1) + 𝑎 (1, 0, - 1) + 𝑏 (0, 1, - 1), 𝑎, 𝑏 \in ℝ .

It is affine of dimension two and does not contain zero. Adding two solutions gives coordinates summing to $2$ , so the sum fails the equation. Their midpoint has coordinates summing to $1$ and succeeds.

12. The derivative condition gives $𝑝 = 𝑐 + 𝑡 + 𝑡 2$ , an affine line. The value at zero forces $𝑐 = 3$ , leaving the singleton ${3 + 𝑡 + 𝑡 2}$ . This is a zero-dimensional affine space, a translate of the zero subspace. It is not a vector subspace because it does not contain the zero polynomial.

13. The characteristic equation is $(4 - 𝜆) (2 - 𝜆) = 0$ . The eigenspaces for $4$ and $2$ are respectively $s p a n {(1, 0)}$ and $s p a n {(1, - 2)}$ . Thus we can take

𝑃 = (11 0 - 2), Λ = (4002) .

Since $(3, 2) = 4 (1, 0) - (1, - 2)$ , the requested vector is

𝐴 𝑘 (3, 2) = 4 \cdot 4 𝑘 (1, 0) - 2 𝑘 (1, - 2) = (4 \cdot 4 𝑘 - 2 𝑘, 2 \cdot 2 𝑘) .

14. If $𝑝' = 𝜆 𝑝$ with $𝜆 \neq 0$ and $𝑝$ a nonzero polynomial, the right side has the same degree as $𝑝$ , whereas the derivative has lower degree (or is zero for a constant). This is impossible. For $𝜆 = 0$ the eigenspace consists of constants and has dimension one, so it cannot supply a basis of $𝑃 2$ . The map is not diagonalizable. For $𝜆 \neq 0$ , $𝑒 𝜆 𝑡$ does not belong to $𝑃 2$ ; changing the domain changes the available eigenvectors.