First, watch a matrix
move space.
SVD, PCA, and ICA each select directions in a matrix or dataset. Start by watching how a matrix changes a vector’s direction and length.
A vector is an arrow, described by coordinates. A matrix is a rule for moving that arrow. For a two-dimensional vector, the rule is
The space-moving machine
Experiment 01Try “Shear,” then sweep the angle. Some unit arrows grow much more than others. SVD will find the most- and least-stretched directions.
How a shear changes the basis arrows
The original basis arrows are perpendicular. Under the Shear preset, their images tilt toward each other, changing the angle between them. The matrix is still linear: it respects addition and scaling, so the origin stays put.
Rank counts surviving directions.
“Flatten” sends a two-dimensional circle onto a line. The map has rank 1. A zero matrix collapses everything to a point and has rank 0. Rank measures dimension, rather than physical size.
A complicated move.
Three simple ones.
Every real matrix has an SVD. It finds special perpendicular input arrows that become perpendicular output arrows. Each input–output pair has its own stretch factor; the output can point in a different direction from the input.
Take the transformation apart
Experiment 02The colored arrows begin at v₁ and v₂. After Vᵀ they lie on the coordinate axes. After Σ their lengths are σ₁ and σ₂. After U they point along u₁ and u₂. The faint circle is a reference; the notched outline makes rotations and reflections visible.
Singular values and singular vectors
The columns of V are the input directions vᵢ; the columns of U are the output directions uᵢ. The largest singular value is the maximum output length over all unit inputs. Its right singular vector v₁ is an input that attains that stretch; its left singular vector u₁ gives the output direction. A zero singular value means a direction disappears.
Singular values describe stretch, not variance. They apply to any real matrix, whether it contains data, pixels, or a physical transformation.
The full statement, including rectangular matrices
For A with m rows and n columns, the full real SVD has U of size m × m, Σ of size m × n, and V of size n × n. U and V have orthonormal columns, and Σ has nonnegative entries on its diagonal. The rank is the number of positive singular values. The thin form retains only the needed columns.
An eigenvector keeps its direction under A; a singular vector need not. For example, the shear from chapter 1 stretches the horizontal arrow 2×, but its most-stretched input points about 32° above horizontal and grows about 2.29×.
Since AᵀA = VΣᵀΣVᵀ, the right singular vectors are eigenvectors of AᵀA and the corresponding eigenvalues are σᵢ². Equal singular values permit different bases within their shared subspace. Signs can also change in matching pairs.
Keep the structure.
Spend fewer numbers.
SVD also writes a matrix as a sum of simple pieces. Each piece uses one input direction, one output direction, and one strength.
Reconstructing a 32 × 32 image
Experiment 03Try k = 1, then add pieces slowly. “Energy” means the sum of squared singular values, not perceptual image quality. The squared-error readout uses the raw reconstruction; display intensities are clipped to [0, 1]. Factor entries count real numbers, not compressed file bytes.
Reconstruction error
Among matrices with rank at most k, truncating the SVD minimizes the sum of squared entry errors (the squared Frobenius norm). The discarded singular values account for exactly that error:
It also minimizes spectral-norm error, where the error is σk+1. A small total squared error can still include the loss of a visually important detail.
Which view keeps
the most of the data?
Imagine replacing every two-dimensional observation with one number: its position along a line. PCA chooses the line that preserves the greatest variance. That same line gives the smallest squared reconstruction error.
Variance measures spread using squared deviations from the mean. First subtract the mean of each feature. Then find the direction along which the centered points spread out most. This is the first principal axis. The coordinates of the points along that axis are their scores.
Before clicking “Find PC1,” predict which line will give the widest score distribution.
Find the best one-dimensional view
Experiment 04A wider score distribution means more variance retained. Reversing the axis flips the sign of every score without changing the retained variance.
As you rotate, spread kept along the line plus spread left perpendicular to it stays constant. Those perpendicular offsets are the reconstruction errors. Values use sample variance.
Rotate the line before clicking “Find PC1.” Then change feature units. Covariance-based PCA depends on those units. Standardizing makes each feature have unit sample variance and changes the distance metric being optimized. For a perfectly round covariance, every direction is equally good.
Computing PCA with SVD
From here onward, X has n observations in rows and p features in columns. Let Xc be the centered data. For a unit direction v, the column Xcv lists every observation’s score. Its squared length is the sum of those squared scores. The direction that stretches most therefore produces the greatest score variance:
Keeping the first k axes gives X̂ = XcVkVkᵀ + the mean. This is the same best rank-k approximation from the previous chapter, applied after centering.
Why center? We want variation around the mean, rather than distance from the origin. SVD works on uncentered data too; its leading direction then optimizes an uncentered second moment instead of the usual PCA variance.
Multiplying a feature by c multiplies its variance by c², so changing units can change which direction is PC1. Standardization gives the features equal variance before PCA; whether that is useful depends on what their scales mean.
PC scores are uncorrelated. They need not be independent, meaningful causes, or the most useful features for predicting an outcome. Try the curved data: a line can miss structure even when its retained-variance score looks respectable.
Zero correlation.
Plenty of dependence.
Correlation measures a particular kind of relationship: linear co-variation. Independence asks a stronger question: does knowing one variable tell you anything about the other?
Does knowing x change what you expect about y?
Experiment 05Whitened variables can be dependent
Whitening takes the PCA scores and divides each by its standard deviation. This makes the covariance matrix the identity: unit variance on every axis and zero cross-covariance. That describes second-order statistics, rather than the full joint distribution.
For the parabola, knowing x determines y exactly. The highlighted slice reveals that dependence immediately, even though correlation is zero. A rotation cannot untangle this curved relationship. Now try the rotated square: its plotted coordinates are also dependent, but a rotation can recover the independent coordinates that made it. That is the kind of mixing ICA tackles.
The sensors hear mixtures.
Find what made them.
Two independent sources are linearly mixed into two sensor readings. PCA can remove correlation. ICA uses more of the distribution to estimate the sources.
For one observation, write x = As, where s contains the hidden sources and A is an unknown mixing matrix. In the default example, each source is sampled independently from a uniform distribution. Their joint cloud is a square; mixing turns it into a parallelogram. Switch to Gaussian sources to compare a case where ICA cannot identify the source directions.
Whiten first. Then search for a rotation.
Experiment 06Every rotation keeps covariance equal to the identity.
Original and estimated sources
The optimizer never receives the originals. We use them only here to evaluate this synthetic example. Both traces are standardized for comparison; 80 samples are shown.
“Find a rotation” searches the whitened samples using a simple kurtosis contrast. This is a teaching optimizer for these synthetic distributions, rather than a general-purpose FastICA implementation. Large absolute kurtosis is useful here; kurtosis alone cannot detect every form of non-Gaussianity.
Why does an extra rotation help?
After whitening full-rank, noiseless mixtures of independent unit-variance sources, the remaining mixing is orthogonal. Every rotation preserves identity covariance. To select the source directions, we need information beyond covariance.
Mixtures of independent non-Gaussian variables often look more Gaussian than their constituents. A source direction can therefore stand out through a non-Gaussian marginal distribution. Kurtosis uses fourth powers, giving large deviations extra weight. A Gaussian’s excess kurtosis is 0; a uniform distribution’s is −1.2. Squaring the two kurtoses lets departures on either side of zero contribute to the score. Our demo maximizes κ₁² + κ₂² to seek the original uniform directions.
Try Gaussian sources. In the population, a whitened Gaussian cloud has the same distribution after every rotation. The score is 0 everywhere. A finite sample has small peaks, but optimizing them does not identify the original Gaussian sources.
ICA assumptions and source ambiguities
Standard linear ICA assumes independent sources, a suitable full-rank linear mixing model, and at most one Gaussian source. Here there are two sensors for two sources, with no added noise. Real data can violate the assumptions through dependence, noise, changing mixtures, or too few sensors.
Even in the ideal model, source order, scale, and sign are ambiguous: swapping two sources or multiplying one by a constant can be offset in A. Components are not naturally ranked by importance. An ICA component also needs domain interpretation; independence alone doesn't make it a true cause.
Mixing directions and unmixing directions are different
The columns of A show how each source affects the sensors. The rows of W = A⁻¹ are filters used to extract sources: s = Wx. These need not be orthogonal in the original sensor coordinates. The rotation picture applies after whitening under our stated model.
Algorithms such as Infomax and FastICA optimize different, related independence or non-Gaussianity criteria. The angular search in this lesson uses the squared-kurtosis criterion shown above.
SVD, PCA, and ICA
compared.
SVD describes a matrix. PCA preserves variance in centered data. ICA estimates independent sources. Use the table to compare their objectives and assumptions.
| Property | SVD | PCA | Linear ICA |
|---|---|---|---|
| Main question | How does this matrix act? | Which linear view preserves variance? | Which components are independent? |
| What it uses | Algebraic matrix structure | Mean and covariance | Distributional information beyond covariance |
| Natural ordering | Decreasing singular values | Decreasing explained variance | No inherent importance order |
| Typical use | Low-rank approximation; stable linear algebra | Compression, exploration, decorrelation | Separation of suitably mixed independent sources |
| Not guaranteed | Statistical independence | Independent sources or causal meaning | Unique signs, scales, or ordering; recovery for arbitrary data |
Glossary
- Orthonormal basis
- Perpendicular unit directions. Rotating or reflecting to such a basis preserves lengths and angles.
- Rank
- The dimension of a matrix's column space; equivalently, the number of positive singular values.
- Singular value
- A nonnegative stretch factor associated with a pair of input and output singular directions.
- Principal axis / score
- An axis is a direction in feature space; a score is one observation's coordinate along it.
- Whitening
- Centering, decorrelating, and rescaling retained directions to make covariance the identity.
- Independence
- The joint distribution factorizes into the product of marginals. Zero covariance is generally weaker.
Inspired by nine explainers.
This lesson's prose, figures, and simulations are new. The nine explainers below informed the teaching route and offer further derivations. The links open the original sources.
For original URLs and publication details, see the source list. The archived works retain their original rights. All experiments use synthetic data.