Image Formation & Sampling
Before any algorithm touches an image, the image itself has already made two irreversible decisions: where it looked, and how finely it counted.
1.1 An image as a function
An image is just a big table of numbers. Each number is "how bright is it here?" That brightness actually mixes two separate things — how much light hit that spot, and how much of it bounced back to the camera — and mixing those up is where a lot of registration headaches start.
Physically, an image begins as light — radiant energy — arriving from a scene and striking a sensor. We describe it mathematically as a function of two continuous spatial variables:
Forget the notation for a second. f(x, y) just means: "walk to row y, column
x on the image, and tell me the brightness number written there." That's it — the whole
equation is a fancy way of saying "look up a value in a grid." Everything later in this reference —
blurring, sharpening, edge detection, matching one image to another — is just different ways of
combining these looked-up numbers.
For an ordinary photograph, f(x, y) is really the product of two physically separate things:
how much light was available to illuminate the scene at that point, and how much of that light
the surface at that point reflected back toward the sensor.
where 0 < i(x,y) < ∞ (illumination) and 0 < r(x,y) < 1 (reflectance)
Think of a black car and a white car parked outside at noon, then again at dusk. The white car is
always brighter than the black car in a photo — that ranking never flips, because reflectance
(r) is a property of the paint and doesn't change. But the same white car looks
much brighter at noon than at dusk, because illumination (i) dropped. A photo can't tell
you, from a single pixel value, how much of that brightness was "white paint" versus "bright sun" —
it only gives you the product. Separating the two back out is exactly the kind of problem
illumination-invariant processing tries to solve.
This decomposition matters more than it looks. A shadow falling across a scene changes i, not
r — the material didn't change, only the light hitting it did. Most illumination-invariant
processing (and a good deal of the trouble with matching optical images taken under different sun angles)
traces back to disentangling these two factors. LiDAR intensity images sidestep the illumination term
almost entirely, since the sensor supplies its own light — which is exactly why they're attractive as a
matching reference against passively-lit optical imagery, and exactly why they still don't look like a
photograph.
A DEM encodes elevation, not reflectance or illumination at all — it's not an image of light, it's a
sampled height field. An optical orthoimage encodes i·r under whatever sun angle it was
captured. A LiDAR intensity return encodes something closer to pure r, illuminated by the
sensor's own laser. Three different physical quantities, all rendered as grayscale arrays. Registration
between them isn't really "image matching" in the ordinary sense — it's matching three different
physical signals that happen to share spatial structure (edges, ridgelines, boundaries) even though
their intensities mean completely different things.
1.2 Sampling: from continuous to discrete
A sensor can't measure light at infinitely many points — it takes a fixed grid of readings. Too coarse a grid, and fine patterns don't just blur away, they turn into a completely different (wrong) pattern. This is why "megapixels" and "ground sample distance" are specs people care about.
A camera sensor, or a LiDAR scanner sweeping a footprint, doesn't measure f(x, y) everywhere —
it measures it at a finite grid of positions. This is spatial sampling, and the spacing
of that grid is the single most consequential decision in the entire imaging pipeline: it sets a hard
ceiling on the finest detail the image can ever represent, and nothing downstream can put back what
sampling threw away.
A 12-megapixel phone photo is a grid of about 4000×3000 sample points. Point that camera at a fine fabric pattern (a shirt with tight pinstripes) from far away, and instead of stripes you sometimes see wavy, colorful ripples in the photo that aren't in the real shirt at all — that's moiré, and it's aliasing happening in a $1000 sensor for the same mathematical reason it happens in the demo below. It's not a defect in the phone; it's an unavoidable consequence of any grid of finite samples meeting a pattern finer than that grid can resolve.
Drag the sampling rate below and watch a continuous test pattern collapse into pixels. Below a certain spacing, fine stripes stop looking like stripes and start looking like something else entirely — a completely different, lower frequency. That reappearance of the wrong pattern is aliasing, and it's the reason satellite and airborne sensors are specified partly by their ground sample distance, not just their optics.
The Nyquist condition
The formal statement of when sampling is "enough" is the Nyquist–Shannon sampling theorem: a signal can be perfectly reconstructed from its samples only if the sample rate is at least twice the highest frequency present in the signal.
Say a repeating pattern on the ground has stripes every 2 meters (so it repeats twice every 4 meters — that's its "frequency"). To capture that pattern faithfully, your sensor needs to take at least one sample every 1 meter — twice per stripe-cycle, matching the "≥ 2×" in the formula. Sample only every 3 meters instead, and you're measuring less than once per stripe cycle — the readings you get back are consistent with a completely different, much wider, fake stripe pattern that isn't really there. The sensor isn't broken; it mathematically cannot tell the two patterns apart at that sample spacing.
Below this rate, high frequencies don't just get lost — they get misidentified as lower frequencies that weren't there. This is why anti-aliasing filters (optical blur, or a low-pass filter applied before downsampling) exist: it's better to lose fine detail cleanly than to have it reappear as false structure. For a LiDAR–DEM–optical registration pipeline, this is worth remembering whenever you downsample a high-resolution DEM or optical mosaic to match a coarser LiDAR grid — naive decimation without pre-blurring will inject aliasing artifacts that a feature matcher can mistake for real terrain structure.
1.3 Quantization: from continuous to discrete, again
Sampling decides where you measure. Quantization decides how many possible answers you're allowed to write down at each spot. Real brightness is a smooth continuum; a digital image forces every reading into one of a fixed number of "buckets" (256 for a normal photo).
Sampling discretizes where you measure. Quantization discretizes what value you record. An 8-bit grayscale image doesn't store real numbers — it rounds every measured intensity to one of 256 levels. Fewer levels means less storage and less computation, but coarser levels means visible banding in smooth gradients, and — more importantly for feature matching — a loss of the fine intensity differences that many descriptors rely on to distinguish nearby keypoints.
Old video game consoles and early digital cameras sometimes show visible color "bands" in a smooth sunset sky instead of a perfectly smooth gradient — you can see distinct rings of color rather than an effortless blend. That's quantization banding: too few brightness buckets were available to represent a smooth real-world gradient, so the software had to round each shade to the nearest bucket, and the rounding boundaries became visible as bands. More bits per pixel means more buckets, means smoother gradients — at the direct cost of more storage per pixel.
1.4 The digital image as a matrix
Once sampled and quantized, a "digital image" is nothing mystical — it's a spreadsheet of numbers, one per pixel. Every fancy-sounding operation later in this reference is arithmetic performed on that spreadsheet.
Once sampled and quantized, an image is simply a 2D array — a matrix f(x, y) where
x and y are now integers indexing rows and columns, and the value at each
position is one of a finite set of intensity levels. Every operation in this entire reference — filtering,
transforms, morphology, feature detection — is ultimately just arithmetic on this array.
Hover over the image below. The readout tag follows your cursor and reports the exact row, column, and intensity value under it — the same three numbers that are, together, the entire content of a digital image at that point. This probe will reappear throughout the site: whenever you see an image in a demo, you can interrogate it pixel by pixel.
1.5 Neighborhoods and connectivity
To decide if two pixels are "part of the same blob," you need a rule for which pixels count as touching. Do diagonal neighbors count as touching, or only up/down/left/right? Both rules are used, and picking the wrong one silently changes your results.
A single pixel value is rarely useful alone — almost every meaningful operation looks at a pixel together with its neighbors. The two standard neighborhood definitions:
- 4-connectivity — the pixel directly above, below, left, and right (the "plus sign" neighbors).
- 8-connectivity — adds the four diagonal neighbors, giving a full 3×3 neighborhood.
Picture a checkerboard pattern of single dark pixels touching only at their corners, like two black squares meeting diagonally on a chessboard. Under 4-connectivity, those two dark pixels are considered separate blobs — they don't share an edge. Under 8-connectivity, they're considered the same blob, because diagonal touching counts. Neither answer is "wrong" — they're different rules for different purposes — but a crater-rim tracing algorithm that assumes 8-connectivity when the rest of the pipeline assumes 4-connectivity will disagree with itself about how many separate craters it just found.
This distinction seems pedantic until you implement connected-component labeling or boundary tracing (Chapters 9–10): a diagonal-only connection between two regions is 8-connected but not 4-connected, and algorithms that assume the wrong one will either fuse regions that should stay separate or split one region into two. It resurfaces again in morphological processing (Chapter 6), where the structuring element implicitly chooses a connectivity.
1.6 Imaging modalities and what they actually measure
Not all "grayscale images" measure the same physical thing. Mixing up "brightness because of elevation" with "brightness because of reflected sunlight" is a common, subtle source of registration errors in multimodal terrain-matching pipelines.
It's worth being explicit about what "intensity" means across the sensors relevant to LiDAR–DEM–optical registration, since conflating them is a common source of registration errors:
| Modality | What the value at (x, y) represents | Illumination-dependent? |
|---|---|---|
| Optical / orthoimage | Reflected light, integrated over a spectral band | Yes — varies with sun angle, time of day, atmosphere |
| LiDAR intensity | Strength of the laser return (surface reflectance at the laser wavelength) | No — sensor is its own light source |
| DEM | Elevation (a height, not a light measurement at all) | No — but slope maps derived from it are used as an illumination-like proxy |
This is precisely why direct pixel-intensity matching (correlation, SSD, SIFT-style gradient descriptors applied naively) between an optical image and a LiDAR intensity image performs worse than expected — the images agree on where structure is but disagree on what intensity that structure produces. Chapter 13 covers this in depth.