Computer Vision Basics

Computer vision is the branch of AI concerned with getting a computer to make sense of images — and just like language, it starts with converting something humans perceive effortlessly into raw numbers a model can actually process.

An image is just numbers

To a computer, a color photo is a grid of pixels, and each pixel is a small set of numbers — typically three, for how much red, green, and blue light it contains, each on a scale like 0 to 255. A modest 224×224 pixel color image is therefore an array of roughly 150,000 numbers before any processing happens at all. There's no inherent concept of "cat" or "edge" or "face" anywhere in that array — every bit of understanding a vision model has is entirely learned from patterns across that raw grid of numbers.

What a convolution does, intuitively

The technique that made modern computer vision practical is the convolution: instead of connecting every single pixel to every neuron (which would be an enormous number of parameters for a large image), a small filter — maybe just a 3×3 grid of numbers — slides across the image, computing a small weighted sum at each position. A filter can learn to detect something simple and local, like a vertical edge or a patch of a particular color, and produces a high value everywhere that pattern shows up in the image and a low value everywhere it doesn't. Stack many such filters in layers, the same "stack layers to build complexity" idea from the neural networks lesson, and early layers end up detecting simple things like edges and colors, while deeper layers combine those into increasingly complex patterns — textures, shapes, and eventually whole objects.

Classic vision tasks

  • Classification — what is the single main subject of this image? ("cat," "dog," "car")
  • Object detection — where in the image is each object, drawn as a bounding box, and what is each one?
  • Segmentation — which exact pixels belong to each object, traced out precisely rather than boxed?

These build on each other in difficulty: classification just needs one label for the whole image, detection needs to locate multiple objects, and segmentation needs pixel-level precision for each one.

Vision models fail on what they haven't seen: a model trained mostly on daytime, well-lit, front-facing photos can perform dramatically worse on night photos, unusual angles, or situations underrepresented in its training data — a direct consequence of the "learns patterns from data" nature of these systems, from the machine learning basics lesson. There's no built-in general understanding of "object" filling that gap; there's only whatever the training data happened to cover, which is exactly why real-world deployment of vision systems (self-driving cars, medical imaging) requires enormous, carefully curated, and diverse training data to be trustworthy.
PreviousNatural Language Processing Basics