Given a single image, and a point on this image, the only thing which can be infered geometrically is a ray of projection, but not the position along this ray. The relationship between the point and the ray is best described by projective geometry, a generalization of the familiar Euclidean geometry, which can deal with projections and objects at infinity, but lacks the notions of angles or distances.
Since it is impossible to geometrically infer 3D information from a single image, let now consider a pair of images, for instance the one formed by each of your eyes. If we pick two points at random in each image, they are unlikely to be corresponding points, ie, the images of a single 3D point of the world. So do corresponding points have any special properties ? It turns out that there is a remarkably simple answer to this question: their coordinates - in any arbitrary frame attached to each image - have to satisfy a bilinear relationship whose coefficients are derived from a particular 3 x 3 matrix, that I called the Fundamental matrix, because it encodes the only geometrical information which can be generally computed from pairs of images. This information is projective in nature.
The seven parameters of this matrix can also be interpreted in terms of epipolar geometry. Its computation requires a minimum of eight point correspondences to obtain a unique solution, and in particular numerical methods were investigated to improve the robustness of the computation. The homography matrix describes completely the relationship between corresponding planar points. We investigated the deep relationship between the Fundamental matrix and that matrix, and also with the so-called critical surfaces, which have consequences for computation stability.
Those observations can be generalized when more than two views are available, and we obtain more geometric and algebraic constraints, but still have access only to the projective geometry of the scene. So how is it possible to recover Euclidean information about the world, which matches our experience ?
Previous methods relied on knowledge of the world (surveyed landmarks, artificial calibration objects whose 3D coordinates are known), or on controled motions, which is often impractical. In contrast, we rely on the fact that over multiple views some of the internal characteristics of the cameras remain constant. For instance, if all the internal parameters remains the same, we need only three views to recover them. This is the idea of Self-Calibration. The calibration object is replaced by an entity which has the advantage of being always available for free: the absolute conic. However, this entity is quite abstract (being imaginary and situated at infinity), and to exploit it, we must go through a system of polynomial Kruppa equations, which is quite delicate to solve robustly.
These ideas, initiated in my PhD thesis, have since then received a great deal of attention and helped start a rich subfield of research which has led us within a decade to a more or less complete understanding of the geometry of multiple views. The book "The geometry of multiple images", co-written with my advisor O. Faugeras, describes in a principled and comprehensive way all that.