The Neural Code of the Retina

A local group of four cells from different functional types was found to transmit information about the visual stimulus at approximately 8 bits/s, which can be interpreted as resolving 28 different visual features every second and provides at least one answer to the question posed at the outset of this review.

Abstract

Action potentials are the standard signal conveyed between neurons in the central nervous system. It is a long-standing question how these spikes represent sensory input, internal states of the brain, or motor commands (83Perkel D.H. Bullock T.H. Neural coding.Neurosci. Res. Program Bull. 1968; 6: 221-348Google Scholar, 87Rieke F. Warland D. de Ruyter van Steveninck R.R. Bialek W. Spikes. MIT Press, Cambridge, MA1997Google Scholar). To fully understand communication among neurons, one would like to obtain a dictionary for this language, in which each spike or pattern of spikes is assigned meaning within the processing task under study. This review will focus on the neural code employed by the ganglion cells of the vertebrate retina in conveying visual information from the eye to the brain: what are the rules by which the spike trains of optic nerve fibers encode the visual scene? Communication between the retina and the brain is particularly amenable to experimental analysis for several reasons. First, we know exactly what is being represented by these action potentials, namely the time-dependent visual image as projected by the optics of the eye. Second, one can readily stimulate the retina with its natural sensory input, making use of well developed technology for presenting movies. Similarly, the output of the retina can be monitored with relative ease by extracellular recording from ganglion cells, optic nerve fibers, or terminals in the lateral geniculate nucleus. Finally, our retina performs a significant amount of processing, which compresses the visual signal from a neural population of 108 photoreceptors into just 106 optic nerve fibers. Nowhere else in the visual system is the scene represented with as few neurons as in the optic nerve, and thus one might expect to discover interesting principles of efficient coding. Since the electrical spikes on ganglion cell axons are the only source of our visual experience, there is considerable interest in the power of retinal processing and how it shapes our visual perception. Our understanding of retinal coding has come a long way since the pioneering recordings from retinal ganglion cells by 63Kuffler S.W. Discharge patterns and functional organization of mammalian retina.J. Neurophysiol. 1953; 16: 37-68PubMed Google Scholar and 13Barlow H.B. Summation and inhibition in the frog's retina.J. Physiol. 1953; 119: 69-88PubMed Google Scholar. Sadly, even recent Neuroscience textbooks limit themselves to a qualitative treatment that does not reflect these advances. One goal of this review is to illustrate how the relationship between visual images and optic nerve firings can be captured quantitatively, to a degree that comes close to the ideal of the "dictionary" mentioned above. We then highlight some recent observations that have drawn attention to novel aspects of retinal processing. What should a useful description of the retinal code contain? One way to find out is to try and answer a simple question, such as: "How many levels of gray can the retina distinguish?" This problem is taken from real life: it was posed to us by a colleague who expected a quick answer. It is also of some practical relevance, given that the price of home video systems is strongly related to how many gray levels they will produce. To tackle it experimentally, we imagine taking the following approach: record the spike train of a retinal ganglion cell, project a uniform gray field on the retina, vary the intensity of the light in small step increments, and ask how small the steps can be to still cause a recognizable difference in neural firing. We soon find that after an intensity step the ganglion cell fires a brief burst of spikes, then settles down within a few seconds to whatever it was doing before the step. Something similar happens for almost every other ganglion cell, so we conclude that steady gray levels are almost indistinguishable. To get a meaningful answer, we therefore decide to vary the light intensity in time, for example switching back and forth between two levels. As we switch faster and faster, we find that the neural response eventually disappears again at very high frequencies. So the initial question regarding resolution of gray levels cannot be answered without specifying the time course of the light intensity, since there is no response for either very slow or very fast changes. Similarly, we soon find that the spatial distribution of intensity on the retina is very important. For example, when we illuminate only a small spot overlying the recorded ganglion cell, we find that the response is stronger than when we illuminate the entire field uniformly. On the other hand, the spot cannot be too small, or the response disappears again. Next, we need to decide what it means to "distinguish" two gray levels. Presumably, someone monitoring the ganglion cell's spike train should be able to identify which gray level is being presented with some degree of confidence. We find that even when presenting the same stimulus repeatedly, the ganglion cell produces somewhat different spike trains, and this variability ultimately limits the ability to discriminate two different stimuli. So, in seeking an answer to the initial question, the variability of the neural response is equally important as the response itself. To quantitate any of this, we need to decide what aspects of the spike train to measure as the "response." Is it sufficient to simply count the number of spikes in some suitable time window, or should we note the exact time of arrival of every spike? Furthermore, we must consider that many ganglion cells are being affected by this stimulus and can therefore contribute to its identification. So, we should try to understand how the responses of different neurons interact. Finally, toward the end of the day, we discover a vexing feature of this experiment: if we gradually step the light intensity upward, we get one answer, and if we gradually step it downward, we get another. Apparently, the relationship between gray level and firing is not permanent and static but varies considerably depending on the recent history of the visual stimulus. This thought experiment illustrates that to answer a seemingly straightforward question about retinal signaling, one needs to know many different facets of how the visual information is encoded. In particular, any useful description of the neural code should specify: (1) the relevant measure of neural activity in the ganglion cell population; (2) how this activity responds to any given visual stimulus; (3) the precision of this response; and (4) the degree of plasticity in this relationship between stimulus and response, specifically, how it varies depending on recent history of the visual input. Of course, similar issues arise in every study of neural communication, whether it regards sensory encoding, signaling among neural populations, or motor control. In this section, we attempt to summarize a consensus notion of how the retina encodes visual stimuli, delineating the four essential components of the neural code introduced above. This understanding has been gained from experiments on a wide range of species, and a comment is in order on how these should be integrated. Different animals clearly employ their visual system for different tasks, and this is reflected in the anatomical and functional diversity of their visual pathways (112Stone J. Parallel Processing in the Visual System. Plenum Press, New York1983Crossref Google Scholar). However, while the postretinal anatomy differs significantly across vertebrates, the structure of the retina is remarkably conserved from fish to primates. One finds the same three-layered arrangement, the same five principal cell types, the same neurotransmitters employed, and in many cases, the same anatomical microcircuitry (46Dowling J.E. The Retina. Harvard University Press, Cambridge, MA1987Google Scholar). A plausible explanation is that the retina is adapted to deal with constraints that are shared among all species: the statistics of visual images from the natural world at one end, and the limited capacity of the optic nerve at the other end. At any rate, many principles of retinal signaling seem to be remarkably conserved. The basic aspects of spatiotemporal processing, light adaptation, contrast gain control, and stochastic variation of the response are documented in animals ranging from tiger salamander to macaque monkey. Models of the light response that successfully predict a ganglion cell's firing rate share a common structure in all these cases. Differences among species affect the quantitative parameters of these models, but not their basic elements. Although the visual scene is conveyed to the brain in parallel by the spike trains of all optic nerve fibers, most of what we know about retinal signaling is derived from recordings of single retinal ganglion cells, one at a time. The same is true, of course, for neural signaling everywhere else in the nervous system. The underlying and often unstated assumption is that such a population code can, in fact, be understood one cell at a time. Two conditions are necessary for this. First, there should be identifiable classes of cells, that group neurons of similar functional properties, such that studying one or a few cells of a given class allows one to estimate the behavior of other cells in this class. If, instead, the population were perfectly heterogeneous, the code could only be understood after observing every neuron. A great deal of effort has gone into sorting retinal ganglion cells into different types based on their visual responses. There is clear evidence for distinct classes—some of which will be discussed below—though their precise number and boundaries are often in dispute (for review see 112Stone J. Parallel Processing in the Visual System. Plenum Press, New York1983Crossref Google Scholar, 91Rodieck R.W. Brening R.K. Watanabe M. The origin of parallel visual pathways.in: Shapley R. Lam D.M.-K. Contrast Sensitivity. MIT Press, Cambridge, MA1993Google Scholar, 89Rodieck R.W. The first steps in seeing. Sinauer Associates, Sunderland, MA1998Google Scholar). Second, the firing of each neuron in the population should depend only on the stimulus, not on the activity of other neurons in the population. If this is so, then the description derived from many single-cell recordings can adequately predict the occurrence of any given response pattern in the population. Unfortunately, this latter condition is not always met, as demonstrated in recent multineuron recordings discussed below. Nevertheless, the classical single-neuron analysis of retinal responses has been highly successful and valuable and is still the focus of much research. Within the spike train of a single ganglion cell, the important response feature is generally taken to be the neuron's instantaneous firing probability at various times throughout the stimulus presentation. In experiments, this firing rate is estimated by repeating the same stimulus many times and counting spikes in the corresponding time bin of many such trials. Of course, during natural vision, we do not enjoy the luxury of many identical stimulus trials. It is often assumed that the brain, instead, estimates this response function by counting spikes from many essentially identical ganglion cells (68Levick W.R. Zacks J.L. Responses of cat retinal ganglion cells to brief flashes of light.J. Physiol. 1970; 206: 677-700PubMed Google Scholar, 51Enroth-Cugell C. Robson J.G. Schweitzer-Tong D.E. Watson A.B. Spatio-temporal interactions in cat retinal ganglion cells showing linear spatial summation.J. Physiol. 1983; 341: 279-307PubMed Google Scholar). This idea conflicts with another commonly held notion, namely that ganglion cells of any given functional type "tile" the retina, such that each point is serviced by just one neuron of that type (125Wässle H. Boycott B.B. Functional architecture of the mammalian retina.Physiol. Rev. 1991; 71: 447-480PubMed Google Scholar, 39DeVries S.H. Baylor D.A. Mosaic arrangement of ganglion cell receptive fields in rabbit retina.J. Neurophysiol. 1997; 78: 2048-2060PubMed Google Scholar). We will revisit this topic below when considering distributed coding by retinal ganglion cells. With these assumptions, the central problem of the retinal code is how a ganglion cell's firing rate depends on visual stimulation. Early experiments explored this relationship with simple stimuli, such as a small spot flashed on a uniform background (13Barlow H.B. Summation and inhibition in the frog's retina.J. Physiol. 1953; 119: 69-88PubMed Google Scholar, 63Kuffler S.W. Discharge patterns and functional organization of mammalian retina.J. Neurophysiol. 1953; 16: 37-68PubMed Google Scholar). Generally, the spot altered the firing rate only if presented within a small region on the retina—termed the "receptive field"—a few tens to hundreds of micrometers in diameter surrounding the cell body. The nature of the light response within the receptive field immediately pointed to the existence of very different cell types. In some ganglion cells, a spot flashed near the center of the receptive field produced a transient increase of firing at light onset and a brief reduction of firing at offset (ON cells). In other ganglion cells, the firing rate decreased at onset and increased at offset (OFF cells). For either cell type, a spot placed at some distance from the center—in the so-called receptive field surround—had the opposite effect of a spot in the center. When center and surround were illuminated simultaneously, the center response was significantly suppressed. Still other ganglion cells responded with a brief burst of spikes at both onset and offset, no matter where the spot was flashed in the receptive field (ON/OFF cells). Two important aspects of retinal processing are already recognizable in this early work: lateral inhibition in space and differentiation in time. Because of the antagonistic action of the center and surround regions of the receptive field, ganglion cells respond strongly to stimuli whose intensity varies in space over the receptive field, such that center and surround are illuminated differently. And because the response to a light step lasts only a short time—typically tens of milliseconds to seconds—many ganglion cells seem to emphasize stimuli that change in time over static ones. The visual world, of course, does not consist of spots and annuli. Thus, one needs to cast the stimulus–response relationship in a quantitative form that generalizes to arbitrary patterns of visual input. In its most general form, the stimulus is given by the intensity distribution I(x, t, λ) on the retina, as a function of position x, time t, and wavelength λ. Under the above assumptions, the response of the retina consists of the firing rates Ri(t) of each of its ganglion cells. To capture retinal processing, one thus seeks a mathematical function whose input is the stimulus I(x, t, λ) and whose output is the time course of a ganglion cell's firing rate R(t). This function will have a number of free parameters, which are optimized based on the measured responses to experimental stimuli. Finally, one can test the performance of this model with other types of stimuli. The following sections will illustrate some examples of this powerful approach. Spatiotemporal Integration. 88Rodieck R.W. Quantitative analysis of cat retinal ganglion cell response to visual stimuli.Vision Res. 1965; 5: 583-601Crossref PubMed Scopus (520) Google Scholar made an early and influential attempt at a quantitative description of cat ganglion cell responses. As observed earlier, a small spot of light flashed briefly on the receptive field center of an ON cell produced a brief increase in firing followed by an undershoot and gradual recovery of the baseline firing rate. The shape of this time course was approximated as 1 where δ(t) denotes the delta function pulse of firing and h is the size of the subsequent undershoot, which decays with time constant τ. As in previous experiments, the amplitude of this response depended on the location of the spot: large and positive in the center, small and negative in the surround, and zero somewhere in between. This spatial profile of the response amplitude was formalized as a "difference of Gaussians" 2 where kc and ks are the amplitude of the center and the surround Gaussians and rc and rs are their respective radii. Thus, the change in the firing rate produced by flashing a spot at time t = 0 and location x is A(t)·B(x). Now any given light intensity pattern, I(x, t), such as a white bar moved across the retina, can be decomposed into many small flashed spots. Rodieck's model assumed that the effects of all these spots simply sum up. Thus, the firing rate, R(t), produced by the visual stimulus becomes 3 where R0 is the cell's maintained firing rate without stimulation. This expression can also be viewed as a cascade of a few simple transformations of the stimulus (Figure 1A). First, the stimulus I(x, t) is summed over all space, with the weighting function B(x). Then the resulting signal is passed through a filter with impulse response A(t). The result is added to the baseline firing rate R0, and negative values R0 of the resulting firing rate R(t) are truncated to zero. After Rodieck, 1965; Rodieck and Stone, 1965. (A) At every time point, the intensity pattern on the retina is integrated over space with a weighting function that represents the receptive field profile (top, thick line). This profile is shaped as the difference of two concentric Gaussian surfaces (thin lines, see Fd 2), here shown in a one-dimensional section through the center. The resulting signal is convolved in time with the retina's flash response (middle): a delta function (approximated in the graphic by a brief square pulse) followed by an exponential undershoot (Fd 1). The result is added to a maintained firing rate and truncated (Fd 3) to eliminate negative values (bottom). The signal processing cascades described in this and subsequent figures contain three types of elements, and we will use the following graphic conventions to represent them: weighted spatial summation of light intensity is shown by the profile of the weighting function, with the horizontal axis labeled "Space" (top box in [A]); temporal filtering is represented by the impulse response of the filter, with the horizontal axis labeled "Time" (middle box in [A]); instantaneous transform of a signal is represented by a graph of output versus input, with the axes unlabeled (bottom box in [A]). (B) Firing rate of an ON-type cat retinal ganglion cell in response to a bar swept across its receptive field (left; adapted from Rodieck and Stone 1965), and as predicted by the model in (A) (right; adapted from Rodieck 1965). The bars were either white or black on a gray background, varied in width from 0.5° to 5° (center), and were moved steadily at 10°/s. (C) "Modified difference-of-Gaussians" model of the light response, in which center and surround are treated as separate pathways. Their Gaussian sensitivity curves may not be concentric, and their flash responses may differ. The resulting signals from both pathways are summed and rectified to produce the firing rate. The parameters in this model, namely R0, A(t), and B(x), were derived from the flashing spot measurements. Then the model was tested using very different stimuli, consisting of various shapes moving steadily across the cell's receptive field. As Figure 1B shows, there was a remarkable correspondence between the observed time course of the firing rate and the predictions of the model. This model of the light response is very attractive in its simplicity. For example, the time course of the response to a flash is identical no matter where in the receptive field the flash is presented, except for a scaling factor. This is termed "space–time separability" (123Wandell B.A. Foundations of Vision. Sinauer, Sunderland, MA1995Google Scholar), because the weighting function in Fd 3 separates into a term depending only on time multiplied by a term depending only on space. Subsequent work showed that space-time separability is not quite satisfied in ganglion cell responses: for example, the response to light falling in the surround is delayed relative to the response in the center, owing to the time required for lateral signal flow through horizontal or amacrine cells, and transmission across an additional synapse (47Enroth-Cugell C. Freeman A.W. The receptive-field spatial structure of cat retinal Y cells.J. Physiol. 1987; 384: 49-79PubMed Google Scholar, 97Sakai H.M. Naka K. Response dynamics and receptive-field organization of catfish ganglion cells.J. Gen. Physiol. 1995; 105: 795-814Crossref PubMed Scopus (7) Google Scholar, 22Benardete E.A. Kaplan E. The receptive field of the primate P retinal ganglion cell, I Linear dynamics.Vis. Neurosci. 1997; 14 (a): 169-185Crossref PubMed Scopus (79) Google Scholar). This led to a simple "modified difference-of-Gaussians model" (Figure 1C), in which light is pooled separately within the center and the surround; the two resulting signals are passed through two different filters, then summed to generate the firing rate (51Enroth-Cugell C. Robson J.G. Schweitzer-Tong D.E. Watson A.B. Spatio-temporal interactions in cat retinal ganglion cells showing linear spatial summation.J. Physiol. 1983; 341: 279-307PubMed Google Scholar, 36Dawis S. Shapley R. Kaplan E. Tranchina D. The receptive field organization of X-cells in the cat spatiotemporal coupling and asymmetry.Vision Res. 1984; 24: 549-564Crossref PubMed Scopus (103) Google Scholar). For some ganglion cell types, center and surround also have a different spectral sensitivity, because they are fed by a different mix of photoreceptors. In general, the wavelength dependence of the retinal response is governed by the spectral sensitivities of the rods and cones, an aspect that varies a great deal among species. We will not elaborate on this topic here, but refer the reader to recent reviews of color processing (123Wandell B.A. Foundations of Vision. Sinauer, Sunderland, MA1995Google Scholar, 67Lee B.B. Receptive field structure in the primate retina.Vision Res. 1996; 36: 631-644Crossref PubMed Scopus (164) Google Scholar). Another fundamental feature of Rodieck's model is the linearity of its response. Twice the intensity fluctuation will produce twice the firing rate fluctuation; more generally, the response to the sum of two intensity patterns is the sum of their individual responses, barring truncation in the final step of spike generation. Subsequently, it was found that a linear relationship between stimulus and firing rate holds only for some retinal ganglion cells and only under restricted conditions: the modulations of the light intensity must be small compared to the mean, and the range of these modulations must not change very much over time (48Enroth-Cugell C. Robson J.G. The contrast sensitivity of retinal ganglion cells of the cat.J. Physiol. 1966; 187: 517-552PubMed Google Scholar, 120Victor J.D. The dynamics of the cat retinal X cell centre.J. Physiol. 1987; 386: 219-246PubMed Google Scholar, 23Benardete E.A. Kaplan E. The receptive field of the primate P retinal ganglion cell, II nonlinear dynamics.Vis. Neurosci. 1997; 14 (b): 187-205Crossref PubMed Scopus (36) Google Scholar). These are very narrow constraints, and it now appears that under stimulus conditions resembling those of our natural visual experience, a linear description of retinal ganglion cell responses is of rather limited use. Nonlinear Processing. 120Victor J.D. The dynamics of the cat retinal X cell centre.J. Physiol. 1987; 386: 219-246PubMed Google Scholar has captured some of the nonlinear behavior of cat ganglion cells in a very successful model (Figure 2A): as in Rodieck's scheme, the light distribution is pooled linearly with a spatial weighting function and the result is passed through a temporal filter. However, the properties of this filter depend on its output. In particular, when the output is large—of either sign—the gain of the filter decreases and its waveform sharpens. With only a few parameters, this model accurately predicted the response to a variety of stimulus waveforms (Figure 2B), whereas any purely linear model produced large discrepancies. The net effect of this "contrast gain control" (106Shapley R.M. Victor J.D. Nonlinear spatial summation and the contrast gain control of cat retinal ganglion cells.J. Physiol. 1979; 290: 141-161PubMed Google Scholar, 107Shapley R.M. Victor J.D. How the contrast gain control modifies the frequency responses of cat retinal ganglion cells.J. Physiol. 1981; 318: 161-179PubMed Google Scholar) is that during large light fluctuations the retinal response is less sensitive and faster. This adjustment is very rapid: the time constant τc in Figure 2B was 15 ms, but a value of zero produced indistinguishable results. After Victor, 1987. (A) Only the pathway for the receptive field center is shown; see Figure 1A legend for conventions of this graphic shorthand. Light is integrated over the center's spatial profile. The result is passed through a band-pass temporal filter, then truncated to form the firing rate. To implement the contrast gain control, the output of the filter is full-wave rectified and averaged by a low-pass stage with time constant τc. The resulting signal, c(t), is a neural measure of contrast and modifies the temporal processing properties: the flash response of the band-pass filter is more biphasic at high values of c(t) (thick line) than at low values (thin line). (B) Response of an ON-type X cell to contrast reversal (bottom trace) of a 1 cycle/degree sinusoidal grating at different modulation depths, C. Top plots show the measured firing rate (jagged line) and the prediction from the model in (A) (smooth line). Bottom plots show the neural measure of contrast, c(t), that modulates the band-pass filter in (A). Note that the time course of the firing rate is more transient at large modulation depth, and this modification is reproduced accurately by the model. This type of quantitative analysis has revealed that there are two very distinct types of ganglion cells in the cat retina. The so-called "X cells" (Figure 1 and Figure 2) appear to integrate light from different points in space by simple weighted summation, while for the "Y cells" this is not the case (48Enroth-Cugell C. Robson J.G. The contrast sensitivity of retinal ganglion cells of the cat.J. Physiol. 1966; 187: 517-552PubMed Google Scholar, 56Hochstein S. Shapley R.M. Linear and nonlinear spatial subunits in Y cat retinal ganglion cells.J. Physiol. 1976; 262: 265-284PubMed Google Scholar). The Y cell's receptive field is several times larger than that of a nearby X cell. It is composed of many small spatial "subunits" that appear to process the stimulus independently (122Victor J.D. Shapley R.M. The nonlinear pathway of Y ganglion cells in the cat retina.J. Gen. Physiol. 1979; 74: 671-689Crossref PubMed Scopus (122) Google Scholar). Within the area of each subunit, the light intensity is integrated, again with an antagonistic receptive field and a biphasic impulse response (Figure 3). The result gets rectified, a highly nonlinear operation, and added to the output from all other subunits. This sum, after passing through another filter, specifies the firing rate (121Victor J.D. The dynamics of the cat retinal Y cell subunit.J. Physiol. 1988; 405: 289-320PubMed Google Scholar). There may be as many as 100 such "nonlinear subunits" (122Victor J.D. Shapley R.M. The nonlinear pathway of Y ganglion cells in the cat retina.J. Gen. Physiol. 1979; 74: 671-689Crossref PubMed Scopus (122) Google Scholar). Due to their rectifying nature, a flashing spot anywhere within the receptive field can produce a burst of spikes at both onset and offset. Thus, the Y-type ganglion cell cannot signal the position of a small spot on the retina with the spatial resolution of an X cell. On the other hand, it is very sensitive to a fine textured pattern moving across the receptive field, since that induces intensity fluctuations for all the local subunits. The anatomical identity of the subunits is still uncertain. Their size is about equal to that of X cell centers, and they have been proposed to correspond to bipolar cells, with the rectification occurring in transmission to amacrine cells (122Victor J.D. Shapley R.M. The nonlinear pathway of Y ganglion cells in the cat retina.J. Gen. Physiol. 1979; 74: 671-689Crossref PubMed Scopus (122) Google Scholar). After Hochstein and Shapley, 1976; Victor and Shapley, 1979; see Figure 1A legend for conventions of this graphic shorthand. This pathway initiates in small subunits of the receptive field (thick profiles in top panel). Within each subunit's receptive field, light is integrated, the result is passed through a band-pass filter and then full-wave rectified. The rectified outputs from all subunits are pooled, passed through another linear filter, and converted to the firing rate. As for the X cell (Figure 2A), a contrast gain control modulates the temporal filter within each subunit (gray feedback pathway). The relevant contrast measure c(t) is probably derived from the output of each subunit's rectifier and controls the contrast gain for both Y cells and X cells (Shapley and Victor 1981). Note that the Y cell circuitry includes an additional signaling pathway, not elaborated in this figure, that produces a classical center and antagonistic surround (thin profiles in top panel). This pathway can be isolated by suitable visual stimuli (122Victor J.D. Shapley R.M. The nonlinear pathway of Y ganglion cells in the