Sparse Coding in the Neocortex

It is argued that larger brains have evolved strategies that allow neurons to be both highly selective and invariant across irrelevant dimensions, and this invariance is what allows such highly sparse codes to be useful.

Abstract

Sparse coding has been proposed as a guiding principle in neural representations of sensory input, particularly in the visual system. Because sparse codes are defined as representations with low activity ratios – i.e., at any given time a small proportion of neurons active – they are sometimes proposed as a means to help conserve metabolic costs. Although we accept that such metabolic costs play a role in evolutionary selection, we also argue that sparse coding offers a useful solution to the problem of representing natural data because such a scheme allows the system to take advantage of the sparse structure of the sensory environment. Given the highly regular and sparse structure of natural data (e.g., natural scenes), a variety of recent studies suggest that the sparse coding in the cortex is an adaptation that has evolved to offer an efficient strategy for storing and building associations with natural inputs. However, highly selective sparse codes require a large number of neurons and can also pose problems for learning. We argue that larger brains have evolved strategies that allow neurons to be both highly selective and invariant across irrelevant dimensions. We argue that this invariance is what allows such highly sparse codes to be useful.