multisensory processing

Multisensory processing in primates, like information processing on the Internet, is extensively decentralized. A recent neuroscience review article declared:

The pervasiveness of multisensory influences on all levels of cortical processing compels us to reconsider thinking about neural processing in unisensory terms. Indeed, the multisensory nature of most, possibly all, of the neocortex forces us to abandon the notion that senses ever operate independently during real-world cognition.[1]

Within the ascending auditory pathway prior to the neocortex, neurons in the inferior colliculus respond to auditory signals. Recent research found that about two-thirds of neurons in the inferior colliculus also respond to visual signals. In addition, about 20% of the neurons fire with eye movement motor activity.[2] These empirical results are consistent with the general model of perception-action cycles embedded throughout all levels of sensory processing.

Important communications services filter multisensory information at significant cost. Text and text messaging constrain communication to visual symbols. Telephone calls limit communication to auditory signals. Human beings successfully communicate despite these sensory constraints. Real-world communication behavior, however, indicates that these constraints impose significant bodily costs. Communications services that provide multisensory communication, i.e. audiovisual streams like television, vastly predominate in the allocation of persons’ time using communication services.

Sensory-constrained communication services have some offsetting advantages. Creating a PicturePhone was not enough to create a valuable communication service. Different sensory forms have different implications for multi-person communication, emotional distance, time patterns in dialogue, and many other pragmatic aspects of communication. For example, text messaging can be done surreptitiously in circumstances, such as a class or meeting, where other forms of communication would be more readily noticed. Sensory-limited communication is likely to persist even when multisensory communication is cheap, highly capable, and widely available.

Human beings make multisensory sense of unimodal communications relatively well compared to current non-biological information processors. An efficient distribution of computation between humans and non-biological devices apparently favors non-biological devices creating television programs rather than similar radio programs. But given a stream of silent images, humans can generate a descriptive stream of audio for those images much more quickly, cheaply, and accurately than can current non-biological technology. Understanding better how humans make multisensory sense from unimodal communication might provide insights into improving the weak performance of non-biological devices in such tasks.

Notes:

[1] Ghazanfar, Asif A. and Charles E. Schroeder, “Is neocortex essentially multisensory?” Trends in Cognitive Sciences, v. 10, n. 6 (June 2006) pp. 278-86.

[2] Porter, Kristen Kelly, Ryan R. Metzger, and Jennifer M. Groh, “Visual- and saccade-related signals in the primate inferior colliculus,” PNAS v. 104, n. 45 (Nov. 6, 2007) pp. 17855-17860. Cited (approximate) figures are based on the figures reported in Table 1 (63.9% and 19.4% are figures to the full precision given).