purple motes

Shen Wei Dance Arts painted and played Connect Transfer at the Kennedy Center this weekend. In the open space of the Concert Hall, the performance connected senses and worked across specific to abstract in an intriguing externalization of the human brain.

In caricature, ballet is about the air, and modern dance is about the floor. Shen Wei's company connected to the floor with lines and strokes. Even when the dancers were moving with their torsos against the floor, energy flowed out of their arms and legs, pulling them across the floor. The movement had the earthliness of modern dance, but rather than with the weight of limbs and steps, it imaged neurons firing.

Cross-sensory effects shape the performance. In an interview for Dance Umbrella in October, 2007, Shen Wei explained:

Sometimes you hear a sound and you may see an image. Sometimes you see a movement and you feel its speed. You see the movement and hear the sound as well. ... When I see a physical movement, it's important at the same time to keep my others senses open --- my ears, my feelings, my touch. That way I might understand how I can deepen all of the elements of my experience.

The human neural system integrates sensory modalities across all stages of sensory processing. Connect Transfer abstractly enacts those processes.

The work also images calligraphy being written and written characters unraveling and reforming. About a third of the way through, a single dancer wearing a single, slightly extended, gloved sponge soaked in black paint brushes through the space with circular movements within traveling floorwork. The image of the movement is brush strokes of calligraphy. At one point, the music stops and floor microphones bring out the sound of the dancer's brushing. Nonetheless, much of the movement that evokes brushwork occurs up from brushing paint on the floor. The play of senses and abstraction in brush-movement is an intriguing aspect of the work.

The floor painting was not just a product of brushing. Moving on the floor, the dancer's tights acquired paint and transfered paint. So did their feet. One dancer had one bare hand painted red. Her painted hand created some hand stencils on the floor like those of cave art dating back about 30,000 years. In their diversity of forms and scales, and in the sense of purposive actions and chance occurrences, the resulting paintings are much more evocative of a biological system than abstract expressionist paintings.

The part of Connect Transfer that I liked least was the personal seal that Shen Wei danced. This was done in a spotlight in front-center stage, twice, to very different music. It included the only movement directed through a dancer's eyes. Some of the movement seemed to be body-builder camp. Connect Transfer as a whole contained little parody or irony. But for me, Shen Wei's seal evoked, intentionally or not, a parody of the egotistical artist.

More ambitious staging might better mark the trajectory of the performance and enlarge the sense of the artistic process. Why not pull the canvas out from under the dancers after they have worked on it for awhile? With the canvas as a back drop, the percussive section of the performance might use movement patterned on finger painting and dabbing of thick, oil-based paint. Then maybe move the canvas above them. And, with a sufficiently translucent canvas, why not finish with a section that has the artists dancing behind their work? Such staging might shift some of the work from the brain to more accessible, personal artistic experience.

1 Comment

In a path-breaking decision, the U.S. Supreme Court posted on its website a video along with text of its opinion in Scott v. Harris. Common sense is essentially multimodal. By including video along with its opinion, the Court provided a decision record that communicates its judgment more effectively than would just text.

The Court's recording of its decisions has had significant effects. In the Court's first copyright decision in 1834, the Court declared itself "unanimously of opinion that no reporter has or can have any copyright in the written opinions delivered by this Court, and that the judges thereof cannot confer on any reporter any such right." (Wheaton v. Peters, 33 U.S. 59, 668). The Court's decisions thus became a legally secure component of the public domain. In Burrow-Giles Lithographic Co. v. Sarony (1884), the Court found, "this photograph [“Oscar Wilde, No. 18”] to be an original work of art, the product of plaintiff’s intellectual invention of which plaintiff is the author, and of a class of inventions for which the constitution intended that congress should secure to him the exclusive right to use, publish, and sell." The Court did not however, include a reproduction of this photograph in its decision. It thus did not place in the public domain a work derived from a photograph of significant public importance. Not doing so, among other effects, lessened public understanding of its authoritative judgment

Verbal statements, whether in judicial opinions or elsewhere, are in some circumstances a poor substitute for video. Courts have long recognized the distinctive value of non-verbal evidence through their practice of admitting into evidence objects in addition to personal testimony and documents. A major issue in Scott v. Harris was the danger to the public from a driver fleeing from police cars signaling to the driver to pull over. Video from the police officers' cars communicates extent and nature of the danger in a way that cannot be reduced to a collection of propositional statements of fact. The video also specifies an authoritative judgment with respect to the extent of danger much more clearly, especially to the general public, than does just a text.

The justices apparently did not consciously edit video in the Scott v. Harris case record in making their opinions. The case record included four color videos automatically recorded from police cars when they activated their sirens. The justices' opinions included two very different descriptions of these videos. The decision record for the Court in this case includes, however, just one black-and-white video. That video (the "opinion video") appears to be the video from one police car appended to the video from another.

Selecting relevant video is an important aspect of reasoned argument in Scott v. Harris. In the opinion video, a total of 3.9 minutes of video (24% of the total run time of the opinion video) showed the crash of Harris' vehicle and the subsequent effects and actions (image of the overturned vehicle, the clouds of smoke rising from it, the police officers rushing toward the vehicle and desperately trying to open the car door to get Harris out of danger, etc.; for better understanding, watch the relevant segments of the video). Harris, who was only 19 years old, suffered permanent paralysis of his arms and legs from the crash. Irrespective of the legal issues associated with the crash, compassion is a natural and appropriate human reaction to Harris' suffering. Nonetheless, this specific crash and its terrible human effect is not relevant to judging the reasonableness of the prior police decision to terminate the chase in a way that created a risk of serious injury or death for the fleeing driver. Including the video segment of the crash and its aftermath fosters well-recognized biases, e.g. hindsight bias, outcome bias, affective bias. Video for judging the reasonableness of the police action should excise the video segment showing the crash and its aftermath.

Camera viewpoint is an important, case-relevant aspect of video in Scott. The camera viewpoint of the video approximates that of the eyes of a police officer in a chasing police car. This video, as a resource for immersive, imaginative experience, is thus oriented toward persons imaginatively assuming the position of a police officer. Presenting the video in a way that helps persons imaginatively assume the position of a police officer is the best use of the video. Ensuring that viewers are conscious of that particular video viewpoint helps viewers to judge critically the effects of their viewing experience.

Diegetic time is another important, case-relevant aspect of the video. The video includes the period from when the police officers began to seek to stop the driver to when a police officer terminated the resulting chase. The dangerousness of the chase depends on specific circumstances occurring over time. These circumstances include the fleeing driver's and the police officers' physiological states. Since the human hormonal system has slower-acting but more general and longer-enduring effects than the human nervous system, events that trigger hormonal reactions associated with a situation of danger can shape subsequent actions and perceptions of time and danger. Hence judgments of public danger involve complicated temporal issues.

Watching and editing video can promote and communicate reasoned judgment of public danger. An important issue is the time span in which a judgment of dangerousness is reasonably made and revised. Whether unusual events are discounted in such a judgment, or weighted extra heavily, also is an important issue. Watching and editing video as if it were a transparently objective record of events ("naive realism") obscures how video really works and impedes reasoned understanding of video's truth value.[1]

Included below are two videos created from video in the Scott case record. The first video includes segments of video from the beginning of the chase, and a segment of video up to the end of the chase. The second video highlights relatively dangerous events that occurred during the chase. Both videos are designed for the viewer to assume imaginatively the position of a police officer. Both videos use selective cuts of the action and connect the video segments with half-second cross dissolves to preserves sense of immersion. Both videos preserve real-world temporal sequence, except for in one switch in viewpoint that is obvious. The second video stops action at two points to suggest a circumstance that would have been particularly salient in a model of accumulating indications of danger. These videos are not designed to be deceptive or to be propaganda. They are meant to foster rational argument about the danger to the public of allowing the chase to continue.

(more...)

Offer Comments

Drawing upon neuroscience research on multisensory processing, I suggested that new digital combinations of evidence should be required to be introduced within the evidentiary portion of trials. Here I further consider trial procedure and the actual operation of courts.

Under the proposed rule for digital presentation technologies, counsel would be allowed to present evidence in the media form in which the evidence was introduced, verbally talk about the evidence and verbally combine it in any way to make a case (subject, of course, to existing trial rules), but not, in closing statements, be allowed to use digital technology to combine evidence in new ways. That means that in the Skakel case the digitally combined voice recordings and photographs should not have been allowed in the closing rebuttal because that multimedia presentation was not already introduced in the trial.

In the Skakel case, the defense counsel didn't object at trial to the multimedia presentation in the prosecution's closing rebuttal statement. The trial court judge thus had no opportunity to rule on such an objection. On appeal, the defense alleged that the closing multimedia presentation was subliminal messaging and prejudicial. The appeals court ruled, "After viewing the audiovisual presentation, we are not persuaded that there is any reasonable likelihood that the state’s presentation confused the jury or prejudiced the defendant in any way."[1]

To me, the defense's claim on appeal of subliminal messaging was silly and it's claim of prejudice was too generic. At the same time, the appeals court judgment on this point seems to me to be overly emphatic and rationally unsatisfying. It doesn't provide reasons, but merely states a rather broad judgment. It seems to me uncontroversial that the multmedia presentation strongly communicated that the defendant was guilty. The question, as I see it, is whether the particular technological combination of evidence used in the closing portion of the trial was consistent with the defendant's right to a fair trial. The appeals court addressed that question with a non-specific judgment about confusion and prejudice. Specifically examining the effects of new technological combinations of evidence and ruling that such combinations of evidence must be introduced within the evidentiary portion of trial would better serve justice.

Moreover, it seems to me a mistake for appeals courts to defer strongly, implicitly or explicitly, to the process of objection and trial-court judgment in considering new technological combinations of evidence in closing arguments.[2] Media technologies are becoming cheaper, more powerful, and more prevalent. Media technology can be powerful tools for making cases. The process of objection and trial-court judgment for defining permissible digital presentations can create significant disparities in actual trail procedures across trial courts. Moreover, trial judges' rulings on objections aren't written, so they don't provide a written, case basis for precedent concerning the use of technology in trials.

A rule that new digital combinations of evidence must be introduced within the evidentiary portion of the trial does more than simply relocate objection and judgment at trial. Not allowing surprising use of new digital combinations of evidence in closing statements better places such combinations within adversarial case-making. If a digital combination of evidence is admitted in evidence, the opposing counsel has the opportunity to introduce alternative digital combinations of evidence. In addition, opposing counsel can prepare to prompt jurors to think about the construction and effects of particular digital combinations of evidence. In other words, they can prompt jurors to augment their low-level, subconscious processing of evidence with conscious reflection on how such evidence is created and how the human body processes it. Knowledge and high-level reasoning can help persons to understand their sub-conscious reactions and to overcome natural biases.

Moreover, trial procedures for types of case (civil or criminal) and jurisdictions (federal, individual states) have specific rules for the introduction of evidence. New digital combinations of evidence could be introduced under these rules. Moreover, institutions that maintain these rules provide low-transaction-cost paths for developing new evidentiary rules for digital combinations of evidence. The Daubert standard for the admissibility of expert witnesses' testimony, which has evolved operationally both through appellate court decisions and amendments to rules of evidence, provides an example that standards for digital combinations of evidence might follow.

Digital presentation technologies in trials deserves specific, sustained attention. Digital presentation technologies can have important effects on trials. They are not the same as words and hand gestures.

[1] Supreme Court of the State of Connecticut, State of Connecticut v. Michael Skakel, (SC 16844) (Jan. 24, 2006), p. 74.

[2] Id. p. 89, ft. 107 states:

We note, finally, that defense counsel raised no objection to the state’s rebuttal presentation. Apparently, defense counsel did not believe that the state’s use of the audiovisual aides was misleading or otherwise inappropriate. Although that fact is by no means dispositive of the defendant’s claim on appeal, as we have explained; see, e.g., State v. Stevenson, supra, 269 Conn. 576; we nevertheless consider that fact in reviewing the merits of an unpreserved claim of prosecutorial misconduct.

Offer Comments

Digital multimedia presentations can powerfully affect legal trials. Consider the Skakel trial:

During the Connecticut District Attorney's closing argument in the trial of Michael Skakel for the murder, twenty-seven years before, of fifteen-year-old Martha Moxley, jurors heard and read Skakel's own words [which appeared] on the screen before them. And in the instant that Skakel admitted to feeling a sense of "panic" when he saw Martha Moxley's mother on the morning after the killing, there on the very same screen appeared the image of Martha Moxley's lifeless body, just as it was found at the scene of the murder. ... The picture of Martha's battered lifeless form immediately explains the implicit meaning of his words. The viewer instantly makes the connection: immediately upon being reminded that morning of the night before, Skakel must have recalled with horror what he had done.[1]

Digital technology and the human body work together to combine words and images. The result in this case was probably a strong physiological reaction involving both neurons and hormones: muscle tension, brain activity constructing a casual sequence, increased heart rate, and other bodily effects typically associated with horror.

Digital technology and the human body together annihilated time. The image was recorded in 1975, on the day after the murder. Skakel's words were recorded in 1997, while Skakel was speaking to a ghostwriter in an early stage of producing a book. The words and the image were combined in 2002, in a digital presentation shown for the first time in the prosecution's closing rebuttal statement.

Associating Skakel's words with different images might have produced a rather different sense. Skakel recalled in 1997 that, in 1975, on the night before Moxley was murdered, he had been drinking alcohol, and that he had decided to get a kiss from Martha, who he said liked him. The closing multimedia presentation included Skakel, who was then 15 years old, saying, "I woke up to [Mrs.] Moxley saying, "Michael, have, have you seen Martha?" It also included Skakel subsequently saying, "I was like, 'Oh my God, did they see me last night? And I'm like, 'I don't know,' I'm like, and I remember just having a feeling of panic." [2] If these words had been combined with images of underage drinking, drunken sleep, and being wakened by a mother's fearful face, rather than images of an alive and smiling girl and then her freshly murdered body, the jury might have had some reasonable doubt that Skakel's recollection of panic meant guilt of her murder.[3]

A prosecutor's fundamental public responsibility is to work to serve justice. When a prosecutor believes that a jury acting justly under law could find a defendant guilty beyond a reasonable doubt, the prosecutor's responsibility is to make, as convincingly as she can, the case of guilt to the jury. The prosecutor's job is not to make the defense's case.

A prosecutor, however, should not introduce, and should not be allowed to introduce, new evidence during closing statements. The images that prosecutor presented in his closing rebuttal statement were in evidence, as was the audio of Skakel's words. The technological combination of words and images that the prosecutor deployed in his closing rebuttal had not been introduced in evidence. Lawyers have always been permitted to arrange freely in verbal closing statements evidence introduced in the trial. Are technological combinations of words and images new evidence?

In a feature article in Criminal Justice magazine, the president of the consulting firm that designed the presentation and a law professor stated that the use of the multimedia presentation was "completely fair and appropriate." These authors stated:

While the crime scene images may very well have increased jurors' sympathy toward the Moxleys and their resentment toward Skakel, and while those images, precisely timed with Skakel's words, probably increased their conviction that Skakel was guilty of murder, using this kind of visual rhetoric, instead of words alone, to help the jurors understand the evidence is legally appropriate.[4]

According to these authors, the multimedia presentation merely provided better technology for an authorized form of closing rebuttal statement:

Indeed, Jonathan Benedict [the prosecutor] unquestionably could have played the same portions of the audiotape during closing and held up before the jury the same photographs of the murder victim, even enlarged and mounted on posterboard, that he used in the multimedia display. The only difference is that the interactive multimedia system allowed Benedict to juxtapose words and images more smoothly, preventing the jurors from being distracted from the content of his argument: that Michael Skakel was guilty of murdering Martha Moxley.[5]

More precise timing of image-word co-occurrence, in this view, is merely an external technology for furthering jurors' understanding.

New digital technologies used in trials can create bodily effects that might be best judged as new evidence that juries must seek to understand. The human body combines words and images from pre-conscious neural processing to high-level processing. For example, recent evidence indicates that multisensory processing occurs very early in the main auditory pathway. Recent work in neuroscience indicates:

low-level multisensory interactions are characterized by a high degree of temporal precision. For example, during audiovisual vocalization processing in auditory cortex, the sign of the integration appeared to be dependent on the timing between the initial mouth movement and the onset of the auditory component of the vocal signal. The longer the time interval between the initial mouth movement and the onset of the voice, the greater the likelihood of observing response suppression. By contrast, a short time interval leads to enhanced responses. [6]

Digital technology that precisely times the co-occurrence of words and images produces meaningfully different neural processing than does older technology that much less precisely combines words and images. Eliminating jurors' bodily activities other than processing in prefrontal cortex ("rational deliberation") is not physiologically realistic. Respect for physiological realism suggests that technology that produces significant, new subjective effects should be considered within the evidentiary portion of the trial.

"Subliminal messaging" does not provide a scientifically reasonable concept for judging the use of visual persuasion technology in court. Stimulating sub-conscious processing of highly prejudicial, case-irrelevant material clearly is an unfair legal tactic. However, sensory processing below the level of consciousness occurs normally and continually in a living human body. Subliminal messaging implicitly conveys a false model of how the human sensory system works.[7]

New visual persuasion technologies can produce powerful effects at low-levels of sensory processing. Opposing parties at trial need to have the opportunity to prompt jurors to bring these low-level effects to high-level processing. Such a requirement would give reasonable meaning to full and fair deliberation that includes new multisensory presentations.

Notes:

[1] Sherwin, Richard K. (2007), "A Manifesto for Visual Legal Realism", p. 10; also published in Loyola of Los Angeles Law Review, vol. 40, 2007. The presentation occurred in the prosecution's closing rebuttal. Note the discrepancy in the textual description of the presentation timing. In the quoted text, the second sentence indicates that the image cut occurred when Skakel acknowledged panic. The fourth sentence suggest that the image cut occurred with reference to "last night". Consider as well Carney, Brian and Neal Feigenson (2004), "Visual Persuasion in the Michael Skakel Trial: Enhancing Advocacy through Interactive Media Presentations," Criminal Justice, v. 19 n. 1. Carney at the time of publication was the president of the firm that created the interactive media presentation for the Skakel trial. That article offers the following textual description:

[Screen 2] "I was like 'Oh my God, did they see me last night?' And I'm like 'I don't know,' I'm like, and I remember just having a feeling of panic." [Photograph #2 of the corpse of Martha Moxley is shown]

That description is not sufficient to identify precisely when the image cut occurred. The text of the opinion of Supreme Court of Connecticut, ruling on appeal, suggests that the image cut occurred after the audio of Skakel's sentence ending in the word "panic." See State of Connecticut v. Michael Skakel (2006) (SC 16844), p. 73. It seems to me that the effect would have been largest if the cut occurred on the initial sound of the word "panic". Perhaps the text of Supreme Court ruling did not attempt to describe the presentation to that timing resolution. However, as noted above, precisely specified image timing is an important distinguishing feature of a digital multimedia presentation. For more information about the Skakel case, see considerable original analysis at TalkLeft.

[2] The quotes are as reported in State v. Skakel (2006) p. 89, ft. 105. The first quote differs slightly and insignificantly from that given in Carney and Feigenson (2004).

[3] The Supreme Court appellate opinion, State v. Skakel (2006) p. 89 ft. 105, states: "The defendant claims that when he stated ‘‘Oh my God, did they see me last night?’’ he was referring to whether they had seen him masturbating." The defendant did not testify at the trial. The defense's brief noted the prosecution's audiovisual presentation omitted a section of Skakel's words relating to masturbation. See Brief of the Defendant-Appellant, pp 78-9, ft. 80. The transcript section that the defense cited does not clearly indicate that a new perception that someone had seen him masturbating was the cause of Skakel's panic. The full transcript is consistent with the reasons for panic described above. They would also probably make a more effective defense-counsel visual presentation than one of Skakel masturbating.

Insightful voices in the blogsphere (Simple Justice, Norm Pattis, a public defender) suggest that Skakel had ineffective assistance of council. Perhaps the above is additional evidence of poor representation.

[4] From Carney and Feigenson (2004). Criminal Justice is a magazine that the Criminal Justice Section of the American Bar Association publishes. The quoted phrase "completely fair and appropriate" comes from this source, which stated: "Because Benedict's presentation was directly and closely connected to the evidence, his visual argument was completely fair and appropriate."

[5] From Carney and Feigenson (2004).

[6] Ghazanfar, Asif A. and Charles E. Schroeder, "Is neocortex essentially multisensory?" Trends in Cognitive Sciences, v. 10, n. 6 (June 2006) p. 284; the quoted text omits endnote and figure references.

[7] In State v. Skakel, Brief of the Defendant-Appellant, p. 79 accused the prosecution of conveying subliminal messages to the jury with its multimedia presentation in its closing rebuttal statement. Carney and Feigenson (2004) assert in contrast:

No subliminal content was concealed in the Skakel prosecution team's audiovisual presentation. All of the images, audio, and text that the prosecution put before the jurors in closing argument were properly admitted into evidence. ... The multimedia system allowed prosecutors to present images and audio already in evidence so clearly and so memorably that their impact on the jurors was profound.

These arguments show no appreciation for how human sensory processing actually works.

4 Comments

Multisensory processing in primates, like information processing on the Internet, is extensively decentralized. A recent neuroscience review article declared:

The pervasiveness of multisensory influences on all levels of cortical processing compels us to reconsider thinking about neural processing in unisensory terms. Indeed, the multisensory nature of most, possibly all, of the neocortex forces us to abandon the notion that senses ever operate independently during real-world cognition.[1]

Within the ascending auditory pathway prior to the neocortex, neurons in the inferior colliculus respond to auditory signals. Recent research found that about two-thirds of neurons in the inferior colliculus also respond to visual signals. In addition, about 20% of the neurons fire with eye movement motor activity.[2] These empirical results are consistent with the general model of perception-action cycles embedded throughout all levels of sensory processing.

Important communications services filter multisensory information at significant cost. Text and text messaging constrain communication to visual symbols. Telephone calls limit communication to auditory signals. Human beings successfully communicate despite these sensory constraints. Real-world communication behavior, however, indicates that these constraints impose significant bodily costs. Communications services that provide multisensory communication, i.e. audiovisual streams like television, vastly predominate in the allocation of persons' time using communication services.

Sensory-constrained communication services have some offsetting advantages. Creating a PicturePhone was not enough to create a valuable communication service. Different sensory forms have different implications for multi-person communication, emotional distance, time patterns in dialogue, and many other pragmatic aspects of communication. For example, text messaging can be done surreptitiously in circumstances, such as a class or meeting, where other forms of communication would be more readily noticed. Sensory-limited communication is likely to persist even when multisensory communication is cheap, highly capable, and widely available.

Human beings make multisensory sense of unimodal communications relatively well compared to current non-biological information processors. An efficient distribution of computation between humans and non-biological devices apparently favors non-biological devices creating television programs rather than similar radio programs. But given a stream of silent images, humans can generate a descriptive stream of audio for those images much more quickly, cheaply, and accurately than can current non-biological technology. Understanding better how humans make multisensory sense from unimodal communication might provide insights into improving the weak performance of non-biological devices in such tasks.

Notes:

[1] Ghazanfar, Asif A. and Charles E. Schroeder, "Is neocortex essentially multisensory?" Trends in Cognitive Sciences, v. 10, n. 6 (June 2006) pp. 278-86.

[2] Porter, Kristen Kelly, Ryan R. Metzger, and Jennifer M. Groh, "Visual- and saccade-related signals in the primate inferior colliculus," PNAS v. 104, n. 45 (Nov. 6, 2007) pp. 17855-17860. Cited (approximate) figures are based on the figures reported in Table 1 (63.9% and 19.4% are figures to the full precision given).

3 Comments

Persons' spatial representations of an action described verbally are biased in the spatial orientation of the action. Recent research indicates that a left-to-right bias develops from experience of a culture's left-to-right writing system, and a right-to-left bias likewise from a right-to-left writing system.

The Morgan Bible of Louis IX and the Shahnameh of Shah Tahmasp, two magnificent books, illustrate this cultural effect. In the Morgan Bible:

The repetition of a figure moved rightward in a painting corresponds to temporal sequence. Figures generally enter a scene from the left, and exit to the right. Bringing Benjamin back and the repulsion of the Israelites are painted with predominate right-to-left directions of action. In these and other instances, the reversal of the visual convention of European text signals a spatial or conceptual reversal.

The Shahnameh, created in a Persian culture with a right-to-left writing convention, depicts action in the opposite direction:

Among the 258 figural paintings in the Shahnameh of Shah Tahmasp, on my count 70 of these to have a predominate horizontal line of action: 50 from right to left, and 20 from left to right. Left to right lines of action associated with reverse meanings include folio 42v (Faridun's eldest son retreating), 98v (Turanians invading Iran), and 102v (retaliatory killing).

Writing systems are not neutral communications technology. Like any communications technology, they depend on standard physical artifacts that inevitably have sensory effects.

Offer Comments

Video circulation from Massachusetts public libraries doubled from 1998 to 2006. Over the same period, print item circulation fell slightly. Audio and video items together now account for 32% of total item circulation from Massachusetts public libraries.[1] Audiovisual circulation in libraries in other states might account for somewhat less, perhaps a quarter of total circulation.

YouTube, which was launched in November, 2005, now generates about 10% of broadband subscribers' Internet traffic in North America (Ellacoya findings).

Video attracts much more attention than print. Historically, persons have spent on average relatively little leisure time reading. Most of the growth in leisure time from 1925 to the present has been absorbed in watching television. Now persons have on-demand access to large, diverse collections of videos, similar to what they have had for books.

Book digitization and internet publishing makes texts more readily accessible. But changes in access to video will have much larger effects than changes in access to print.

Reference:

[1] Data from Massachusetts Board of Library Commissioners, Public Library Data, Summary Report 2006, p. 29. This is also the source for the above graph.

3 Comments

In 1914, books accounted for 74% of items circulated from the Cincinnati Public Library. Prints accounted for 13% of circulation, lantern slides, 6%, and music rolls, 3%.[1] The Cincinnati Public Library at this time was one of the largest and most lavish public libraries in the U.S. These statistics indicate the scope of services that a leading public library provided.

Data for U.S. public libraries in 1955 show less format concentration in holdings and greater concentration in circulation. Books comprised an estimated 67% of libraries' items and 94% of libraries' circulation. Photos, pictures, and prints, which made up 20% of items, accounted for only 2.2% of circulation. While sound recordings and films accounted for small shares of items and circulation, these formats had relatively rapid turnover in lending (see Table). A film was lent on average 13.3 times per year. A film could be viewed much more quickly (perhaps a half hour for films of this time) than a book could be read, and loan periods for films were probably much shorter than those for books. The ratio of circulation per item suggests considerable interest in viewing films in public libraries' collections.

Notes:

[1] Data from Papers and Proceedings of the Berkeley Conference of the American Library Association, uly 1915, published in the ALA Bulletin, v. 9. The number of (book) volumes at the end of 1914 was 463,521. The source does not give item counts for the other formats. Total circulation for all formats was 2,164,310. A large number of piano rolls are available digitally here and here.

[2] Data from U.S. Dept. of Health, Education, and Welfare, Biennial Survey of Education in the United States 1954-56, Chapter 5, Statistic of Public Libraries: 1955-56, Tables 9-13. The number of library systems reporting non-book items was only 26% of the number of systems reporting book volumes. I've scaled all reported figures by number of systems reporting. If systems reporting non-book items had larger than average non-book holdings, the non-book figures are over-estimates. Because nearly the same number of systems reported items and circulation, scaling matters little to the circ./item figures. For further analysis, see the underlying data for this table.

Offer Comments

Brains operate using perception-action cycles. Good programmers know that a key to fast code is a good data structure. Sensory inputs are like a data structure for the brain. Using your body and physical implements, you can shape the sensory data structure that your brain uses. Moving around to get a good look is thinking on your feet. Tinkering is thinking with your hands. If you just use your head to think, you're mentally handicapped.

Tetris game play provides a good case study of how the brain and hands work together. Careful study of play indicates that players rotate the falling blocks more frequently than is plausible for a linear cognitive program of play.[1] Study also indicates that using keystrokes, a player can rotate a falling block about ten times as fast as a person can mentally rotate such a block.[2] In figuring out where to direct a block, players use keystrokes to rotate blocks at least in part because doing so is more cost effective than performing such actions on representations within one's head.

Trade-offs between external sense and cognitive effort also occurs in choices between different sensory forms of media. The rapid shift of "soap operas" from radio to television suggests that adding visual stream to the programming lowered the cost of making sense of it. More generally, the cost of making sense of presence falls with richer sensory inputs.

Playing Tetris and making sense of presence both involve perception-action cycles. In Tetris, the player rotates blocks to create new sensory inputs so that higher level cognitive processes can more efficiently plan trajectories for placing blocks. In sense of presence, attunement to another occurs as a good created through the social evolution of human nature. This attunement can occur at different cognitives levels, from awareness of twittering of another to a face-to-face, heart-to-heart talk with a best friend. At each level, attunement is associated with characteristic patterns of action such as textual response or eye-tracking.

Perception-action cycles are built into human biology from the lowest to the highest levels of cognitive complexity. As a cognitive scientist explained:

At all levels of the central nervous system, the processing of sensory-guided sequential actions flows from posterior (sensory) to anterior (motor) structures, with feedback at every level. Thus, at cortical levels, information flows in a circular fashion through a series of hierarchically organized areas and connections that constitute the perception-action cycle. Automatic and well-rehearsed actions in response to simple stimuli are integrated at low levels of the cycle, in sensory areas of the posterior (perceptual) hierarchy and in motor areas of the frontal (executive) hierarchy. More complex behavior, guided by more complex and temporally remote stimuli, requires integration at higher cortical levels of both perceptual and executive hierarchies, namely areas of higher sensory association and prefrontal cortex.[3]

Interactivity, which new Internet services emphasize, is deeply embedded in human biology.

Notes:

[1] David Kirsh and Paul Maglio (1994), "On distinguishing epistemic from pragmatic actions," Cognitive Science 18, pp. 15-20.

[2] Id. p. 24.

[3] Joaquín M. Fuster (2004), "Upper processing stages of the perception-action cycle," TRENDS in Cognitive Sciences, vol. 8, no. 4 (April) p. 144. Note that id. Figure 2 describes the general structure of Kirsh and Maglio (1994), p. 39, Figure 16. For additional description of the perception-action cycle, see Paul Baxter's Memoirs of a Postgrad.

1 Comment

Typing text on a keyboard and manipulating a mouse are recent, conventional muscular routines for communication. Those routines have little relation to the muscular practices of communication that humans have used throughout their evolutionary history. Moreover, those routines are much different from muscular activities many people do for enjoyment, such as walking, playing catch, running, playing tag, swimming, and curling. Making communication services more muscularly natural and muscularly enjoyable could create additional value.

In conjunction with the use of sight and clever extra-body technology, a person can write with any muscle at speeds comparable with those of current keyboard routines. The Dasher Project allows a person to write text by directing a point across dynamic, letter-coded regions. With the appropriate linking technology, any muscle, including eye gaze movements, can direct the point to write. Such technology obviously has great value to disabled persons. For persons with a wide range of muscular possibilities, such technology allows communication service providers to offer muscular routines that are natural, enjoyable, and propitious for the specific circumstances of use.

You can use your hands in ways that are much more natural and satisfying than typing on a keyboard. Jeff Han has developed a multi-point graphical interaction surface that is pleasurable even to watch. The forthcoming Apple iPhone, 8 million of which are expected to be sold in its first year on the market, incorporates some touch-screen gestures for controlling the phone.

Highly successful products suggest the value of innovation in the space of muscular movement. Dance Dance Revolution (Dancing Stage) has brought large-muscle leg movement to video games. The Wii video console controller includes motion sensors that enable, for example, sports games to incorporate sports-typical gestures. The Apple iPhone also incorporates motion sensors, as does the recently announced DoCoMo D904i.

I'm still hoping for a mobile camera-phone that better communicates "Look at this!"

Offer Comments