The Epistemic Reverse Side of Instrumental Images

The hand, Aristotle writes, is »the instrument of all instruments « (instrumentum instrumentorum). More recently, this body part has become central to theoretical debates thanks to smartphones, tablets, and other new technology (Ruf). While case studies of the hand exist in the fields of medical, cultural, and art history (Leroi-Gourhan; Gadebusch Bondio), a study of the hand from an image studies perspective has not yet been undertaken. This project takes the hand as its object of study in order to fill this important gap in research.

The project examines the hand as a dialectical object between form and function and addresses questions concerning the hand as a figure in drawings and instrumental imaging in the context of hand surgery and its emergence as a unique discipline. The study will also look at the field’s epistemological prehistory together with its role in the history of images. Other important areas to be touched upon are technological developments in medicine and how they relate to advancements in arms technology as well as the geometrization and disciplining of the body with regard to both Early Modern theoretical debates on art and architecture and current issues concerning changes in media. In addition, the project will analyze the practice of drawing as an epistemic tool and its embeddedness in pictorial practices used in scientific research and other realms of collecting, contrasting, and organizing information. The study also looks at exchanges between different media and drawing, for example, photography (slides, x-rays), print media such as newspaper pictures, and reproductions of artworks. Informed by the theory of drawing (Kemp) and the history of hand-related knowledge (Sherman), the project utilizes the posthumous archive of the hand surgeon J. William Littler as the basis for investigations into the material culture and pictorial practices of hand surgery. This archive contains images of the hand used for research and in patient consultations. Key questions ask about the concept of »style« in surgery, the interrelationship between photography and hand-drawn pictures, as well as how changes in media technology play a role in the use of images in hand surgery.

Within the research area Affective Computing digital devices have been developed which automatically capture and calculate emotions based on physical  patterns. Based on large databases the new technologies promise to capture a wide range of bodily expressions and ›true‹ emotional experience. The Facial Action Coding System (FACS) established in 1978 is used as standard tool of capturing emotional expressions. However, the FACS reduces emotional patterns to only six basic emotions and the muscular movements of  the face.

From a media and culture studies perspective, the dissertation project examines the use of empirical and image based methods that capture emotions in psychology laboratories from the mid-20^th century. In addition, the dissertation analyzes digital tools that capture emotions in the field of Affective Computing with attention to specific cases in the field of psychology in which digital tools are used as with cyberpsychology and the treatment of autism spectrum disorders. The overarching goal is to investigate the lingering effects of historical knowledge within the realm of current affect studies as they pertain to the development of FACS in the age of Affective Computing.

Associate Researcher(s): Martin Grewe (ZIB)

Models of human facial morphology and its dynamics provide the foundations for developing computer-based tools to plan and analyze facial surgery. They are also crucial in the neurocognitive study of how facial expressions are perceived and processed. Medical and neuroscientific applications have long used models of facial expression such as the Facial-Action-Coding-System. However, these models pose significant limitations: In addition to promoting a categorical coding system and a terminological and methodological conflation of emotion with induced facial expressions, they are also restricted by their primarily qualitative description of morphological patterns of expression (e.g., the FACS uses photographs of faces as examples). Furthermore they reduce facial dynamics to mere muscular movement. New applications and questions, however, call for a quantitative model of dynamic facial forms that is better able to take into account subtle differences.

The dissertation project therefore focuses on establishing a highly detailed 3D face and expression database (the ”Mimik Archiv”) intended to support the statistical analysis of the quantitative characteristics of facial morphology. The resulting morphological patterns make it possible to generate a combined model of facial expressions, called the »Mimik-Atlas«. It is intended to serve as a tool for medical and neuroscientific applications.

The first phase of the project aims to develop and improve existing stereo-photogrammetric methods for the acquisition of 3D facial morphology. The Camera Facialis project at the Konrad-Zuse-Center for Information Technology (ZIB) created a multi-camera setup that makes it possible to precisely measure the 3D geometry and colors of facial surfaces within a fraction of a second. The level of detail contained in the digital models, both in terms of geometric and photometric data, surpasses the capacities of currently available systems. Over the course of the project, the Camera Facialis will be used to collect a wide database of facial expressions.

Further research will be concerned with the development of new methods for the automatic processing of newly captured facial expression data. The goal is to enable the rapid integration of digital 3D models into the archive as well as to facilitate retrieval and analysis of the stored morphological data.