Researchers from the University of Cologne and various European universities working in the fields of linguistics, psychology, phonetics and other disciplines wish to make research into language and communication more transparent, and ensure that greater account is taken of non-verbal information. To this end, they have developed a framework concept intended to guide the collection of multimodal data and its analysis, while at the same time leaving room to organize the research process according to the specific research question under consideration.
The study was funded by the German Research Foundation (DFG) as part of the Priority Programme “Visual Communication” (ViCom). The workflow was published in the study entitled “Data Collection in Multimodal Language and Communication Research: A Flexible Decision Framework” in the journal “Advances in Methods and Practices in Psychological Science”.
Language and communication amount to more than just spoken words or sign language signs. When people communicate, they simultaneously use hand gestures, facial expressions, eye movements and a variety of head and body movements. In linguistics, the different ways in which people express and perceive language or information are referred to as modalities. Signals from different modalities – speech, manual and non-manual gestures, and other meaningful elements – are closely interlinked and together contribute to the transmission of meaning. The coordinated use of such signals across different modalities is referred to as multimodality. Until the 1980s and 1990s, this complexity was largely overlooked in linguistics, where research focused predominantly on spoken or written language.
One reason for this was that scientific methods often failed to adequately capture this complexity. Multimodal data sets are complex, costly, and methodologically demanding. Data collection can take place in very different research contexts – from classic, strictly controlled laboratory experiments to the creation of extensive multimodal corpora and naturalistic field studies. Each of these environments is associated with specific trade-offs in terms of precision, realism and scalability that researchers need to consider when planning.
To address these challenges, the interdisciplinary research team proposes a flexible decision-making framework that structures multimodal research into three key stages: (1) defining the research question, population, and design; (2) implementing the study, including technical and ethical considerations; and (3) planning data sharing and reuse. Rather than prescribing a single ‘best’ method, the workflow specified in the paper is intended to provide researchers with a framework and help them reach transparent and comprehensible decisions that are tailored to their particular research objectives. “Multimodal research always involves trade-offs,” explains the first author of the study, Dr Anastasia Bauer from the University of Cologne, who conducts research at the Institute of Linguistics as part of the Key Profile Area “Skills and Structures in Language and Cognition” (SSLAC). “Our aim was not to eliminate these trade-offs, but rather to make them visible and manageable.”
In the article, the proposed workflow is illustrated using three exemplary case studies that represent fundamentally different research contexts: a controlled laboratory experiment, the creation of a large-scale sign language corpus, and observations of non-human primate communication in the natural environment. In addition to methodological aspects, the workflow described in the article also addresses fundamental questions of research transparency, ethics and data transfer in the context of multimodal research. “The systematic documentation of decisions should improve reproducibility and facilitate the meaningful re-utilization of multimodal data sets – from study planning to data collection, and data management and sharing,” says Bauer.
Media Contact:
Dr Anastasia Bauer
Department of Linguistics
anastasia.bauer(at)uni-koeln(dot)de
Professor Dr Petra Schumacher
Department of German Language and Literature I
petra.schumacher(at)uni-koeln(dot)de
Press and Communications Team:
Mathias Martin
+49 221 470 1705
m.martin(at)verw.uni-koeln(dot)de
More Information:
https://vicom.info/
Publication:
https://doi.org/10.1177/25152459261442338
Sign language video on the study, by Roman Poryadin:
https://doi.org/10.6084/m9.figshare.32228727