Balancing speed and precision: a comparative study of ASR systems in multimodal collaborative environments

Abstract

Automatic Speech Recognition (ASR) systems are increasingly critical for analyzing collaborative problem-solving (CPS) tasks, yet their segmentation and transcription accuracy in dynamic, multimodal environments remain underexplored. This study evaluates the performance of OpenAI's Whisper (Large, Medium, Turbo) and Vosk ASR systems in segmenting and transcribing collaborative dialogue, with a focus on implications for CPS annotation workflows. Leveraging a dataset of triads solving a multimodal task—comprising oracle (human-segmented), Google-segmented, and Whisper-segmented audio—we measure transcription accuracy via Word Error Rate (WER) and assess segmentation alignment through start time deviations, segment length ratios, and pause dynamics. Results reveal that while Whisper Turbo achieves the lowest overall WER (52.5%), its semantic segmentation strategy fragments coherent CPS moves, complicating annotation. Conversely, Vosk's pause-based approach under-segments rapid exchanges, obscuring interruptions and cross-talk. The study highlights a fundamental tension: Whisper prioritizes intent preservation at the cost of over-segmentation, while Vosk and Google ASR sacrifice nuance for efficiency. Annotation fidelity is further eroded by ASR-induced errors, including insertions (e.g., hallucinated phrases during silence) and temporal misalignments. These findings underscore the need for hybrid segmentation strategies and adaptive annotation frameworks that explicitly account for ASR limitations. Practical recommendations are proposed, including model-specific post-processing and context-aware annotation tools. By bridging technical evaluation with real-world application, this work advances the design of ASR systems tailored for collaborative environments, ensuring their outputs align with the complexities of human interaction.