Human-AI teaming for water quality documentation analysis with large language models

Abstract

Water quality management in urban systems depends on extensive documentation, including laboratory reports, field observations, regulatory guidance, and operational records. Manual analysis of these textual sources is time-intensive, inconsistent across reviewers, and often limited by workforce capacity. This study evaluates the use of large language models (LLMs) as part of a human-AI teaming workflow for identifying water quality concerns in documentation. Twenty water quality field and test documents were manually coded for concerns such as corrosion, pollutants, and chemical imbalances, then analyzed using OpenAI's GPT-4o, GPT-4o-mini, and o1-mini models. The models were tested using chunked document inputs and compared under open-text and localized prompting strategies that incorporated standard operating procedures and utility-specific water quality definitions. Performance was assessed using percent agreement, accuracy, precision, recall, and Fleiss’ Kappa. Results showed that localized chunked data improved model accuracy, agreement across iterations, and true positive identification compared with open-text prompting. The findings suggest that LLM-assisted document analysis can reduce interpretive workload, support more consistent review, and help water managers access previously underexamined textual data. Human-AI teaming offers a promising approach for improving the efficiency and reliability of water quality documentation analysis in urban water management.