LLM2image: A novel framework for accurate diagnosis of diarrhea viruses using pathological images and semantic information.

Abstract

Accurate diagnosis of diarrheal viruses from histopathological images is critical for veterinary medicine and animal disease control. Existing deep learning approaches, however, often rely solely on visual features, lacking integration of clinical semantics and struggling with limited annotated data. To overcome these limitations, we propose LLM2image, a multimodal framework that enhances pathological image classification by integrating visual features with semantically rich textual descriptions generated by large language models (LLMs). The framework comprises: (1) a pixel-level MLP encoder for image representation, (2) GPT-4.0 for generating class-specific pathological descriptions, and (3) a cross-modal fusion transformer that aligns visual and textual features via multi-head attention. Evaluated on an in-house dataset of 516 porcine intestinal images across four categories healthy, ETEC, PDCoV, and PoRVour method achieved an accuracy of 89.51%, surpassing state-of-the-art models and matching the diagnostic performance of veterinarians with 10 years of experience. External validation on the public PAIP2020 colorectal cancer dataset further confirmed its generalizability, with 87.42% accuracy. Ablation studies and visual attention analysis demonstrate that the inclusion of LLM-generated text significantly improves both classification accuracy and interpretability. The model has been deployed as a lightweight Android application, supporting offline rapid diagnosis, highlighting its potential for real-world veterinary and medical applications.