A method for constructing a multi-modal knowledge graph of sheep disease based on a pre-trained model.

Abstract

With the gradual enrichment of knowledge types in animal husbandry, the importance of multi-modal knowledge graphs (MMKG) in the field of intelligent diagnosis of animal diseases has become increasingly prominent. However, the construction of multi-modal knowledge graph faces challenges such as lack of professional knowledge, scarcity of multi-modal data and insufficient labeled samples. This paper proposes a construction method of sheep disease multi-modal knowledge graph, which includes three core steps. Firstly, a text knowledge graph is constructed according to the RoBERTa + BiLSTM + CRF sequence labeling model, which provides a framework for multi-modal knowledge alignment; Secondly, the vision-language pre-training two-stream model is introduced and a LoRA fine-tuning method with Bayesian optimization for domain adaptation is used to realize cross-modal representation learning; Finally, the GraphXR tool and the rules proposed by this method were used to realize multi-modal knowledge alignment, storage and visualization. Based on the ViT visual model series, this study carried out experiments on the self-built sheep disease dataset and the public dataset EuroSAT, the image-text matching accuracy reached 81.82% and 93.11%, respectively, which were 40.91% and 38.04% higher than those before fine-tuning. The experimental results show that the method of constructing multi-modal knowledge graph of sheep disease by fusing pre-trained model fine-tuning has significant advantages. This method can significantly improve the model efficiency and generalization ability under the condition of small sample in the vertical field, enhance the semantic representation ability of sheep disease, and provide effective technical support and knowledge basis for the realization of applications like intelligent diagnosis of sheep disease.