Research on multi-task learning methods for traditional chinese medicine visual question answering

With the rapid development of artificial intelligence technology, visual question answering (VQA) has gradually emerged in the field of traditional Chinese medicine (TCM). Given a TCM image, the model can answer questions related to the herb. By leveraging VQA technology, people can gain a more convenient and intuitive understanding of TCM, facilitating the dissemination and promotion of TCM culture. However, the current TCM VQA field faces severe dataset shortages and low adaptability of existing VQA models. To address this, a specialized TCM VQA dataset was constructed, and a multi-task learning-based TCM VQA model (TCMML) was proposed. The model employs Faster R-CNN and Chinese BERT to extract image and text features, respectively, and achieves multimodal feature fusion through an end-to-end joint attention network based on self-attention and cross-attention. Additionally, the model incorporates a multi-task learning strategy: a task-shared layer enables cross-modal alignment and learning of task correlations, while five task-specific modules handle five subtasks of TCM VQA, ultimately achieving high-precision answer prediction. Experimental results demonstrate that TCMML achieves high accuracy in TCM VQA tasks, significantly outperforming mainstream models, validating the effectiveness of the multi-task learning strategy in this task.