Aiming at the problem that the current multi-modal models can't fully excavate the spatial relationship of non-significant regions and the semantic relationship between contexts, resulting in poor inference of multimodal relationship, this paper proposed a multi-modal model based on cross-modal multi-dimensional relationship enhancement, which was used to extract the spatial relation information between the image elements under the latent layer structure, and reasoning the semantic correlation between visual and language. Firstly, the model designed a feature diversity module to mine the sub-significant region features associated with significant regions in the image, thus enhancing the image spatial relationship feature representation. Secondly, it learned the context relationship of language in the image by the context guided attention module to achieve cross modal relationship alignment. Experiments on the MSCOCO dataset show that the proposed model achieves better performance, with BLEU-4 and CIDEr scores are improved by 0.5% and 1.3%, respectively. This approach is also applied to the visual question answering task, and the performance is improved by 0.62% on the VQA 2.0 dataset, which proves the wide applicability of the approach in multimodal tasks.

[2023-01-05] Accepted Paper /doc/2022.11.0517/v1

[2023-08-05] Printed Article /doc/2022.11.0517/v2