A Modular AI-Powered Framework for Semantic Retrieval and Dialogue in Serious Games for Cultural Heritage

This work presents a modular framework that integrates vision-language models and multimodal Retrieval-Augmented Generation (RAG) to enhance interaction in Serious Games for cultural heritage. The system leverages CLIP to extract visual embeddings from in-game scenes and FAISS to perform efficient similarity search against a pre-indexed image database. Retrieved metadata is used to generate context-aware responses via a locally hosted large language model (LLM), enabling a conversational agent to provide semantically aligned and culturally inclusive dialogue. The architecture consists of three main components: a preprocessing pipeline for image embedding and metadata serialization, a Unity-based client for user interaction, and a FastAPI-based backend for coordinating retrieval and generation. Tested in an archaeological virtual museum, the framework demonstrates improved user engagement, educational potential, and support for adaptive narrative experiences. By combining real-time visual understanding with generative language capabilities, this approach advances the design of interactive and accessible cultural heritage experiences driven by AI.