Recent advancements in Generative Artificial Intelligence (GenAI), driven by the development of Large Language Models (LLMs), have created opportunities for innovative solutions across multiple sectors, including manufacturing. However, deploying LLMs in industrial settings presents significant challenges due to their general-purpose nature, often resulting in inaccurate or irrelevant responses when applied to domain-specific tasks. This paper explores the integration of Retrieval-Augmented Generation (RAG) to improve the performance of LLMs in industrial environments. The study investigates the effectiveness of five LLMs (GPT-4o, Mixtral-8 × 22B-Instruct-v0.1, Llama-3.3-70B-Instruct, DeepSeek-V3, Qwen2.5-72B-Instruct) in processing industrial technical documentation. The models’ performance was evaluated with and without RAG augmentation, using a dataset of 100 verified frequently asked questions (FAQs) from an industrial 3D printer manual. The evaluation employed three approaches: expert-based evaluation, embedding-based evaluation, and LLM-based comparative assessment. Additionally, the study examines whether the results from the embedding-based and LLM-based evaluations align with expert judgments. Findings indicate that RAG significantly enhances the accuracy and relevance of LLM-generated responses, demonstrating its potential as a scalable and flexible approach for adapting LLMs to specialized manufacturing contexts. However, expert-based evaluation remains essential in assessing LLM performance, as automated evaluation methods alone may not fully capture domain-specific nuances. This research contributes to a deeper understanding of LLM implementation in industrial applications.

Proietti, S., Sabetta, N., Fiocco, E., Ranaldi, F., Cesarotti, V., Costantino, F., et al. (2026). Assessing LLMs Capabilities for Industrial Knowledge Management: Performance Analysis of Embedded Knowledge and Retrieval Augmented Generation Approach. In Springer Proceedings in Mathematics and Statistics (pp.1-13). Springer [10.1007/978-3-032-19167-0_1].

Assessing LLMs Capabilities for Industrial Knowledge Management: Performance Analysis of Embedded Knowledge and Retrieval Augmented Generation Approach

Serena Proietti;Emanuele Fiocco;Federico Ranaldi;Vittorio Cesarotti;Fabio Massimo Zanzotto
2026-01-01

Abstract

Recent advancements in Generative Artificial Intelligence (GenAI), driven by the development of Large Language Models (LLMs), have created opportunities for innovative solutions across multiple sectors, including manufacturing. However, deploying LLMs in industrial settings presents significant challenges due to their general-purpose nature, often resulting in inaccurate or irrelevant responses when applied to domain-specific tasks. This paper explores the integration of Retrieval-Augmented Generation (RAG) to improve the performance of LLMs in industrial environments. The study investigates the effectiveness of five LLMs (GPT-4o, Mixtral-8 × 22B-Instruct-v0.1, Llama-3.3-70B-Instruct, DeepSeek-V3, Qwen2.5-72B-Instruct) in processing industrial technical documentation. The models’ performance was evaluated with and without RAG augmentation, using a dataset of 100 verified frequently asked questions (FAQs) from an industrial 3D printer manual. The evaluation employed three approaches: expert-based evaluation, embedding-based evaluation, and LLM-based comparative assessment. Additionally, the study examines whether the results from the embedding-based and LLM-based evaluations align with expert judgments. Findings indicate that RAG significantly enhances the accuracy and relevance of LLM-generated responses, demonstrating its potential as a scalable and flexible approach for adapting LLMs to specialized manufacturing contexts. However, expert-based evaluation remains essential in assessing LLM performance, as automated evaluation methods alone may not fully capture domain-specific nuances. This research contributes to a deeper understanding of LLM implementation in industrial applications.
31st International Joint Conference on Industrial Engineering and Operations Management, IJCIEOM 2025
ita
2025
Rilevanza internazionale
2026
Settore IIND-05/A - Impianti industriali meccanici
English
Industry 5.0
Large language models
Smart manufacturing
Intervento a convegno
Proietti, S., Sabetta, N., Fiocco, E., Ranaldi, F., Cesarotti, V., Costantino, F., et al. (2026). Assessing LLMs Capabilities for Industrial Knowledge Management: Performance Analysis of Embedded Knowledge and Retrieval Augmented Generation Approach. In Springer Proceedings in Mathematics and Statistics (pp.1-13). Springer [10.1007/978-3-032-19167-0_1].
Proietti, S; Sabetta, N; Fiocco, E; Ranaldi, F; Cesarotti, V; Costantino, F; Colabianchi, S; Zanzotto, Fm
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2108/476648
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact