Large Language Models in Industrial Systems Engineering: a pilot study on solving production systems problems

Large Language Models (LLMs) are becoming increasingly accessible within engineering education, yet their reliability in supporting quantitative analysis of production systems remains largely unexplored. When applied to typical Industrial Systems Engineering tasks, such as throughput estimation or capacity evaluation, LLM-generated solutions may vary in correctness, transparency, and technical consistency depending on how the problem is formulated. This paper presents a pilot study aimed at evaluating and improving the operational reliability of LLMs when applied to core topics in an undergraduate Industrial Systems Engineering course. Specifically, we investigate their behaviour in solving problems directly related to production systems analysis. The experimental setting covers three fundamental topics: material input rate determination in a production line, bottleneck identification, and theoretical production capacity sizing. We systematically vary selected prompt features, derived from the dominant literature on the topic, and test their effect on output quality using three LLMs. Preliminary results indicate that some prompt configurations are associated with a higher proportion of correct solutions, although performance remains sensitive to task structure and model differences. Building on these findings, the study derives methodological guidelines for reliable LLM use in Industrial Systems Engineering contexts. While this study focuses on technical reliability, understanding the impact of LLM use on student learning and potential cognitive offloading will require dedicated future investigations. Establishing reliable usage conditions is a necessary first step toward that objective.