Introduction

Engineering knowledge, especially for cyber-physical systems like the one in the image above, spans several domains. Mechanical, electrical, and software engineering all describe different parts of the same system, often using different tools and document formats. Some important details often only exist in the minds of individual experts.

That fragmentation makes the knowledge difficult to access, not only for people but also for AI systems. While coding agents can already navigate software repositories surprisingly well, engineering data rarely comes in such a consistent, machine-readable form. This means many of the benefits generative AI is bringing to software development are much harder to realise in other engineering disciplines.

SysML v2 is a modelling language standardised by the Object Management Group (OMG) for describing complex systems in a structured way. For my bachelor thesis, I explored whether a textual SysML v2 model could act as a shared context source for an AI agent: can an LLM reliably find and reason about the engineering information stored in it?

Setup

To investigate this, I built an evaluation framework for testing different combinations of SysML v2 models and AI agent architectures. Each configuration answered a predefined set of questions, covering everything from simple facts to relationships and questions that required some reasoning. An LLM-as-a-judge compared the generated answers with reference answers, while the framework also tracked latency, token usage, and cost.

I compared three approaches:

• In-Context: The complete SysML model is placed directly in the prompt. This is simple and requires no retrieval step, but the prompt grows along with the model.
• Agentic Search: Using ripgrep (rg) in a sandboxed environment, the agent searches the SysML files on demand, reviews the results, and searches again when it needs more context. It behaves much like a coding agent exploring a repository.
• Vector RAG: The model is split into meaningful sections and stored in a vector database. The most similar sections are retrieved before a single LLM call.

Results and Conclusion

The encouraging result is that all three approaches could extract and use information from the SysML v2 models. Across the benchmark configurations, average pass rates ranged from roughly 75% to 97%. So, at least for question answering over an existing model, SysML v2 can work as an AI-readable engineering context source.

The diffrent architectures came with very different trade-offs, though. In-Context was fast, simple, and inexpensive for smaller models, but became less attractive as the amount of model data increased. Agentic Search achieved the most consistently accurate results and handled larger models well, but its repeated searches made it noticeably slower. Vector RAG kept latency low and avoided loading the full model into every prompt, but its retrieval step sometimes selected the wrong sections or missed an important relationship. In this implementation, it needs more work before it can match the reliability of Agentic Search.

My biggest takeaway, however, sits outside the benchmark itself. Getting an LLM to read a well-formed SysML v2 model is possible. Creating that trustworthy model and keeping it aligned with a real system as it changes is the much harder problem.

Real engineering data is incomplete, distributed, and constantly evolving. A useful context model therefore needs clear ownership, regular expert review, automated consistency checks, and a way to flag differences between the model and other engineering artefacts. Without that work, even a very capable agent will confidently return information that is incomplete or outdated.

In short: the retrieval technology is promising, but the quality and maintenance of the underlying engineering knowledge will determine whether it is genuinely useful.