DCMS DLC Coatings Producing Assistant Testing Data
Description
This dataset supports the research paper on the development of a specialized intelligent assistant for diamond-like carbon (DLC) coatings. The central hypothesis is that a Large Language Model (LLM) enhanced with a domain-specific Retrieval Augmented Generation (RAG) technique can significantly outperform base LLMs in handling complex technical tasks within the niche field of magnetron-sputtered DLC coatings. The data was gathered to train and rigorously evaluate this hypothesis. It represents a curated knowledge base of scientific publications and a series of technical queries and answers related to DLC coating processes, properties, and problem-solving. Each row in the dataset corresponds to a specific scientific document. The core data includes the original Filename, Title, and summaries of the source papers. The key generated fields are the Questions and answers and the Relevant papers retrieved by the RAG system for each query. The performance data is captured through multiple evaluation columns. For both the specialized Assistant (powered by either DeepSeek or GLM) and the Base LLMs, the dataset provides the model's generated answers, overall evaluation verdicts, and fractional scores indicating the rate of fully correct answers (A evals fraction) and partially correct answers (A,C evals fraction). The data shows a notable finding: the RAG-enhanced Assistant achieved a dramatically higher accuracy of 87% in responding to technical questions compared to the 25% accuracy of the base LLM, quantitatively demonstrating the value of domain-specific knowledge augmentation. Researchers can use this dataset to analyze the types of questions where the specialized system succeeds or fails, understand the relevance of the retrieved papers for accurate answering, and potentially use the question-answer pairs as a benchmark for developing their own domain-specific AI assistants in materials science.