Head.AI Platform
Description
Head.AI is a free, AI-powered clinical decision support platform developed to classify headache disorders based on the full structure of the International Classification of Headache Disorders, 3rd edition (ICHD-3). The platform leverages the GPT-4o API (OpenAI) and integrates a structured, expert-curated JSON knowledge base. This database includes detailed metadata for each ICHD-3 diagnosis, such as diagnostic criteria, minimum episode duration, and classification codes. Head.AI was designed to support multilingual input—particularly in Brazilian Portuguese—while consistently generating structured diagnostic outputs in English. Rather than relying on model fine-tuning or supervised training, the platform achieves high performance through prompt engineering and strict adherence to the ICHD-3 diagnostic logic embedded in the knowledge layer. To validate its accuracy, an independent dataset of 315 expert-written clinical vignettes was used, covering 215 distinct ICHD-3 classifications. These vignettes were developed by board-certified neurologists without the use of AI tools and reflected real-world clinical variability. Each case was then processed by Head.AI and three other LLM platforms (Gemini 2.5, Grok 3.0, and Claude Sonnet 4.0), and diagnostic outputs were compared. The dataset employed in this study consisted of 315 expert-generated clinical vignettes, each representing a unique headache case classified according to the International Classification of Headache Disorders, 3rd edition (ICHD-3). The vignettes were synthetically created by two board-certified neurologists with expertise in headache medicine. Importantly, no generative AI tools were used during vignette development, ensuring unbiased and clinically grounded case construction. Each vignette was annotated with the following metadata: The ICHD-3 code corresponding to the correct diagnosis; The headache category (primary or secondary); An expert-assigned difficulty rating (easy, moderate, or difficult). A JSON file is also available here.
Files
Steps to reproduce
Reference Framework All clinical cases were constructed in accordance with the International Classification of Headache Disorders, 3rd edition (ICHD-3), as published by the International Headache Society. The classification structure and diagnostic criteria served as the exclusive reference standard. Case Development Two board-certified neurologists with expertise in headache medicine independently created synthetic clinical vignettes. Each vignette was designed to reflect real-world diagnostic complexity and was mapped to a single ICHD-3 diagnosis. No generative AI tools were used during case generation. Diagnostic Annotation Each vignette was annotated with: The corresponding ICHD-3 diagnostic code; Headache classification (primary or secondary); Difficulty level assigned by expert consensus (easy, moderate, or difficult). Language and Format All vignettes were originally written in Brazilian Portuguese to simulate typical documentation in clinical settings within Brazil. Each vignette followed a semi-structured narrative format, including presenting complaint, history of illness, and physical/neurological examination. Data Structure The dataset was compiled into a structured Excel file, where each row corresponds to one vignette. Columns include: Unique case ID; Full text of the clinical vignette; ICHD-3 code; Classification type; Difficulty rating; Outputs from each evaluated AI platform (Head.AI, Gemini, Grok, Claude); Ranking of correct diagnosis by each platform.
Institutions
- Universidade Federal de Sao Paulo