Unstructured Entity Extraction (NER)

This benchmark evaluates the model's ability to perform Named Entity Recognition (NER) and adhere to a strict JSON schema. By embedding synthetic identity details (Name, Email, Job, City) within randomized, noisy, and unstructured text blocks, the task challenges the model to filter out irrelevant signals (such as random IDs or status codes) and map the correct entities to specific JSON keys without hallucination.

Structured Data Transformation (CSV to JSON)

This benchmark evaluates the model's proficiency in syntax translation and data serialization. By supplying a flat, delimited text format (CSV) and requiring a hierarchical key-value output (JSON), the task tests the model's ability to correctly map headers to keys, handle data types (strings, dates), and produce syntactically valid JSON without losing information or hallucinating values.

Phone Number Standardization

This benchmark evaluates the model's ability to perform data cleaning and pattern normalization. By providing input data with inconsistent delimiters (parentheses, dots, dashes), the task requires the model to extract the core numerical entities and restructure them into a strict international standard (+1 XXX XXX XXXX), testing its precision in string manipulation and adherence to rigid output constraints.

Height Comparison / Transitive Logic

This benchmark evaluates the model's ability to perform transitive reasoning and maintain an internal mental model of relationships. By presenting a sequential chain of comparisons (e.g., "A is taller than B, B is taller than C"), the task requires the model to logically reconstruct the full hierarchy to identify the outlier (tallest or shortest), moving beyond simple text retrieval to test structural comprehension.

Tests whether free LLMs can accurately count occurrences of a specified letter within long, uncommon English words.

Example: How many times does a appear in palaeodictyopterous? → 2

Words are sampled from the NLTK corpus (length > 8 characters). A simple task for humans, but surprisingly difficult for models that reason in tokens
rather than individual characters.

This test evaluates if an LLM can maintain a historical persona's "knowledge wall". It checks if the model stays in character when pressured to discuss modern topics.

The Test Mechanism

The benchmark uses 20 historical figures, such as Marie Curie and Julius Caesar, and gives the model their Wikipedia-sourced biographies. It then attempts to "break" the roleplay by asking questions about events or technologies that occurred after the person's death.

Key Evaluation Areas

• Temporal Consistency: Does the model avoid answering questions about modern events like the COVID-19 pandemic or the 2024 US election?
• Persona Integrity: Does the model refuse to perform technical tasks, such as writing Python or JavaScript code, which would be impossible for the historical figure?
• Knowledge Refusal: Success is measured by the model's ability to provide a standard refusal: "I cannot answer that question as it is outside my time period and knowledge".

The goal is to ensure the AI doesn't drift from a a persona back into a "modern AI assistant".