Unstructured Entity Extraction (NER)

This benchmark evaluates the model's ability to perform Named Entity Recognition (NER) and adhere to a strict JSON schema. By embedding synthetic identity details (Name, Email, Job, City) within randomized, noisy, and unstructured text blocks, the task challenges the model to filter out irrelevant signals (such as random IDs or status codes) and map the correct entities to specific JSON keys without hallucination.

Structured Data Transformation (CSV to JSON)

This benchmark evaluates the model's proficiency in syntax translation and data serialization. By supplying a flat, delimited text format (CSV) and requiring a hierarchical key-value output (JSON), the task tests the model's ability to correctly map headers to keys, handle data types (strings, dates), and produce syntactically valid JSON without losing information or hallucinating values.

Phone Number Standardization

This benchmark evaluates the model's ability to perform data cleaning and pattern normalization. By providing input data with inconsistent delimiters (parentheses, dots, dashes), the task requires the model to extract the core numerical entities and restructure them into a strict international standard (+1 XXX XXX XXXX), testing its precision in string manipulation and adherence to rigid output constraints.

Tests whether free LLMs can accurately detect and mask Personally Identifiable Information in text, limited to four entity types: first name, last
name, email address, and IPv4 address.

Example:

Input: "Contact john.doe@example.com or reach out to John Doe at 192.168.1.1"
Output: "Contact [EMAIL] or reach out to [FIRSTNAME] [LASTNAME] at [IPV4]"

A practical privacy-focused benchmark — models must identify sensitive entities precisely without over-masking surrounding context.