elluminate
Agentic evaluation. Offline scoring, experiment comparison, and quality gates before anything reaches production.
Learn more
Frontier AI work for regulated industries.
We evaluate AI systems, build the data and environments behind them, and bring them into production for health insurance, banking, and public administration.
Made in Germany · working with
Made in Germany working with
The products.
Evaluate AI systems before they ship, and test the interfaces they live in.
Accessibility Agent
BITV & BFSG accessibility testing for the interfaces your agents live in.
Learn more
Our platform: Uniting reliability dimensions
From agentic workflows to EU AI Act compliance: Evaluate, measure, and improve your AI in one place.
- Eval Agent
- Run experiments
- Track progress
- Analyze results
- Compliance
Latest research artifacts
All research Model | Dataset | Paper arXiv:2602.12414
propella-1
A family of small multilingual LLMs for annotating text documents across six categories: core content, classification, quality and value, audience and purpose, safety and compliance, and geographic relevance. These annotations help filter, select, and curate LLM training data at scale. The models outperform much larger general-purpose baselines.
DATA-FM @ ICLR 2026 (Spotlight) 57 languages Any text, any format
View on Hugging Face Model | Paper arXiv:2601.08472
sui-1: Summarization with Unique Identifiers
A 24B-parameter LLM for abstractive summarization with inline citations. Every claim is traceable to its source sentence. It supports documents with more than 2M tokens and outperforms all tested open-weight baselines, including models with 3x more parameters.
24B params 5 languages inline citations
View on Hugging Face Benchmark Apache-2.0
base-eval
Curated lm-evaluation-harness task configurations for evaluating English and German base models. Every task is validated against reference models, enabling benchmark suites for early-stage pretraining and in-loop evaluation.
English German lm-eval
View on GitHub Tool Apache-2.0
inference-hive
Distributed LLM inference at scale for SLURM clusters. Configure cluster, server, and data settings, then scale across thousands of GPUs with near-linear throughput.
SLURM native OpenAI API vLLM / SGLang
View on GitHub Latest articles
All articles
Benedikt Droste 9 min read
HuggingFace Lite: When Agents Share Passwords and Build Password Crackers
Agents shared passwords, tried public lookups and wrote code to guess missing passwords. An invoice-review test shows what completion reports can leave out.
AI Agents AI Safety
+1 more
Maximilian Idahl 20 min read
Tokenization Tax Report 2026
We measured how many tokens 20 current model tokenizers need for identical text across different languages and text types, including proprietary (Claude, GPT, Gemini, Grok) and open-weight models. Some are great, others will make you pay a token tax. For example, Claude 4.7+ needs twice as many tokens for German as GPT-6 Astra.
Tokenizers Multilingual
+2 more
Benedikt Droste 10 min read
Terminal-Bench 4: When Codex Stops and Claude Keeps Checking
Claude Opus 5 performs best overall in the public Terminal-Bench 4 results. GPT-5.6 Sol costs less and matches it on business operations. Their traces show why.
Terminal-Bench Claude Code
+3 more
Careers at ellamind
We're hiring people who want to build AI that stands up to reality: regulation, scale, and responsibility. Open positions available across engineering, AI, product, and sales.
Unlock the power of AI
See how our products help you evaluate AI systems and test digital accessibility.