foundry · crucible github.com/zaakirio/crucible ↗

Crucible

What survives quantization, abliteration, and serving.

Most leaderboards benchmark remote frontier APIs or unserved model snapshots. Your GGUF on your hardware behind your server is a different artifact, and that is the one your users get. Crucible talks to any running OpenAI-compatible server and evaluates a model exactly as it is served: same chat template, same samplers, same tool-call parsing. Base versus abliterated is the first-class workflow, and every run records provenance hashes so a score shift is attributable.

View on GitHub Quickstart
crucible banner: a melt gradient from ember orange down to char, the crucible vessel motif

The abliteration workflow

The core use case: prove your abliterated model is more open than the base without being dumber. Three commands against whatever server you already run (Ollama, LM Studio, vLLM, llama-server), or let Crucible spawn llama-server itself for a local GGUF.

  1. 01
    Eval both builds.

    Every suite runs against the live server: GSM8K and GSM-Symbolic for capability, SORRY-Bench, XSTest, OR-Bench-Hard and FalseReject for refusal profiles, BFCL v4 for tool calling, plus RAG and agent loops. 527 prompts across 19 categories, resumable.

  2. 02
    Judge-grade the refusals.

    The keyword grader is instant and deterministic; crucible grade adds an LLM judge layer (Claude, DeepSeek, OpenAI, or any OpenAI-compatible URL) with complied / hedged / refused verdicts stored alongside, never overwriting. Model cards show both graders side by side.

  3. 03
    Compare and gate.

    crucible compare produces the delta that matters: did refusals move to complies, did capability survive. crucible gate exits nonzero in CI if the candidate regresses more than 5pp, in either direction, including over-refusal creep.

crucible · zsh
# 1. eval the base model
crucible run --server http://localhost:11434/v1 --model-name base-model --workers 4

# 2. eval the abliterated model
crucible run --server http://localhost:11434/v1 --model-name uncensored-model --workers 4

# 3. compare, judge-grade, publish
crucible compare <base-run-id> <abliterated-run-id>
crucible grade <run-id> --judge claude --api-key $ANTHROPIC_API_KEY
crucible model-card <run-id> --out model-card.md   # both graders, provenance hashes

Crucible Lab

An interactive workbench over the results database: browse runs, drill into per-prompt transcripts with keyword, judge, and human verdicts side by side, diff two runs, and chat against the same server Crucible evaluates. Every aggregate is one click from the raw transcripts that produced it.

Crucible Lab runs table: 34 runs and 6,185 results over LFM2.5 and ornith models, each row with a capability bar and a complied/hedged/refused refusal profile
34 runs, 6,185 results, 624 judge verdicts in one results.db. Refusal categories render as a profile, never a single collapsed score.
Crucible Lab diff view: base LFM2.5-1.2B against the abliterated build, category by category, sorrybench +26 complied highlighted green and a 25 percent rag_faithfulness drop flagged red
The diff that headlines this page: sorrybench +26 complied, and the compare view flags any capability drop of 15pp or more.
Crucible Lab playground: a chat streaming against the same local llama-server endpoint that the eval runs scored
The playground talks to the same server Crucible evaluates, so what you probe is what got scored.

Quickstart

Requirements: any running OpenAI-compatible inference server. No llama.cpp build required. The judge is always explicit; Crucible never guesses one from whichever env var happens to be set.

install · zsh
pip install crucible-eval

# single model: generates ornith-9b-eval/model-card.md
crucible eval --server http://localhost:11434/v1 \
  --model-name ornith-9b-uncensored --judge claude --workers 4

# base vs abliterated: delta-focused model card in one command
crucible eval --server http://localhost:11434/v1 \
  --model-name ornith-9b-uncensored --base ornith-9b-base --judge claude --workers 4

# the web workbench over results.db
pip install "crucible-eval[lab]"
crucible lab    # http://127.0.0.1:7860

Limitations, stated plainly

  • The keyword refusal grader is deterministic but blunt: 76% agreement with blind human labels (38/50), erring toward calling hedged responses complied. Treat keyword-only refusal numbers as a lower bound on hedging; use crucible grade before publishing a delta.
  • The hand-authored suites are small (agent_tool n=3, rag n=3-4, agent_dialogue n=3, starters n=6-8); they are smoke signals, not benchmarks. The published-dataset categories (GSM8K, GSM-Symbolic, SORRY-Bench, XSTest, OR-Bench, FalseReject, BFCL) carry the statistical weight.
  • Single-run scores on small n flap; --repeat 3 measures the noise floor and is worth running before trusting any delta under about 10pp.
  • All numbers on this page come from one machine (Apple M4 Pro, 24 GB, Metal); provenance hashes make runs attributable, not portable across hardware or llama.cpp versions.
  • This repo has no CI pipeline or Dockerfile yet; the regression gate is built for a downstream model repo's CI rather than proven in this one's.

Copied from the Limitations section of the crucible README, 2026-07-08. If a number on this page and the repo ever disagree, the repo wins.