Sign in
Explore
Skills · SkillData & AIMachine learningLLM apps & RAGReviewedRuns shell commandsNeeds secrets

huggingface-community-evals

Runs evaluations of Hub models with inspect-ai and lighteval on local hardware, including the choice between vLLM, Transformers and accelerate.

View on GitHub
Kind
Skill
Works with
Claude Code
Source
Hugging Face Skills @ 25910b3d0ba3
Path
skills/huggingface-community-evals/SKILL.md
Jobs
Testing
Topics
Machine learning, LLM apps & RAG

Credit

Made by
huggingface
Original
github.com/huggingface/skills/blob/25910b3d0ba31a2ba102f901cd…
License
Apache-2.0

Listed by Skill Manifest; not made by us.

InstallInstall it by hand, following the source.
hf skills add huggingface-community-evals
Install factsScanned 2 Nov 2026
Adds
  • .claude/skills/huggingface-community-evals/ (the skill folder)
  • 4 file(s) the skill refers to
Runs scripts
Yes
  • scripts/inspect_eval_uv.py
  • scripts/inspect_vllm_uv.py
  • scripts/lighteval_vllm_uv.py
  • shell block in the text
Uses the network
Yes
  • downloads Python packages
Needs secrets
Yes
  • HF_TOKEN
Reads environment variables
  • HF_TOKEN
Source revision
25910b3d0ba3
Licence
Apache-2.0
Last activity
30 Oct 2026
Review
Reviewed

Found by a static scan of the stored text; nothing is run. “None found” is not a guarantee, so read what you install before you run it.

  • Only the main file is read; other files in the folder are not scanned
# Overview

This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**.

It covers:
- `inspect-ai` with local inference
- `lighteval` with local inference
- choosing between `vllm`, Hugging Face Transformers, and `accelerate`
- smoke tests, task selection, and backend fallback strategy

It does **not** cover:
- Hugging Face Jobs orchestration
- model-card or `model-index` edits
- README table extraction
- Artificial Analysis imports
- `.eval_results` generation or publishing
- PR creation or community-evals automation

If the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill.

If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that publishing step to `~/code/community-evals`.

> All paths below are relative to the directory containing this `SKILL.md`.

# When To Use Which Script

| Use case | Script |
|---|---|
| Local `inspect-ai` eval on a Hub model via inference providers | `scripts/inspect_eval_uv.py` |
…