nemo-curator Skill
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or d Published by NousResearch in hermes-agent.
What is nemo-curator Skill?
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or d Published by NousResearch in hermes-agent. This profile combines repository metadata with install, compatibility, and usage signals so developers can quickly decide whether it fits their agent workflow before opening the source repository.
Automated repository signals based on public metadata such as recency, license, installation evidence, and adoption. These are not a security audit or endorsement.
Key capabilities
- Includes SKILL.md support
- Reusable instructions support
- Data analysis
- Design and media
- Data analysis use cases
- Design and media use cases
Technical details
- Install or run with Copy skill directory
When to use nemo-curator Skill
- Use it for data analysis.
- Use it for design and media.
Built with
Editorial notes
Source
- Creator: NousResearch
- Repository: NousResearch/hermes-agent
- Skill file: optional-skills/mlops/nemo-curator/SKILL.md
What it does
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or d
Skill instructions
NeMo Curator - GPU-Accelerated Data Curation NVIDIA's toolkit for preparing high-quality training data for LLMs. When to use NeMo Curator Use NeMo Curator when: - Preparing LLM training data from web scrapes (Common Crawl) - Need fast deduplication (16× faster than CPU) - Curating multi-modal datasets (text, images, video, audio) - Filtering low-quality or toxic content - Scaling data processing across GPU cluster Performance: - 16× faster fuzzy deduplication (8TB RedPajama v2) - 40% lower TCO vs CPU alternatives - Near-linear scaling across GPU nodes Use alternatives instead: - datatrove: CPU-based, open-source data processing - dolma: Allen AI's data toolkit - Ray Data: General ML data processing (no curation focus) Quick start Installation bash Text curation (CUDA 12) uv pip install "nemo-curator[textcuda12]" All modalities uv pip install "nemo-curator[allcuda12]" CPU-only (slower) uv pip install "nemo-curator[cpu]" Basic text curation pipeline python from nemocurator import ScoreFi
Explore related resources
Frequently asked questions
What is nemo-curator?
nemo-curator is a open-source AI agent skill with Copy skill directory. GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW.
Who is nemo-curator best for?
nemo-curator is best for reusing agent instructions, scripts, and references, data analysis workflows, design and media workflows.
How do I install nemo-curator?
Install or run nemo-curator using Copy skill directory. Check nemo-curator for the latest setup command.
Is nemo-curator actively maintained?
nemo-curator may need a closer maintenance check before production use.
Auto-fetched from GitHub.
