MinishLab/semhash
by {"name"=>"Tom Doerr", "url"=>"https://github.com/tom-doerr"}
Semantic text deduplication tool
tags:
View on index · View in 3D Map
Related repos
- tool for finding and removing duplicate images in big datasets
- Lock-free deduplication cloud backup tool
- cross-platform tool to find and manage duplicate files
- Probabilistic record linkage and deduplication at scale
- finds similar code and patterns across files
- Plain text spaced repetition system
- Pipeline with a web UI for LLM training and inspection
- process, filter, and deduplicate large-scale text data with customizable pipelines
- Finds and removes file duplicates
- Splits text into semantically meaningful chunks
