deduplicate
There are 43 repositories under deduplicate topic.
knjcode/imgdupes
Identifying and removing near-duplicate images using perceptual hashing.
tc39/proposal-array-unique
ECMAScript proposal for Deduplicating method of Array
evrignaud/fim
File Integrity Manager -
shevchenkoartem/lastfm-smart-deduper
JS script that allows you to remove duplicates from your Last.fm scrobbles library.
sysulq/dataloader-go
Go implementation of Facebook's DataLoader with 200+ lines of code.
WSH032/sd-webui-fast-dataset-maker
A funny extension that integrates image-browsing , downloader , deduplicate , cluster , can quickly collect, classify and process your images. | 一个有趣的扩展，整合了 图库，下载，去重，聚类 ，可以快速搜集、分类、处理你的图片。
thijse/DSPhotoSorter
A command line tool for sorting photo's from the Synology DSPhoto auto-upload tool
zapalote/js-deduplicate-array-benchmark
Benchmarking javascript array operations
routineLife1/AVFDU
动漫一拍N自动识别算法
svandriel/cachify-promise
Smart caching for promises. Like memoization, but better.
infogulch/uniq
Package uniq provides primitives for getting the first unique elements of (aka deduplicate) your existing sorted sort.Interface.
yaroslaff/hashget
Deduplication/backup tool with extremely high 'compression' rate
bmiller1009/deduper
General deduping engine for JDBC sources with output to JDBC/csv targets
mterron/swuniq
A command-line tool for deduplicating entries in a file or stream with constant memory usage
rsalmei/refine
Refine your file collection using Rust!
gblach/reflicate
Deduplicate data by creating reflinks between identical files.
mahendraHegde/node-idempotency
makes any request idempotent across nodejs frameworks like nestjs, express, fastify
mimose/deduplicate
😺一款用于防止重复攻击的组件，基于SPI机制实现核心功能 >>> 防重放组件 (A component used to prevent duplicated attacks, such as repeat request, replay attack)
ojarva/maildir-deduplicate
Deduplicates maildir contents using hard links.
helloall1900/vhash
A C++ reimplementation of Near Duplicate Video Detection - Get a 64-bit comparable hash-value for any video (Video Hash).
routineLife1/AFI-ForwardDeduplicate
Efficient Deduplicate for Anime Video Frame Interpolation
jaymoulin/google-musicmanager-dedup-api
Deduplication API for Google MusicManager
jshwi/borgini
ini config for borg backup
rongrimes/zipfile-dedup
Project to take two similar zipfiles, and to dedupe files that have the same tiemstamp in the older file.
stdlib-js/iter-union
Create an iterator which returns the union of two or more iterators.
stdlib-js/iter-unique
Create an iterator which returns unique values.
stdlib-js/iter-unique-by
Create an iterator which returns unique values according to a predicate function.
stdlib-js/iter-unique-by-hash
Create an iterator which returns unique values according to a hash function.
hekmon/deduper
Analyse 2 paths to found identical files and hard link them to save space
stdlib-js/array-base-dedupe
Remove consecutive duplicated values.
stdlib-js/array-base-to-deduped
Copy elements to a new generic array after removing consecutive duplicated values.
stdlib-js/iter-dedupe
Create an iterator which removes consecutive duplicated values.
stdlib-js/iter-dedupe-by
Create an iterator which removes consecutive values that resolve to the same value according to a provided function.
stdlib-js/iter-intersection
Create an iterator which returns the intersection of two or more iterators.
stdlib-js/iter-intersection-by-hash
Create an iterator which returns the intersection of two or more iterators according to a hash function.
Exisi/UText
多行与行内文本去重工具