duplicut

github.com ↗

Quickly remove duplicates, without changing the order, and without getting OOM on huge wordlists.