webmagic

github.com ↗

Scalable crawler with downloading, url management, content extraction and persistent.