gotokenizer

github.com ↗

A tokenizer based on the dictionary and Bigram language models for Golang. (Now only support chinese segmentation)