Package it.unipi.di.tokenizer

Interface Summary
Tokenizer Basic methods to implement for a brand-new Tokenizer.
 

Class Summary
FixedTokenizer Splits a text into a list of fixed-size tokens.
TermTokenizer Splits a text into a list of tokens, '\n' is added.
URLTokenizer A Tokenizer for a list of URLs.