|
||||||||||
PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES |
Interface Summary | |
---|---|
Tokenizer | Basic methods to implement for a brand-new Tokenizer . |
Class Summary | |
---|---|
FixedTokenizer | Splits a text into a list of fixed-size tokens. |
TermTokenizer | Splits a text into a list of tokens, '\n' is added. |
URLTokenizer | A Tokenizer for a list of URLs. |
|
||||||||||
PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES |