it.unipi.di.tokenizer
Interface Tokenizer
- All Known Implementing Classes:
- FixedTokenizer, TermTokenizer, URLTokenizer
public interface Tokenizer
Basic methods to implement for a brand-new Tokenizer
.
- Author:
- Claudio Corsi, Paolo Ferragina
Method Summary |
void |
close()
|
it.unimi.dsi.mg4j.util.MutableString |
next()
|
void |
reset()
|
next
it.unimi.dsi.mg4j.util.MutableString next()
throws IOException
- Throws:
IOException
close
void close()
throws IOException
- Throws:
IOException
reset
void reset()
throws IOException
- Throws:
IOException