Uses of Interface
it.unipi.di.tokenizer.Tokenizer

Packages that use Tokenizer
it.unipi.di.textdb   
it.unipi.di.tokenizer   
 

Uses of Tokenizer in it.unipi.di.textdb
 

Methods in it.unipi.di.textdb with parameters of type Tokenizer
static TextDB BucketedHuffword.build(Tokenizer tokenizer, String inputfile, String outfile, int bucketLen, PrintStream log)
          Compresses the input file with Bucketed Huffword using a set of custom parameters.
static TextDB RSHuffword.build(Tokenizer tokenizer, String inputfile, String outfile, PrintStream log, boolean withFields, char separator)
          Compress the input file with the bucketed huffword technique using customized parameters.
 

Uses of Tokenizer in it.unipi.di.tokenizer
 

Classes in it.unipi.di.tokenizer that implement Tokenizer
 class FixedTokenizer
          Splits a text into a list of fixed-size tokens.
 class TermTokenizer
          Splits a text into a list of tokens, '\n' is added.
 class URLTokenizer
          A Tokenizer for a list of URLs.