Interprets the characters of EDI input as a series of tokens.