tok.tfilesource file

Tokenizer - customizable tokenizer class for use with the intrinsic class 'grammar-production' parser.

This tokenizer implementation is parameterized with a set of rules (see below); a basic set of rules is provided, but users can customize the tokenizer quite extensively simply by subclassing the Tokenizer class and overriding the 'rules_' property with a new set of rules declarations.

Summary of Classes  

Tokenizer  TokenizerError  TokErrorNoMatch 

Summary of Global Objects  

(none)

Summary of Global Functions  

main 

Global Functions  

main (args)tok.t[396]

Test Section

TADS 3 Library Manual
Generated on 5/16/2013 from TADS version 3.1.3