Lexer Module
Links:
Tips and FAQ
| Sources Description
| Lexer Generator Tools
| Compatibility
| Editor Bridge
Description
This is home page of the lexer module.
Lexer module allows to build and maintain a list of tokens over a textual input.
The input can be an immutable character sequence (java.lang.CharSequence)
(or a stream of characters provided by java.io.Reader)
or a mutable text input such as javax.swing.text.Document.
For mutable inputs the lexer framework updates token list automatically
as the text modifications are being done.
Lexer module is not a generator of lexical scanners (like e.g. JavaCC or Antlr).
Instead various lexical scanners can be plugged into the lexer module to perform incremental updates
of token lists for mutable input sources.
New API
A new API of the Lexer module has passed final API review and it is now present in CVS trunk.
It supports
Language Embedding,
Token Hierarchy Snapshots and other useful features
described
here.
Details including checkout instructions can be found
here.
Usecases
- Syntax coloring - token identification names are used to find the coloring information
- Brace matching
- Simple code completion or code folding analysis
- Parsing - lexical analysis in the parser may be skipped
Advantages over the Original Lexical Support
- Bidirectional iteration over token list with no cost
- Smaller token updates after document modification
- Optimized handling of large tokens
- Language embedding greatly simplified
- Support for lexer generators
- Easier writing of handcoded lexers
- Extensive testing support including random modification tests
Lexer Generators
Batch lexers used in the lexer module can be generated using popular java
lexer generators such as JavaCC
or Antlr. For the new API the support for lexer generators is being developed but it's not finished yet.
Compatibility
The lexer should be fully integrated into Editor APIs in NetBeans 6.0.
However it can be used as well with previous NetBeans releases
by using
Editor Bridge module - see the
New API for more information.
The lexer can be used with J2SE 1.5 and newer releases.
