New Lexer API
Version: 1.0
Author: Miloslav Metelka, Sun Microsystems/NetBeans
Document History:
available in CVS
Checkout and Building Instructions
The new API is present in the CVS trunk since October 04, 2006 and the lexer and lexer/editorbridge
are part of the stable build.
The following statement will checkout stable NB modules including the lexer module:
cvs co -P stable
(stable_nowww can be used as well to not checkout www modules' subdirs)
Calculator syntax coloring example
The lexer module can be used for syntax coloring with the current editor module or previous releases (NB 5.0 should work but JDK 1.5 must be used) by using lexer/editorbridge module.
The calculator example is in lexer/editorbridge/calc module.
Building
The following modules should be built automatically by standard NB building in nbbuild:
- editor/util - lexer uses some utility classes and methods from this module
- lexer - main lexer API, SPI and infrastructure
- lexer/editorbridge - module allowing to do syntax coloring by the lexer module over existing editor
- java/lexer - optional; current implementation of Java language lexer
and Javadoc embedded languge and java string literal embedded language.
Temporarily there is an implementation of HTML lexer as well but it will be moved to html/lexer.
The following modules require a manual build:
- lexer/editorbridge/calc - optional; example using lexer for syntax coloring of calculator source files. It uses a generic syntax coloring layer (
LexerLayer) from lexer/editorbridge
The following commands may be used:
cd lexer/editorbridge/calc; ant
Testing the Calculator example
When running the IDE the Ctrl-N should have registered "Calc File" template under "Other" category.
The file may be created and opened. It should be syntax-colored.
Changes to the Original API
- Language Embedding support
- Token Hierarchy snapshots creation by
TokenHierarchy.createSnapshot().
- API and SPI separation. Only expose what's really necessary for API/SPI clients.
Lexer and LexerInput moved from API to SPI.
SampleTextMatcher removed completely. The flyweight tokens creation
is controlled directly by the corresponding lexer.
TokenUpdater renamed to TokenListUpdater and moved
to implementation.
swing subpackage of the API was removed.
- Character preprocessing (e.g. Unicode escape sequences) handling by
CharPreprocessor
- Simplified writing of the handcoded lexers
