semantic-lex-tokens

semantic-lex-tokens is a variable defined in `lex.el'.
Its value is shown below.

Documentation:
An alist of semantic token types.
As of December 2001 (semantic 1.4beta13), this variable is not used in
any code. The only use is to refer to the doc-string from elsewhere.

The key to this alist is the symbol representing token type that
M-x semantic-flex returns. These are

- bol: Empty string matching a beginning of line.
This token is produced with
`semantic-lex-beginning-of-line'.

- charquote: String sequences that match `\s\+' regexp.
This token is produced with `semantic-lex-charquote'.

- close-paren: Characters that match `\s)' regexp.
These are typically `)', `}', `]', etc.
This token is produced with
`semantic-lex-close-paren'.

- comment: A comment chunk. These token types are not
produced by default.
This token is produced with `semantic-lex-comments'.
Comments are ignored with `semantic-lex-ignore-comments'.
Comments are treated as whitespace with
`semantic-lex-comments-as-whitespace'.

- newline Characters matching `\s-*\(
\|\s>\)' regexp.
This token is produced with `semantic-lex-newline'.

- open-paren: Characters that match `\s(' regexp.
These are typically `(', `{', `[', etc.
If `semantic-lex-paren-or-list' is used,
then `open-paren' is not usually generated unless
the `depth' argument to M-x semantic-lex is
greater than 0.
This token is always produced if the analyzer
`semantic-lex-open-paren' is used.

- punctuation: Characters matching `{\(\s.\|\s$\|\s'\)'
regexp.
This token is produced with `semantic-lex-punctuation'.
Always specify this analyzer after the comment
analyzer.

- semantic-list: String delimited by matching parenthesis, braces,
etc. that the lexer skipped over, because the
`depth' parameter to M-x semantic-flex was not high
enough.
This token is produced with `semantic-lex-paren-or-list'.

- string: Quoted strings, i.e., string sequences that start
and end with characters matching `\s"'
regexp. The lexer relies on @code{forward-sexp} to
find the matching end.
This token is produced with `semantic-lex-string'.

- symbol: String sequences that match `\(\sw\|\s_\)+'
regexp.
This token is produced with
`semantic-lex-symbol-or-keyword'. Always add this analyzer
after `semantic-lex-number', or other analyzers that
match its regular expression.

- whitespace: Characters that match `\s-+' regexp.
This token is produced with `semantic-lex-whitespace'.

Value:

((bol)
 (charquote)
 (close-paren)
 (comment)
 (newline)
 (open-paren)
 (punctuation)
 (semantic-list)
 (string)
 (symbol)
 (whitespace))