HEXOLANGGitHub ↗

Lexer

Reads source characters and produces tokens.

The lexer reads source code and splits it into tokens: identifiers, integer and decimal numbers, strings, mathematical and logical operators, keywords, symbols and delimiters.

Each token stores its type, its contents and its position in the source, including the line number. Identifiers are interned (string interning), so equal names share one canonical representation.

Last updated 2026-11-21Suggest an edit ↗