Recognizing tokens from characters
Source text consists of characters; a lexer recognizes tokens such as identifiers, numbers, keywords, and punctuation. A token can carry its category together with a spelling, value, or location needed by later processing.
LLVM’s Kaleidoscope example returns numeric values with number tokens and names with identifier tokens. Its treatment of whitespace and comments belongs to that example language, not a universal rule that whitespace never matters.
References: [1]
Overlapping matches need a selection rule
Input can match several patterns: a keyword may also satisfy an identifier pattern. A lexer needs a defined choice among them rather than treating successful pattern recognition as a unique classification.
A flex scanner selects the longest match and resolves equal-length ties by the earlier rule. Pattern order can therefore affect results. Such decisions are part of the scanner’s specified contract, not properties supplied by regular expressions alone.
References: [2]
Recognized tokens still need structure
Recognizing numbers and operators does not establish which operation groups first. Parsing supplies hierarchical organization, and later checks can resolve names and types.
Lexing also needs invalid-input handling. LLVM explicitly notes incomplete validation in its tutorial’s numeric scanner. A small teaching implementation demonstrates a mechanism without establishing that every malformed input or full-language rule has been handled.