Package lexer

Types

Generates a list of tokens from the given input. This is usually the first step when parsing code.

data class Token( val tokenType: TokenType, val lexeme: String, val line: Int, val column: Int, val value: String? = null)

The smallest meaningful unit of information in the input.

Represents the type of Token