Token

data class Token( val tokenType: TokenType, val lexeme: String, val line: Int, val column: Int, val value: String? = null)

The smallest meaningful unit of information in the input.

A list of tokens is generated during lexical analysis and used to build an AST during syntax analysis.

Parameters

value

Some token types, such as TokenType.NUMBER have a value associated with them

fun Token( tokenType: TokenType, lexeme: String, line: Int, column: Int, value: String? = null)

the column number in the input on which this token was found

the exact string from the source from which this token was generated

the line number in the input on which this token was found

the type of this token

val value: String? = null