Token and Lexeme in Compiler

In this class, We discussed Token and Lexeme in Compiler.

For Complete YouTube Video: Click Here

The reader should have prior knowledge of lexical analysis. Click Here.

The below example program is given as input to the lexical analysis phase.

The lexical analysis phase will separate main as a word.

Main – lexeme.

Next, it will separate ‘(‘ from the source code.

‘(‘ is a lexeme.

Similarly, all the symbols, keywords, and identifiers are separated.

Lexeme: is a sequence of characters.

Each lexeme is assigned with a token.

For example: if the lexeme is ‘if.’ The lexeme ‘if’ is a keyword, and this lexeme has a token name and attribute.

The below table shows the lexeme and taken table.

For keywords, we do not have any token attributes.

we give <if> token for if lexeme.

In our program, ‘i’ is taken as a lexeme.

‘i’ is an identifier. The pointer to the symbol table is the attribute.

We give the token <id, 1> for identifier ‘i’.

The identifier details are placed in the first line of the symbol table.

Similarly, for the relational operator, less than.

We give <relop, LT > token for symbol ‘<. ‘

The lexical analysis phase will take the source code as input and generate tokens as output.