3.6. The Parser and Tokeniser

As mentioned above, the first stage in Perl's operation is to "understand" your program. This is done by a joint effort of the tokeniser and the parser. The tokeniser is found in toke.c, and the parser in perly.c. (although you're far, far better off looking at the YACC source in perly.y)

The job of the tokeniser is to split up the input into meaningful chunks, or tokens, and also to determine what type of thing they represent - a Perl keyword, a variable, a subroutine name, and so on. The job of the parser is to take these tokens and turn them into "sentences", understanding their relative meaning in context. We'll examine their operation in more detail in Chapter 5.