You are browsing a version that has not yet been released.
Maintained
3.1 (upcoming) 3.0.1 (current)
Unmaintained
2.1 2.0 1.2 1.1 1.0

Introduction

Doctrine Lexer is a library that can be used in Top-Down, Recursive Descent Parsers. This lexer is used in Doctrine Annotations and in Doctrine ORM (DQL).

To write your own parser you just need to extend Doctrine\Common\Lexer\AbstractLexer and implement the following three abstract methods.

1/**
 * Lexical catchable patterns.
 *
 * @return string[]
 */
abstract protected function getCatchablePatterns();

/**
 * Lexical non-catchable patterns.
 *
 * @return string[]
 */
abstract protected function getNonCatchablePatterns();

/** Retrieve token type. Also processes the token value if necessary. */
abstract protected function getType(string &$value): int;


2


3


4


5


6


7


8


9


10


11


12


13


14


15


16

These methods define the lexical catchable and non-catchable patterns and a method for returning the type of a token and filtering the value if necessary.

The Lexer is responsible for giving you an API to walk across a string one character at a time and analyze the type of each character, value and position of each token in the string. The low level API of the lexer is pretty simple:

  • setInput($input) - Sets the input data to be tokenized. The Lexer is immediately reset and the new input tokenized.
  • reset() - Resets the lexer.
  • resetPeek() - Resets the peek pointer to 0.
  • resetPosition($position = 0) - Resets the lexer position on the input to the given position.
  • isNextToken($token) - Checks whether a given token matches the current lookahead.
  • isNextTokenAny(array $tokens) - Checks whether any of the given tokens matches the current lookahead.
  • moveNext() - Moves to the next token in the input string.
  • skipUntil($type) - Tells the lexer to skip input tokens until it sees a token with the given value.
  • isA($value, $token) - Checks if given value is identical to the given token.
  • peek() - Moves the lookahead token forward.
  • glimpse() - Peeks at the next token, returns it and immediately resets the peek.