Package com.tangosol.coherence.dsltools.precedence

Class OPToken

java.lang.Object
com.tangosol.coherence.dsltools.precedence.OPToken
Direct Known Subclasses:
BetweenOPToken, ColonToken, ContainsOPToken, IdentifierOPToken, InfixOPToken, InfixRightOPToken, KeywordOPToken, LiteralOPToken, NestingOPToken, NotOPToken, PathOPToken, PeekOPToken, PrefixOPToken, PunctuationOPToken
public class OPToken extends Object
OPToken is the root class for the Top Down Operator Precedence Parsing framework's tokens. This framework was first done by Vaughan Pratt in 1973 an is undergoing a bit of a renaissance with people that find the typical formal grammer tool a bit to heavyweight. The fundamental idea behind "Pratt Parsers" is that Tokens are objects that posses methods that allow them to make precedence decisions, match other tokens, and build abstract syntax trees (AST). The central issue of the precedence problem is that given an operand object between two operators, should the operand be bound to the left operator or the right operator? obj1 OP1 obj2 OP2 obj3 like: (1 + 2 * 3) Does obj2 bind to OP1 or to OP2? The technique we will use has Token objects "know" their precedence levels, and implement methods called "nud" (the null denotation in Pratt speak) and "led" (the left denotation). A nud method "should" have no interest in the tokens to the left while a "led" method does. A nud method is typically used on values such as variables and literals and by prefix operators like '-', or 'not'. A led method is used by infix operators and suffix operators. A token will often have both a nud method and a led method with the canonical example being '-' which is both a prefix operator and an infix operator. The heart of Pratt's technique is the "expression" function. It takes a right binding power that controls how aggressively that it should bind to tokens on its right. We also pass the parser to the token objects so that their functions may look at context and have access to the tokenizer.
Author:
djl 2009.03.14
See Also:

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static String
    BINARY_OPERATOR_NODE
    The AST node name for a Binary Operator node.
    static String
    BINDING_NODE
    The AST node name for a Binding Node.
    static String
    CALL_NODE
    The AST node name for a Method Call Node.
    static String
    DEREF_NODE
    The AST node name for a De-referencing Node.
    static String
    FIELD_LIST
    The AST node name for a Field List Node.
    static String
    IDENTIFIER_NODE
    The AST node name for an Identifier Node.
    static String
    LIST_NODE
    The AST node name for a List of Values Node.
    static String
    LITERAL_NODE
    The AST node name for a Literal Node.
    protected int
    m_nBindingPower
    The power that this token binds, typically to the left.
    protected String
    m_sLedASTName
    A functor for building ast for led method.
    protected String
    m_sNudASTName
    A functor for building ast for nud method.
    protected String
    m_sValue
    The string value of the literal.
    static final int
    PRECEDENCE_ASSIGNMENT
    The binding precedence for assignment operators assignment such as = += -= *= /= %= &= ^= |= <<= >>= >>>=
    static final int
    PRECEDENCE_BITWISE
    The binding precedence for bitwise operators such as >>> >> and <<
    static final int
    PRECEDENCE_EXPONENT
    The binding precedence for exponent arithmetic, i.e. raising by a power
    static final int
    PRECEDENCE_IDENTIFIER
    The binding precedence for identifier tokens
    static final int
    PRECEDENCE_KEYWORD
    The binding precedence for keyword tokens
    static final int
    PRECEDENCE_LOGICAL
    The binding precedence for logical tokens such as &&, ||, etc
    static final int
    PRECEDENCE_LOGICAL_BITWISE
    The binding precedence for bitwise logical tokens such as &, |, ^ etc
    static final int
    PRECEDENCE_PARENTHESES
    The binding precedence for parentheses ( ) [ ]
    static final int
    PRECEDENCE_PRODUCT
    The binding precedence for product arithmetic, multiplication, division, mod * / %
    static final int
    PRECEDENCE_RELATIONAL
    The binding precedence for relational operators such as ==, <=, like, contains etc
    static final int
    PRECEDENCE_SUM
    The binding precedence for sum arithmetic, i.e. + and -
    static final int
    PRECEDENCE_UNARY
    The binding precedence for other unary operators: pre-increment, pre-decrement, plus, minus, logical negation, bitwise complement, type cast ++expr --expr +expr -expr !
    static final int
    PRECEDENCE_UNARY_POST
    The binding precedence for unary post operators such as post-increment and post-decrement of the form expr++ or expr--
    static String
    UNARY_OPERATOR_NODE
    The AST node name for a Unary Operator Node.

  • Constructor Summary

    Constructors
    Constructor
    Description
    OPToken()
    Construct a new OPToken.
    OPToken(String sId)
    Construct a new OPToken with the given parameters.
    OPToken(String sId, int nBp)
    Construct a new OPToken with the given parameters.
    OPToken(String sId, int nBp, String sAstName)
    Construct a new OPToken with the given parameters.
    OPToken(String sId, int nBp, String sLedASTName, String sNudASTName)
    Construct a new OPToken with the given parameters.
    OPToken(String sId, String sAstName)
    Construct a new OPToken with the given parameters.

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    equals(Object o)
     
    int
    getBindingPower()
    Get The binding precedence of this token.
    String
    getId()
    Obtain the string representation of this token.
    String
    getLedASTName()
    Get led AST Name for this token
    String
    getNudASTName()
    Get nud AST Name for this token
    String
    getValue()
    Get a string value that identifies this token
    int
    hashCode()
     
    Term
    led(OPParser parser, Term leftNode)
    Process this token and possibly the given leftNodein the context of a parser with the left denotation.
    int
    leftBindingPower()
    Obtain the power that this token will bind to the left.
    protected Term
    newAST(String sAstName, Term term)
    Create an Abstract Syntax Node for the given arguments.
    protected Term
    newAST(String sAstName, String sFunctor)
    Create an Abstract Syntax Node for the given arguments.
    protected Term
    newAST(String sAstName, String sFunctor, Term term)
    Create an Abstract Syntax Node for the given arguments.
    protected Term
    newAST(String sAstName, String sFunctor, Term t1, Term t2)
    Create an Abstract Syntax Node for the given arguments.
    protected Term
    newAST(String sAstName, String sFunctor, Term t1, Term t2, Term t3)
    Create an Abstract Syntax Node for the given arguments.
    Term
    nud(OPParser parser)
    Process this token in the context of parser p with the null denotation.
    void
    setBindingPower(int nBp)
    Set The binding precedence that this token will use for binding to the right.
    void
    setId(String sId)
    Set the string representation of this token to the given id.
    void
    setLedASTName(String sAstName)
    Set the led AST Name for this token to be the given string
    void
    setNudASTName(String sAstName)
    Set the nud AST Name for this token to be the given string
    void
    setValue(String s)
    Set the AST Name for this token to be the given string
    String
    toString()
    Return a human-readable description for this token.

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

  • Field Details

    • BINARY_OPERATOR_NODE

      public static String BINARY_OPERATOR_NODE
      The AST node name for a Binary Operator node.

    • BINDING_NODE

      public static String BINDING_NODE
      The AST node name for a Binding Node.

    • CALL_NODE

      public static String CALL_NODE
      The AST node name for a Method Call Node.

    • DEREF_NODE

      public static String DEREF_NODE
      The AST node name for a De-referencing Node.

    • FIELD_LIST

      public static String FIELD_LIST
      The AST node name for a Field List Node.

    • IDENTIFIER_NODE

      public static String IDENTIFIER_NODE
      The AST node name for an Identifier Node.

    • LIST_NODE

      public static String LIST_NODE
      The AST node name for a List of Values Node.

    • LITERAL_NODE

      public static String LITERAL_NODE
      The AST node name for a Literal Node.

    • UNARY_OPERATOR_NODE

      public static String UNARY_OPERATOR_NODE
      The AST node name for a Unary Operator Node.

    • PRECEDENCE_KEYWORD

      public static final int PRECEDENCE_KEYWORD
      The binding precedence for keyword tokens
      See Also:

    • PRECEDENCE_IDENTIFIER

      public static final int PRECEDENCE_IDENTIFIER
      The binding precedence for identifier tokens
      See Also:

    • PRECEDENCE_ASSIGNMENT

      public static final int PRECEDENCE_ASSIGNMENT
      The binding precedence for assignment operators assignment such as = += -= *= /= %= &= ^= |= <<= >>= >>>=
      See Also:

    • PRECEDENCE_LOGICAL

      public static final int PRECEDENCE_LOGICAL
      The binding precedence for logical tokens such as &&, ||, etc
      See Also:

    • PRECEDENCE_LOGICAL_BITWISE

      public static final int PRECEDENCE_LOGICAL_BITWISE
      The binding precedence for bitwise logical tokens such as &, |, ^ etc
      See Also:

    • PRECEDENCE_RELATIONAL

      public static final int PRECEDENCE_RELATIONAL
      The binding precedence for relational operators such as ==, <=, like, contains etc
      See Also:

    • PRECEDENCE_BITWISE

      public static final int PRECEDENCE_BITWISE
      The binding precedence for bitwise operators such as >>> >> and <<
      See Also:

    • PRECEDENCE_SUM

      public static final int PRECEDENCE_SUM
      The binding precedence for sum arithmetic, i.e. + and -
      See Also:

    • PRECEDENCE_PRODUCT

      public static final int PRECEDENCE_PRODUCT
      The binding precedence for product arithmetic, multiplication, division, mod * / %
      See Also:

    • PRECEDENCE_EXPONENT

      public static final int PRECEDENCE_EXPONENT
      The binding precedence for exponent arithmetic, i.e. raising by a power
      See Also:

    • PRECEDENCE_UNARY

      public static final int PRECEDENCE_UNARY
      The binding precedence for other unary operators: pre-increment, pre-decrement, plus, minus, logical negation, bitwise complement, type cast ++expr --expr +expr -expr ! ~ (type)
      See Also:

    • PRECEDENCE_UNARY_POST

      public static final int PRECEDENCE_UNARY_POST
      The binding precedence for unary post operators such as post-increment and post-decrement of the form expr++ or expr--
      See Also:

    • PRECEDENCE_PARENTHESES

      public static final int PRECEDENCE_PARENTHESES
      The binding precedence for parentheses ( ) [ ]
      See Also:

    • m_sValue

      protected String m_sValue
      The string value of the literal.

    • m_sLedASTName

      protected String m_sLedASTName
      A functor for building ast for led method.

    • m_sNudASTName

      protected String m_sNudASTName
      A functor for building ast for nud method.

    • m_nBindingPower

      protected int m_nBindingPower
      The power that this token binds, typically to the left.

  • Constructor Details

    • OPToken

      public OPToken()
      Construct a new OPToken.

    • OPToken

      public OPToken(String sId)
      Construct a new OPToken with the given parameters.
      Parameters:
      sId - string representation of the literal

    • OPToken

      public OPToken(String sId, int nBp)
      Construct a new OPToken with the given parameters.
      Parameters:
      sId - string representation of the token
      nBp - The binding precedence for this token

    • OPToken

      public OPToken(String sId, String sAstName)
      Construct a new OPToken with the given parameters.
      Parameters:
      sId - string representation of the token
      sAstName - the type code for this literal token

    • OPToken

      public OPToken(String sId, int nBp, String sAstName)
      Construct a new OPToken with the given parameters.
      Parameters:
      sId - string representation of the token
      nBp - The binding precedence for this token
      sAstName - the name for this tokens AST

    • OPToken

      public OPToken(String sId, int nBp, String sLedASTName, String sNudASTName)
      Construct a new OPToken with the given parameters.
      Parameters:
      sId - string representation of the token
      nBp - The binding precedence for this token
      sLedASTName - the name for this tokens AST
      sNudASTName - the name for this tokens AST

  • Method Details

    • leftBindingPower

      public int leftBindingPower()
      Obtain the power that this token will bind to the left.
      Returns:
      the left binding power

    • nud

      public Term nud(OPParser parser)
      Process this token in the context of parser p with the null denotation. A nud method typically will have no interest in the token to the left. The processing results in an Abstract Syntax Tree Node that captures the meaning
      Parameters:
      parser - the parser that is the context for parsing
      Returns:
      an AstNode

    • led

      public Term led(OPParser parser, Term leftNode)
      Process this token and possibly the given leftNodein the context of a parser with the left denotation. A led method typically will be interested t in the token to the left. The processing results in an Abstract Syntax Tree Node that captures the meaning
      Parameters:
      parser - the parser that is the context for parsing
      leftNode - an ast Term that the token is possibly interested in
      Returns:
      an AstNode

    • newAST

      protected Term newAST(String sAstName, String sFunctor)
      Create an Abstract Syntax Node for the given arguments. If the astName argument is not null then use it for the functor and the given functor argument become the first child Term.
      Parameters:
      sAstName - classification functor or null
      sFunctor - functor for ast node to be constructed
      Returns:
      a Term representing the AST

    • newAST

      protected Term newAST(String sAstName, Term term)
      Create an Abstract Syntax Node for the given arguments. If the astName argument is not null then use it for the functor otherwise just assume the Term t is good.
      Parameters:
      sAstName - classification functor or null
      term - an Term that is part of the ast
      Returns:
      a Term representing the AST

    • newAST

      protected Term newAST(String sAstName, String sFunctor, Term term)
      Create an Abstract Syntax Node for the given arguments. If the astName argument is not null then use it for the functor and the given functor argument become the first child Term.
      Parameters:
      sAstName - classification functor or null
      sFunctor - functor for ast node to be constructed
      term - an Term that is part of the ast
      Returns:
      a Term representing the AST

    • newAST

      protected Term newAST(String sAstName, String sFunctor, Term t1, Term t2)
      Create an Abstract Syntax Node for the given arguments. If the astName argument is not null then use it for the functor and the given functor argument become the first child Term.
      Parameters:
      sAstName - classification functor or null
      sFunctor - functor for ast node to be constructed
      t1 - an Term that is part of the ast
      t2 - an Term that is part of the ast
      Returns:
      a Term representing the AST

    • newAST

      protected Term newAST(String sAstName, String sFunctor, Term t1, Term t2, Term t3)
      Create an Abstract Syntax Node for the given arguments. If the astName argument is not null then use it for the functor and the given functor argument become the first child Term.
      Parameters:
      sAstName - classification functor or null
      sFunctor - functor for ast node to be constructed
      t1 - an Term that is part of the ast
      t2 - an Term that is part of the ast
      t3 - an Term that is part of the ast
      Returns:
      a Term representing the AST

    • getId

      public String getId()
      Obtain the string representation of this token.
      Returns:
      the string representation

    • setId

      public void setId(String sId)
      Set the string representation of this token to the given id.
      Parameters:
      sId - the string representation for the token

    • getBindingPower

      public int getBindingPower()
      Get The binding precedence of this token.
      Returns:
      The binding precedence

    • setBindingPower

      public void setBindingPower(int nBp)
      Set The binding precedence that this token will use for binding to the right.
      Parameters:
      nBp - the power power for this token

    • getNudASTName

      public String getNudASTName()
      Get nud AST Name for this token
      Returns:
      the nud ast name for this token

    • setNudASTName

      public void setNudASTName(String sAstName)
      Set the nud AST Name for this token to be the given string
      Parameters:
      sAstName - the nud ast name for this token

    • getLedASTName

      public String getLedASTName()
      Get led AST Name for this token
      Returns:
      the led ast name for this token

    • setLedASTName

      public void setLedASTName(String sAstName)
      Set the led AST Name for this token to be the given string
      Parameters:
      sAstName - the led ast name for this token

    • getValue

      public String getValue()
      Get a string value that identifies this token
      Returns:
      the a string that identifies this token

    • setValue

      public void setValue(String s)
      Set the AST Name for this token to be the given string
      Parameters:
      s - the ast name for this token

    • toString

      public String toString()
      Return a human-readable description for this token.
      Overrides:
      toString in class Object
      Returns:
      a String description of the token

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object