Package com.google.protobuf
Class TextFormat.Tokenizer
- java.lang.Object
-
- com.google.protobuf.TextFormat.Tokenizer
-
- Enclosing class:
- TextFormat
private static final class TextFormat.Tokenizer extends java.lang.Object
Represents a stream of tokens parsed from aString
.The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:
java.io.StreamTokenizer
: This almost does what we want -- or, at least, something that would get us close to what we want -- except for one fatal flaw: It automatically un-escapes strings using Java escape sequences, which do not include all the escape sequences we need to support (e.g. '\x').java.util.Scanner
: This seems like a great way at least to parse regular expressions out of a stream (so we wouldn't have to load the entire input into a single string before parsing). Sadly,Scanner
requires that tokens be delimited with some delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"),Scanner
would recognize it only as a single token. Furthermore,Scanner
provides no way to inspect the contents of delimiters, making it impossible to keep track of line and column numbers.
Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need
Matcher.usePattern()
, which is new in Java 1.5.) So, we can use that, at least. Unfortunately, this implies that we need to have the entire input in one contiguous string.
-
-
Field Summary
Fields Modifier and Type Field Description private int
column
private boolean
containsSilentMarkerAfterCurrentToken
containsSilentMarkerAfterCurrentToken
indicates if there is a silent marker after the current token.private boolean
containsSilentMarkerAfterPrevToken
private java.lang.String
currentToken
private static java.util.regex.Pattern
DOUBLE_INFINITY
private static java.util.regex.Pattern
FLOAT_INFINITY
private static java.util.regex.Pattern
FLOAT_NAN
private int
line
private java.util.regex.Matcher
matcher
private int
pos
private int
previousColumn
private int
previousLine
private java.lang.CharSequence
text
private static java.util.regex.Pattern
TOKEN
private static java.util.regex.Pattern
WHITESPACE
-
Constructor Summary
Constructors Modifier Constructor Description private
Tokenizer(java.lang.CharSequence text)
Construct a tokenizer that parses tokens from the given text.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) boolean
atEnd()
Are we at the end of the input?(package private) void
consume(java.lang.String token)
If the next token exactly matchestoken
, consume it.boolean
consumeBoolean()
If the next token is a boolean, consume it and return its value.(package private) ByteString
consumeByteString()
If the next token is a string, consume it, unescape it as aByteString
, and return it.private void
consumeByteString(java.util.List<ByteString> list)
LikeconsumeByteString()
but adds each token of the string to the given list.double
consumeDouble()
If the next token is a double, consume it and return its value.float
consumeFloat()
If the next token is a float, consume it and return its value.(package private) java.lang.String
consumeIdentifier()
If the next token is an identifier, consume it and return its value.(package private) int
consumeInt32()
If the next token is a 32-bit signed integer, consume it and return its value.(package private) long
consumeInt64()
If the next token is a 64-bit signed integer, consume it and return its value.java.lang.String
consumeString()
If the next token is a string, consume it and return its (unescaped) value.(package private) int
consumeUInt32()
If the next token is a 32-bit unsigned integer, consume it and return its value.(package private) long
consumeUInt64()
If the next token is a 64-bit unsigned integer, consume it and return its value.private TextFormat.ParseException
floatParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse a float or double.(package private) int
getColumn()
(package private) boolean
getContainsSilentMarkerAfterCurrentToken()
(package private) boolean
getContainsSilentMarkerAfterPrevToken()
(package private) int
getLine()
(package private) int
getPreviousColumn()
(package private) int
getPreviousLine()
private TextFormat.ParseException
integerParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse an integer.(package private) boolean
lookingAt(java.lang.String text)
Returnstrue
if the current token's text is equal to that specified.(package private) boolean
lookingAtInteger()
Returnstrue
if the next token is an integer, but does not consume it.(package private) void
nextToken()
Advance to the next token.(package private) TextFormat.ParseException
parseException(java.lang.String description)
Returns aTextFormat.ParseException
with the current line and column numbers in the description, suitable for throwing.(package private) TextFormat.ParseException
parseExceptionPreviousToken(java.lang.String description)
Returns aTextFormat.ParseException
with the line and column numbers of the previous token in the description, suitable for throwing.private void
skipWhitespace()
Skip over any whitespace so that the matcher region starts at the next token.(package private) boolean
tryConsume(java.lang.String token)
If the next token exactly matchestoken
, consume it and returntrue
.boolean
tryConsumeDouble()
If the next token is a double, consume it and returntrue
.boolean
tryConsumeFloat()
If the next token is a float, consume it and returntrue
.(package private) boolean
tryConsumeIdentifier()
If the next token is an identifier, consume it and returntrue
.(package private) boolean
tryConsumeInt64()
If the next token is a 64-bit signed integer, consume it and returntrue
.(package private) boolean
tryConsumeString()
If the next token is a string, consume it and return true.boolean
tryConsumeUInt64()
If the next token is a 64-bit unsigned integer, consume it and returntrue
.
-
-
-
Field Detail
-
text
private final java.lang.CharSequence text
-
matcher
private final java.util.regex.Matcher matcher
-
currentToken
private java.lang.String currentToken
-
pos
private int pos
-
line
private int line
-
column
private int column
-
previousLine
private int previousLine
-
previousColumn
private int previousColumn
-
WHITESPACE
private static final java.util.regex.Pattern WHITESPACE
-
TOKEN
private static final java.util.regex.Pattern TOKEN
-
DOUBLE_INFINITY
private static final java.util.regex.Pattern DOUBLE_INFINITY
-
FLOAT_INFINITY
private static final java.util.regex.Pattern FLOAT_INFINITY
-
FLOAT_NAN
private static final java.util.regex.Pattern FLOAT_NAN
-
containsSilentMarkerAfterCurrentToken
private boolean containsSilentMarkerAfterCurrentToken
containsSilentMarkerAfterCurrentToken
indicates if there is a silent marker after the current token. This value is moved tocontainsSilentMarkerAfterPrevToken
every time the next token is parsed.
-
containsSilentMarkerAfterPrevToken
private boolean containsSilentMarkerAfterPrevToken
-
-
Method Detail
-
getPreviousLine
int getPreviousLine()
-
getPreviousColumn
int getPreviousColumn()
-
getLine
int getLine()
-
getColumn
int getColumn()
-
getContainsSilentMarkerAfterCurrentToken
boolean getContainsSilentMarkerAfterCurrentToken()
-
getContainsSilentMarkerAfterPrevToken
boolean getContainsSilentMarkerAfterPrevToken()
-
atEnd
boolean atEnd()
Are we at the end of the input?
-
nextToken
void nextToken()
Advance to the next token.
-
skipWhitespace
private void skipWhitespace()
Skip over any whitespace so that the matcher region starts at the next token.
-
tryConsume
boolean tryConsume(java.lang.String token)
If the next token exactly matchestoken
, consume it and returntrue
. Otherwise, returnfalse
without doing anything.
-
consume
void consume(java.lang.String token) throws TextFormat.ParseException
If the next token exactly matchestoken
, consume it. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
lookingAtInteger
boolean lookingAtInteger()
Returnstrue
if the next token is an integer, but does not consume it.
-
lookingAt
boolean lookingAt(java.lang.String text)
Returnstrue
if the current token's text is equal to that specified.
-
consumeIdentifier
java.lang.String consumeIdentifier() throws TextFormat.ParseException
If the next token is an identifier, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
tryConsumeIdentifier
boolean tryConsumeIdentifier()
If the next token is an identifier, consume it and returntrue
. Otherwise, returnfalse
without doing anything.
-
consumeInt32
int consumeInt32() throws TextFormat.ParseException
If the next token is a 32-bit signed integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeUInt32
int consumeUInt32() throws TextFormat.ParseException
If the next token is a 32-bit unsigned integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeInt64
long consumeInt64() throws TextFormat.ParseException
If the next token is a 64-bit signed integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
tryConsumeInt64
boolean tryConsumeInt64()
If the next token is a 64-bit signed integer, consume it and returntrue
. Otherwise, returnfalse
without doing anything.
-
consumeUInt64
long consumeUInt64() throws TextFormat.ParseException
If the next token is a 64-bit unsigned integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
tryConsumeUInt64
public boolean tryConsumeUInt64()
If the next token is a 64-bit unsigned integer, consume it and returntrue
. Otherwise, returnfalse
without doing anything.
-
consumeDouble
public double consumeDouble() throws TextFormat.ParseException
If the next token is a double, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
tryConsumeDouble
public boolean tryConsumeDouble()
If the next token is a double, consume it and returntrue
. Otherwise, returnfalse
without doing anything.
-
consumeFloat
public float consumeFloat() throws TextFormat.ParseException
If the next token is a float, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
tryConsumeFloat
public boolean tryConsumeFloat()
If the next token is a float, consume it and returntrue
. Otherwise, returnfalse
without doing anything.
-
consumeBoolean
public boolean consumeBoolean() throws TextFormat.ParseException
If the next token is a boolean, consume it and return its value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeString
public java.lang.String consumeString() throws TextFormat.ParseException
If the next token is a string, consume it and return its (unescaped) value. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
tryConsumeString
boolean tryConsumeString()
If the next token is a string, consume it and return true. Otherwise, return false.
-
consumeByteString
ByteString consumeByteString() throws TextFormat.ParseException
If the next token is a string, consume it, unescape it as aByteString
, and return it. Otherwise, throw aTextFormat.ParseException
.- Throws:
TextFormat.ParseException
-
consumeByteString
private void consumeByteString(java.util.List<ByteString> list) throws TextFormat.ParseException
LikeconsumeByteString()
but adds each token of the string to the given list. String literals (whether bytes or text) may come in multiple adjacent tokens which are automatically concatenated, like in C or Python.- Throws:
TextFormat.ParseException
-
parseException
TextFormat.ParseException parseException(java.lang.String description)
Returns aTextFormat.ParseException
with the current line and column numbers in the description, suitable for throwing.
-
parseExceptionPreviousToken
TextFormat.ParseException parseExceptionPreviousToken(java.lang.String description)
Returns aTextFormat.ParseException
with the line and column numbers of the previous token in the description, suitable for throwing.
-
integerParseException
private TextFormat.ParseException integerParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse an integer.
-
floatParseException
private TextFormat.ParseException floatParseException(java.lang.NumberFormatException e)
Constructs an appropriateTextFormat.ParseException
for the givenNumberFormatException
when trying to parse a float or double.
-
-