[Feature request] Support range operator for tokens

[KvanTTT]

During developing my fork YANTLR I've encountered, that ranges can be implemented not only for character, but for tokens too. It's extremely useful to use this feature when a grammar has a lot of tokens that could be identifiers, for instance, our SQL grammars.

Consider the current simplified PlSql grammar and its id rule:

id
    : REGULAR_ID
    | ABSENT
    | A_LETTER
    | AGENT
    | AGGREGATE
    | ANALYZE
    | AUTONOMOUS_TRANSACTION
    | BACKINGFILE
    | BATCH
    | BINARY_INTEGER
    | BOOLEAN
    | C_LETTER
    | CHAR
    | CHARSETID
    | CHARSETFORM
    | CLUSTER
    | CONSTRUCTOR
    | CUSTOMDATUM
    | CASESENSITIVE
    | DECIMAL
    | DELETE
    | DEPRECATE
    | DETERMINISTIC
    | DSINTERVAL_UNCONSTRAINED
    | DURATION
    | E_LETTER
    | ERROR_INDEX
    | ERROR_CODE
    | E_LETTER
    | ERR
    | EXCEPTION
    | EXCEPTION_INIT
    | EXCEPTIONS
    | EXISTS
    | EXIT
    | EXTEND
    | FILESTORE
    | FLOAT
    | FORALL
    | G_LETTER
    | INDICES
    | INOUT
    | INTEGER
    | JSON_TRANSFORM
    | K_LETTER
    | LANGUAGE
    | LONG
    | LOOP
    | MAXLEN
    | MOUNTPOINT
    | M_LETTER
    | MISSING
    | MISMATCH
    | NUMBER
    | ORADATA
    | OSERROR
    | OUT
    | OVERRIDING
    | P_LETTER
    | PARALLEL_ENABLE
    | PIPELINED
    | PLS_INTEGER
    | PMEM
    | POSITIVE
    | POSITIVEN
    | PRAGMA
    | PUBLIC
    | RAISE
    | RAW
    | RECORD
    | REF
    | RENAME
    | RESTRICT_REFERENCES
    | RESULT
    | SDO_GEOMETRY
    | SELF
    | SERIALLY_REUSABLE
    | SET
    | SEQ
    | SHARDSPACE
    | SIGNTYPE
    | SIMPLE_INTEGER
    | SMALLINT
    | STRUCT
    | SQLDATA
    | SQLERROR
    | SUBTYPE
    | T_LETTER
    | TDO
    | TIMESTAMP_LTZ_UNCONSTRAINED
    | TIMESTAMP_TZ_UNCONSTRAINED
    | TIMESTAMP_UNCONSTRAINED
    | TIMEZONE
    | TRIGGER
    | VARCHAR
    | VARCHAR2
    | VARIABLE
    | WARNING
    | WHILE
    | WM_CONCAT
    | XMLAGG
    | YMINTERVAL_UNCONSTRAINED
    | REGR_
    | VAR_
    | VALUE
    | COVAR_
    | ERROR_INDEX
    | ERROR_CODE
    ;

With the range operator, it can be rewritten in the following way:

id
    | REGULAR_ID
    | ABSENT..ERROR_CODE
    ;

The range operator decreases the size of grammar and, more importantly, it makes grammar less error-prone because all tokens declared between ABSENT and ERROR_CODE are always considered in id and it decreases the chance that a user forget to add a newly added token to id rule.

I suppose the feature is not very complicated to implement, because the range operator is already supported for characters and it looks very natural to support ranges for tokens as well (as tokens are "characters" for parser). It's much easy than token value comparison operator (this feature could significantly reduce number of declared tokens in SQL grammars but in another way) that requires changing of runtimes.

@parrt if you give me green light, I can try to implement the feature for ANTLR 4.

[parrt]

I think the problem is that tokens are officially unordered so range doesn't really have a meaning