Reject invalid numeric underscore separators - #2426
Conversation
| if self.dialect.supports_numeric_literal_underscores() | ||
| && chars.peek() == Some(&'_') | ||
| { | ||
| return self.tokenizer_error(chars.location(), "Unexpected character '_'"); | ||
| } |
There was a problem hiding this comment.
i couldn't see if/why this block was necessary given is_number_separator checks with a lookahead - is there a fallthrough code that captures the trailing underscore somewhere below?
I have the same comments about some of the other similar diffs below, is there a way to solve this in a more holistic manner vs the individual if checks - the latter is unclear/not-obvious how they interact with the rest of the code
There was a problem hiding this comment.
Thanks for pointing this out. You are right that the first check was redundant. I replaced the separate checks with one helper that validates separators consistently across integer, fractional, hexadecimal, and exponent parts, and added coverage for the related edge cases. The full test suite, Clippy, and formatting checks pass. Thanks again for the careful review.
Summary
Fixes #2421 by rejecting invalid numeric underscore separators for dialects that support underscores in numeric literals.
Details
Numeric separators should appear between digits. This keeps valid literals such as
10_000and1_000.123_456, while rejecting trailing, consecutive, and decimal-point-adjacent underscores such as10_00_,10___0,1_000.123_, and1._000.The change adds coverage at both the tokenizer and parser test levels.
Validation
git diff --checkcargo fmt --all --checkcargo test tokenize_numeric_literal_underscore -- --nocapturecargo test parse_numeric_literal_underscore --test sqlparser_common -- --nocapturecargo test