Repository navigation
Expand use of sql parsing string expressions in DataFrame #1278
Description
Activity
I think when we have a string we want to first check to see if it is a column name. If so, create a column expression. If not, try sql parsing.
Have you changed your mind about select Tim? I'm stuck next few days but may help if needed
I do think we want
selectbut it just can't be the snippet I shared before. Instead I think it needs to check if the string matches a column name first. If so, make a column expression. Otherwise try parsing. What do you think?I have no strong opinion Tim, I'm split between keeping it as it is and change it
i hope to tick off few of them off the list this weekend
looking at spark it looks like we're aligned with
selectandselectExprmaybe we keep both as they are ?I can see how it might be preferable to some to have it split out, but I also feel like just having one
selectis very ergonomic. Asking for another opinion from some users I know @ntjohnson1 @MrPowersReacted by Marko Milenkovićif we do not have special handle for column names I would agree with you, one select may make more sense. If we have to check for column name before we
parse_sqlI believe it would be simpler from user perspective to leave them as they are. Anyway leats get consensusI definitely think a single
selectwould be nice ergonomically.aggregatemay have the same issue likeselectas it expects column name as a string, I'll leave those two to you @timsaucer
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This is a follow on to #1273
There are a number of places where it would be convenient to pass SQL strings as expressions. For example, it would be nice to do
This should intuitively know that we are getting column
a, followed bycol("a") - col("b")followed by column 'c'.Describe the solution you'd like
Using the sql parsing on the DataFrame make the following functions handle SQL strings. We must be very careful that we do not break things like cases where users have a column name that is not SQL parseable.
Describe alternatives you've considered
Status quo
Additional context
DataFrame functions to update:
select_exprswith_columnfeat:with_columnsupports SQL expression #1284with_columnsfeat: Add SQL expression support forwith_columns#1286repartition_by_hashfeat: Add SQL expression forrepartition_by_hash#1285We do not want to apply this treatment to joins because there is no easy way to know which DataFrame to perform the SQL parsing against.