[coverage] Conformance findings: DATATYPE-043 #1002

Summary

Surfaced by the multi-language coverage fan-out while conformance-testing these SPEC-IDs against databricks/databricks-sql-python. Each finding is committed as an expected-failure (xfail) test in the coverage PR — the test asserts the CORRECT (post-fix) behavior and stays red until THIS driver (databricks/databricks-sql-python) is fixed, then flips green as a tripwire.

Findings

• DATATYPE-043 [thrift]: A TIMESTAMP_NTZ result column reports PEP 249 type_code 'timestamp' on the Thrift path, identical to a zoned TIMESTAMP column, even though the Arrow Spark:DataType:SqlName metadata _col_to_description already reads carries the distinction — its logical-type regex matches only variant|geometry|geography
  • failing test: test_timestamp_ntz_wall_clock_and_type_name_under_non_utc_session_zone (see the coverage PR diff under tests/)
• DATATYPE-043 [sea]: On the kernel/SEA path a TIMESTAMP_NTZ result column reports PEP 249 type_code 'timestamp', identical to a zoned TIMESTAMP column: _databricks_type_for_field's databricks.type_name allowlist omits timestamp_ntz, so the name falls through to the Arrow-shape mapping that collapses all timestamps to TIMESTAMP
  • failing test: test_timestamp_ntz_wall_clock_and_type_name_under_non_utc_session_zone (see the coverage PR diff under tests/)
• DATATYPE-043: A TIMESTAMP_NTZ result column reports PEP 249 type_code 'timestamp', identical to a zoned TIMESTAMP column, on both the Thrift and kernel/SEA backends — the distinguishing metadata is available on both wires (Thrift Spark:DataType:SqlName, kernel databricks.type_name) but neither _col_to_description's logical-type regex nor _databricks_type_for_field's allowlist includes timestamp_ntz, so the two types are indistinguishable to a consumer reading cursor.description

Reproduce & Expected

DATATYPE-043 — Verify that under a NON-UTC session time zone a TIMESTAMP_NTZ column and a zoned TIMESTAMP column in the SAME result set are each handled according to their own semantics: - TIMESTAMP_NTZ is a wall c…

Reproduce:

• Set the session time zone through whichever surface the driver exposes -- a
  session parameter on connect (e.g. a timezone session param) or, where no
  such option exists, SET TIME ZONE 'America/New_York' executed on the same
  session before the test query. A NON-UTC zone is REQUIRED: under a UTC session
  the shifted and unshifted NTZ renderings are identical and the case proves
  nothing.
• Read the result's column type names before fetching the first row.
• Read the single row, then exhaust the result set.

Expected (per the shared spec):

• result has exactly 1 row(s)
• full assertion contract:

result:
- row_count: 1
- label: before_iteration
  column_type_name:
    name: zoned
    equals: TIMESTAMP
- label: before_iteration
  column_type_name:
    name: ntz
    equals: TIMESTAMP_NTZ
- label: after_exhaustion
  column_type_name:
    name: zoned
    equals: TIMESTAMP
- label: after_exhaustion
  column_type_name:
    name: ntz
    equals: TIMESTAMP_NTZ
- timestamp_wall_clock:
    column: ntz
    equals: '2024-06-15 09:15:00.123456'
- timestamp_instant:
    column: zoned
    equals: '2024-06-15T13:15:00.123456Z'

Context

• The behavior was first fixed in a DIFFERENT driver — reference PR: TIMESTAMP_NTZ is scanned as a zoned TIMESTAMP (#506) databricks-sql-go#510 — which seeded the shared language-neutral spec. This issue tracks the same conformance gap in databricks/databricks-sql-python; the reference PR is for cross-referencing the intended behavior, NOT a change to this repo.
• Coverage PR carrying the reproducing xfail test(s): https://github.com/databricks/databricks-driver-test/pull/1897