[Parquet][Python] API to decrypt parquet file using one DEK and no metadata

Describe the enhancement requested

I am having trouble decrypting a parquet file encrypted in the C++ Arrow library using PyArrow.

In C++, I encrypt the file with one key and no metadata.

  // Convert vector<uint8_t> to string for the FileEncryptionProperties::Builder
  std::string keyStr(reinterpret_cast<const char*>(key.data()), key.size());

  auto fileEncryptionProps = parquet::FileEncryptionProperties::Builder(keyStr)
                                 .algorithm(parquet::ParquetCipher::AES_GCM_V1)
                                 ->build();

  const auto props = parquet::WriterProperties::Builder()
                         .encryption(fileEncryptionProps)
                         ->build();

  const auto fields = convertSchema(table->schema());
  const auto schemaNode = std::static_pointer_cast<parquet::schema::GroupNode>(
      parquet::schema::GroupNode::Make(
          "schema", parquet::Repetition::REQUIRED, fields));
  auto schema =
      std::static_pointer_cast<parquet::schema::GroupNode>(schemaNode);

  // Open output file
  std::shared_ptr<arrow::io::FileOutputStream> outFile;
  auto result = arrow::io::FileOutputStream::Open(outputFilePath);
  if (!result.ok()) {
    throw std::runtime_error("Failed to open output file: " + outputFilePath);
  }
  outFile = result.ValueOrDie();

  auto parquetStreamWriter = make_unique<parquet::StreamWriter>(
      parquet::ParquetFileWriter::Open(outFile, schema, props));

  return writeToStreamWriter(table, *(parquetStreamWriter.get()));

I can decrypt the file in C++ using the same key and no metadata.

  const std::string keyStr(key.begin(), key.end());
  auto decryptionProps =
      parquet::FileDecryptionProperties::Builder().footer_key(keyStr)->build();
  auto readerProps = parquet::ReaderProperties();
  readerProps.file_decryption_properties(decryptionProps);

  std::unique_ptr<parquet::ParquetFileReader> reader =
      parquet::ParquetFileReader::OpenFile(outputPath, false, readerProps);

  // Read the file content into an arrow table actualTable using reader
  std::shared_ptr<arrow::Table> actualTable;
  std::unique_ptr<parquet::arrow::FileReader> arrowReader;
  auto status = parquet::arrow::FileReader::Make(
      arrow::default_memory_pool(), std::move(reader), &arrowReader);

  status = arrowReader->ReadTable(&actualTable);

However, I can't find an API in PyArrow to do the equivalent.

In Python, I can do the following, but it requires setting extra dummy key metadata in c++ (i.e., this code won't decrypt a file encrypted by the C++ approach at the beginning of this post).

import pyarrow.parquet.encryption as pe

# Create a simple KMS client that returns our DEK
class SimpleKmsClient(pe.KmsClient):
    def __init__(self):
        pe.KmsClient.__init__(self)
    
    def unwrap_key(self, wrapped_key, master_key_identifier):
        return dek
    
    def wrap_key(self, key_bytes, master_key_identifier):
        raise NotImplementedError("wrap_key not needed for decryption")

# Create KMS factory
def kms_factory(kms_connection_configuration):
    return SimpleKmsClient()

crypto_factory = pe.CryptoFactory(kms_factory)

# Simple decryption config
decryption_config = pe.DecryptionConfiguration()
kms_connection_config = pe.KmsConnectionConfig()

# Create file decryption properties
file_decryption_props = crypto_factory.file_decryption_properties(
    kms_connection_config, decryption_config
)

# Read the file
table = pq.read_table(path, decryption_properties=file_decryption_props)

My question is: Is there any Pyhton API to decrypt using only one DEK and no metadata?

Component(s)

Python

@rok

cc @rok @EnricoMi

btw, @kou as discussed on the past should this be converted to a discussion into Q&A instead of an issue?

We currently don't have a Python API where a single DEK could be used which is on purpose to encourage use of KMS over single keys (which is not recommended security-wise).
However users are asking for this and we should provide a good example of how to use a KMS in C++.
Alternatively we could provide a pe.NonSecureKMS in Python API that would provide what is asked for here but would be easy to swap out for a secure KMS when needed. I'd prefer the first option as it doesn't weaken our encryption.

@adamreeve

cc @adamreeve

@rok

Thanks for confirming, @rok.

I'd prefer the first option as it doesn't weaken our encryption.

Can you help me understand why a using a single DEK weaken the encryption? In my use case, we do encrypt the DEK and pass it in a secure channel. In a thought experiment, can users do exactly what the Arrow library does, but just do it external to the library and then pass in the key?

I am not a security expert, so please educate me.

@changhu-m

I second @changhu-m question - it is not 100% clear how using externally managed keys can weaken the security story.

The use of KMS by default is a sensible choice. It seems that there is no other workaround for self-managed keys rather than creating an in-memory KMS and adding dummy metadata which is not ideal

Can you help me understand why a using a single DEK weaken the encryption? In my use case, we do encrypt the DEK and pass it in a secure channel. In a thought experiment, can users do exactly what the Arrow library does, but just do it external to the library and then pass in the key?

Sorry, weakening encryption is not the correct expression here. We're encouraging use of KMSes as it helps users not having to re-implement things needed for envelope encryption, see design doc. I agree it would be nice to have more choices in pyarrow encryption story. We would want to be cautious to encourage good key management practices and make it easy to follow envelope encryption as per spec.

@ggershinsky

For a bit more context, here's the old mailing list discussion and issue where it was decided to only expose the higher level in Python.

The main arguments against exposing the low level in Python (from @ggershinsky ) were:

security: low-level encryption API is easy to misuse (eg giving the same keys for a number of different files; this'd break the AES GCM cipher). The high-level encryption layer handles that by applying envelope encryption and other best practices in data security. Also, this layer is maintained by the community, meaning that future improvements and security fixes can be upstreamed by anyone, and available to all.

compatibility: parquet-mr implements the high-level encryption layer. If we want the files produced by Spark/Presto/etc to be readable by pandas/PyArrow (and vice versa), we need to provide the Arrow users with the high-level API.

@adamreeve

Thanks for the context @adamreeve !

Perhaps a point of interest - we recently had an issue reported #47254 where encrypted readers/writers are not compatible due to one implementation avoiding KMS (I think) and not following the spec.

To be clear @changhu-m @akoshelev - PyArrow / arrow-cpp (and Parquet in general) encryption could use more examples / better examples and documentation (e.g. an example of how to use a real KMS with arrow-cpp and PyArrow). API change proposals are welcome too, we'd just want to discuss them before proceeding to implementation.

To not stifle progress here - @changhu-m could you say what your use case is and if you can see another way forward that doesn't include opening the low-level API in PyArrow?

@adamreeve

Thanks @adamreeve @rok .

Our use case is the following:

Encryption side:

A parquet file is encrypted with a DEK.
The DEK is encrypted with KEK.
The encrypted DEK and related metadata (i.e., KMS info to get KEK) is printed as JSON in a metadata file.
The parquet file and metadata file are uploaded to a cloud location.

Our C++ code, as mentioned, is:

  std::string keyStr(reinterpret_cast<const char*>(key.data()), key.size());

  auto fileEncryptionProps = parquet::FileEncryptionProperties::Builder(keyStr)
                                 .algorithm(parquet::ParquetCipher::AES_GCM_V1)
                                 ->build();

  const auto props = parquet::WriterProperties::Builder()
                         .encryption(fileEncryptionProps)
                         ->build();

  const auto fields = convertSchema(table->schema());
  const auto schemaNode = std::static_pointer_cast<parquet::schema::GroupNode>(
      parquet::schema::GroupNode::Make(
          "schema", parquet::Repetition::REQUIRED, fields));
  auto schema =
      std::static_pointer_cast<parquet::schema::GroupNode>(schemaNode);

  // Open output file
  std::shared_ptr<arrow::io::FileOutputStream> outFile;
  auto result = arrow::io::FileOutputStream::Open(outputFilePath);
  outFile = result.ValueOrDie();

  auto parquetStreamWriter = make_unique<parquet::StreamWriter>(
      parquet::ParquetFileWriter::Open(outFile, schema, props));

  return writeToStreamWriter(table, *(parquetStreamWriter.get()));

Decryption side:

Customers get the KEK using the metadata sent.
Customers decrypt the DEK.
Customer sets up parquet decryption with the DEK.

The Python code is:

import pyarrow.parquet.encryption as pe

# Create a simple KMS client that returns our DEK
class SimpleKmsClient(pe.KmsClient):
    def __init__(self):
        pe.KmsClient.__init__(self)
    
    def unwrap_key(self, wrapped_key, master_key_identifier):
        return dek
    
    def wrap_key(self, key_bytes, master_key_identifier):
        raise NotImplementedError("wrap_key not needed for decryption")

# Create KMS factory
def kms_factory(kms_connection_configuration):
    return SimpleKmsClient()

crypto_factory = pe.CryptoFactory(kms_factory)

# Simple decryption config
decryption_config = pe.DecryptionConfiguration()
kms_connection_config = pe.KmsConnectionConfig()

# Create file decryption properties
file_decryption_props = crypto_factory.file_decryption_properties(
    kms_connection_config, decryption_config
)

# Read the file
table = pq.read_table(path, decryption_properties=file_decryption_props)

And the Spark code is:

    
    # Create Spark session with encryption support
    spark = SparkSession.builder \
        .appName("ParquetModularEncryption") \
        .config('spark.jars', 'parquet-hadoop-1.13.1-tests.jar')\
        .config("spark.sql.adaptive.enabled", "true") \
        .config("spark.sql.adaptive.coalescePartitions.enabled", "true") \
        .config("spark.hadoop.parquet.encryption.kms.client.class", "org.apache.parquet.crypto.keytools.mocks.InMemoryKMS") \
        .config("spark.hadoop.parquet.encryption.key.list", "k1:{DEK}") \
        .config("spark.hadoop.parquet.crypto.factory.class", "org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory") \
        .getOrCreate()


    # Read the encrypted parquet file (automatically decrypted with Hadoop config)
    df = spark.read.parquet(encrypted_path)

Right now, with the existing Python API, I have to set some dummy metadata on the encryption side to satisfy it, so the encryption code becomes:

  std::string keyStr(reinterpret_cast<const char*>(key.data()), key.size());

  // Set up key metadata
  folly::dynamic metadata = folly::dynamic::object("keyMaterialType", "PKMT1")(
      "internalStorage", true)("isFooterKey", true)("doubleWrapping", false)(
      "kmsInstanceID", "dummy_kms_instance_id")(
      "kmsInstanceURL", "dummy_kms_instance_url")(
      "masterKeyID", "dummy_master_key_id")("wrappedDEK", "dummy_wrapped_dek");

  const std::string& footerKeyMetadata = folly::toJson(metadata);

  auto fileEncryptionProps = parquet::FileEncryptionProperties::Builder(keyStr)
                                 .algorithm(parquet::ParquetCipher::AES_GCM_V1)
                                 ->footer_key_metadata(footerKeyMetadata)
                                 ->build();

  const auto props = parquet::WriterProperties::Builder()
                         .encryption(fileEncryptionProps)
                         ->build();

  const auto fields = convertSchema(table->schema());
  const auto schemaNode = std::static_pointer_cast<parquet::schema::GroupNode>(
      parquet::schema::GroupNode::Make(
          "schema", parquet::Repetition::REQUIRED, fields));
  auto schema =
      std::static_pointer_cast<parquet::schema::GroupNode>(schemaNode);

  // Open output file
  std::shared_ptr<arrow::io::FileOutputStream> outFile;
  auto result = arrow::io::FileOutputStream::Open(outputFilePath);
  outFile = result.ValueOrDie();

  auto parquetStreamWriter = make_unique<parquet::StreamWriter>(
      parquet::ParquetFileWriter::Open(outFile, schema, props));

  return writeToStreamWriter(table, *(parquetStreamWriter.get()));

Because of this workaround, we are not blocked.

If we want to keep using table = pq.read_table(path, decryption_properties=file_decryption_props), is there a way to identify through decryption_properties that a DEK is already provided, and that no more metadata is required? Right now, metadata parsing is required before DEK retrieval.

To add to this, our interpretation of section 4.3 Parquet encryption specification is that for keys fully managed by the caller code, metadata section is not required

Key metadata can also be empty - in a case the encryption keys are fully managed by the caller code, and passed explicitly to Parquet readers for the file footer and each encrypted column.

So we were looking for ways to avoid providing dummy data for the purpose of using our own encryption keys

Above you describe your use case as:

A parquet file is encrypted with a DEK.

The DEK is encrypted with KEK.

The encrypted DEK and related metadata (i.e., KMS info to get KEK) is printed as JSON in a metadata file.

The parquet file and metadata file are uploaded to a cloud location.

This is exactly what the higher level API does for you if you set internal_key_material to false (although it also uses double wrapping by default, but you can disable that too). So I would suggest looking more closely at the high level API to see if it can work for you, rather than re-implement this yourself.

That said, I think it would be OK to expose the lower-level API in Python for use cases where users want more control over how keys are managed, as long as the downsides are clearly documented and users are directed to the higher level API by default (the C++ docs/examples could probably be improved here).

I believe PyArrow is the only Parquet implementation that supports encryption but doesn't expose the lower level API, so this would seem to actually hinder compatibility. For example, I'm not super familiar with parquet-java, but I think you can set spark.hadoop.parquet.crypto.factory.class to your own class that implements EncryptionPropertiesFactory and DecryptionPropertiesFactory, instead of having to use the PropertiesDrivenCryptoFactory.

+1
In the old thread, my main concern was the lack of a full implementation of the high level API in PyArrow at the time. This would've led to incompatibility with Spark etc, and to a risk of producing files with broken encryption.
Now that the high level API is available for all developers, it would be fine to add an access to the low level APIs for expert use - with a proper documentation outlining the risks and pointing to the high level APIs

@changhu-m

We seem to have consensus on introducing some low-level API in Python. @changhu-m @akoshelev would internal_key_material set to false (Python would need to expose this too) plus addition of EncryptionPropertiesFactory and DecryptionPropertiesFactory solve your issue?

@adamreeve

Thanks @adamreeve, @rok, and @ggershinsky for responding to this thread. The reason we chose to use an externally managed metadata file was to support multiple file types with the same encryption scheme. While Parquet natively supports envelope encryption, other file formats like CSV do not.

I think the low-level API exposed in Python would work for us because we can accomplish what we need in C++. However, I am not sure about internal_key_material, as it still seems to require Pyarrow to manage encryption keys, and the format of this metadata file will not be the same as the one we use on our side. While this solution achieves the same goal in principle, it would require migrating the decryption-side code if we decide to use it.

Overall, exposing a low-level API in Python would be the preferred approach for us.

Yes internal_key_material is only relevant if using the higher level API (and it does look like it's exposed in Python already).

Overall, exposing a low-level API in Python would be the preferred approach for us.

OK great. Is this something you would be able to contribute?

The Parquet writing and reading methods in PyArrow already accept FileEncryptionProperties or FileDecryptionProperties, but at the moment you can only generate them using a CryptoFactory. You'd need to add methods to construct these directly using the encryption keys and key metadata.

@adamreeve

@adamreeve - I can take a stab at it. Should I follow the builder pattern in c++?

Like this:

arrow/cpp/src/parquet/encryption/encryption.cc

Line 86 in 2284234

    
           FileDecryptionProperties::Builder* FileDecryptionProperties::Builder::column_keys(

arrow/cpp/src/parquet/encryption/encryption.cc

Line 154 in 2284234

    
           FileEncryptionProperties::Builder* FileEncryptionProperties::Builder::footer_key_metadata(

@changhu-m

@changhu-m if there's a reason we'd need a builder or a factory go for it.

My impression is the builder pattern isn't used as much in the Python API. You could just provide an __init__ method that uses the C++ builder internally if that works, but maybe there's a good reason to prefer a builder.

I guess it's used more in C++ to help with error handling and to avoid throwing exceptions in constructors.

Status update on this thread: I am de-prioritizing this since it is now not on our project's critical path. I may still get to it in my free time, and I will update this thread once I have a diff.

@changhu-m

@changhu-m see #48009 too

Thanks all for the discussion! Put up #49667 for the Python API here

GH-47435: [Python][Parquet] Add direct key encryption/decryption API (#…

Issue resolved by pull request 49667
#49667

apacheGH-47435: [Python][Parquet] Add direct key encryption/decryptio…

Navigation Menu

Repository navigation

[Parquet][Python] API to decrypt parquet file using one DEK and no metadata #47435

Describe the enhancement requested

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Parquet][Python] API to decrypt parquet file using one DEK and no metadata #47435

Description

Describe the enhancement requested

Component(s)

Activity

raulcd commented on Aug 27, 2025

rok commented on Aug 28, 2025

rok commented on Aug 28, 2025

changhu-m commented on Aug 28, 2025

akoshelev commented on Aug 28, 2025

rok commented on Aug 28, 2025

adamreeve commented on Aug 28, 2025

rok commented on Aug 29, 2025

changhu-m commented on Aug 29, 2025

akoshelev commented on Aug 29, 2025

adamreeve commented on Aug 31, 2025

ggershinsky commented on Sep 1, 2025

rok commented on Sep 1, 2025

akoshelev commented on Sep 1, 2025

adamreeve commented on Sep 1, 2025

changhu-m commented on Sep 3, 2025

rok commented on Sep 3, 2025

adamreeve commented on Sep 3, 2025

changhu-m commented on Sep 11, 2025

rok commented on Nov 11, 2025

smaheshwar-pltr commented on Apr 22, 2026

adamreeve commented on May 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions