From 5b38d856905231e54bdd81fe9d10874d299cf776 Mon Sep 17 00:00:00 2001 From: cbergh Date: Wed, 21 Oct 2026 14:57:31 -0400 Subject: [PATCH 1/3] docs(readme): rewrite around a plain definition, databases, MCP, and editions --- README.md | 138 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 97 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 49cfbe5c..17a3ff38 100644 --- a/README.md +++ b/README.md @@ -1,76 +1,132 @@ -# DataOps Data Quality TestGen +# DataOps TestGen ![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green) [![Latest Version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2Ftags%2F&query=results%5B0%5D.name&label=latest%20version&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Docker Pulls](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2F&query=pull_count&style=flat&label=docker%20pulls&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Documentation](https://img.shields.io/badge/docs-On%20datakitchen.io-06A04A?style=flat)](https://docs.datakitchen.io/testgen/what-is-testgen/) [![Static Badge](https://img.shields.io/badge/Slack-Join%20Discussion-blue?style=flat&logo=slack)](https://data-observability-slack.datakitchen.io/join) -*

DataOps Data Quality TestGen, or "TestGen" for short, can help you find data issues so you can alert your users and notify your suppliers. It does this by delivering simple, fast data quality test generation and execution by data profiling, new dataset screening and hygiene review, algorithmic generation of data quality validation tests, ongoing production testing of new data refreshes, and continuous anomaly monitoring of datasets. TestGen is part of DataKitchen's Open Source Data Observability.

* +DataOps TestGen is an open-source data quality tool that profiles tables, generates tests from the profiling results, and runs them on a schedule with freshness, volume, and schema monitoring. -## Documentation - -[DataOps TestGen Overview](https://datakitchen.io/dataops-testgen-product/) - -[DataOps TestGen Documentation](https://docs.datakitchen.io/testgen/what-is-testgen/) +The open-source edition includes the full web UI, not just a library. You also get a CLI and an MCP server, so you can drive TestGen from a terminal or an AI assistant. It is Apache 2.0 licensed and runs the same engine and UI as TestGen Enterprise. +[Interactive Product Tour](https://datakitchen.storylane.io/share/byag8vimd5tn) | [Product page](https://datakitchen.io/products/dataops-testgen/) | [Documentation](https://docs.datakitchen.io/testgen/what-is-testgen/) -## Features +## What it does -[Interactive Product Tour](https://datakitchen.storylane.io/share/byag8vimd5tn) +- Profiles every column in a table group and records type-specific statistics. +- Flags hygiene issues in new data, so you can review a dataset before it reaches production and send problems back to the supplier. +- Generates data quality validation tests from the profile. You can add your own SQL tests too. +- Runs tests on a schedule, as SQL inside your database. Your data stays where it is. +- Watches each table between test runs with freshness, volume, and schema monitors. +- Gives you one place to manage data quality across datasets, locations, and teams. -What does DataKitchen's DataOps Data Quality TestGen do? It helps you understand and find data issues in new data.

-DataKitchen Open Source Data Quality TestGen Features - New Data +DataOps TestGen: hygiene review of new data

-It constantly watches your data for data quality anomalies and lets you drill into problems. -

-DataKitchen Open Source Data Quality TestGen Features - Data Ingestion and Quality Testing +DataOps TestGen: continuous monitoring and quality testing

-A single place to manage Data Quality across data sets, locations, and teams. -

-DataKitchen Open Source Data Quality TestGen Features - Single Place +DataOps TestGen: one place to manage data quality

-## Installation +## Supported databases + +As of TestGen 5.134.0, TestGen connects to 14 databases: + +- Snowflake +- Databricks SQL +- DuckDB +- Azure Synapse Analytics +- Azure SQL Database +- Microsoft SQL Server +- Microsoft OneLake +- Google BigQuery +- Amazon Redshift +- Amazon Aurora PostgreSQL +- Oracle Database 12c and later +- SAP HANA +- PostgreSQL +- Salesforce Data 360 + +TestGen also reads CSV, Parquet, and JSON files on a local path, in Amazon S3, or in Azure Blob Storage, with no database in front. Every source works in both the open-source and Enterprise editions. + +To set up a connection, see [Connect your data](https://docs.datakitchen.io/testgen/connect-your-data/). + +## Quick start + +The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) program installs TestGen in either Docker or pip mode. + +You need 2 CPUs, 8 GB memory, 20 GB disk space. pip mode needs [Python](https://www.python.org/downloads/) 3.9+ to run the installer. Docker mode also needs [Docker](https://docs.docker.com/get-docker/) 27+ and [Docker Compose](https://docs.docker.com/compose/install/) 5.0+. -The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) program installs TestGen in either Docker or pip mode. For complete instructions, see the documentation: +Download the installer into a new, empty directory: + +```shell +curl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py' +``` + +On Windows, you can instead download [`dk-installer.exe`](https://github.com/DataKitchen/data-observability-installer/releases/download/latest/dk-installer.exe) and double-click it. + +Install TestGen: + +```shell +python3 dk-installer.py tg install +``` + +The installer checks for Docker and asks you to pick Docker or pip. Pass `--docker` or `--pip` to skip the prompt. When it finishes, it writes credentials to `dk-tg-credentials.txt`, generates demo data, and opens the TestGen UI in your browser. Use `--no-demo` to skip the demo data. + +Upgrade later with `python3 dk-installer.py tg upgrade`. Restart with `python3 dk-installer.py tg start`. + +For complete instructions, see the documentation: * [Install on Mac/Linux](https://docs.datakitchen.io/testgen/get-started/install-on-mac-linux/) * [Install on Windows](https://docs.datakitchen.io/testgen/get-started/install-on-windows/) -## What Next? +Then walk through the [Data Observability Overview Demo](https://docs.datakitchen.io/tutorials/quickstart-demo/), and [connect your own database](https://docs.datakitchen.io/testgen/connect-your-data/). -### Getting started guide -We recommend you start by going through the [Data Observability Overview Demo](https://docs.datakitchen.io/tutorials/quickstart-demo/). +## Use it from an AI assistant (MCP) -### Support -For support requests, [join the Data Observability Slack](https://data-observability-slack.datakitchen.io/join) 👋 and post on the `#support` channel. +TestGen ships a Model Context Protocol (MCP) server in the open-source edition. Connect Claude, Claude Code, Cursor, GitHub Copilot, or any MCP client, and ask about your data in plain English. The assistant can profile tables, generate and run tests, and triage failures. Every tool runs under your own TestGen role, so the assistant can do only what you can do. -### Connect to your database -Follow [these instructions](https://docs.datakitchen.io/testgen/connect-your-data/) to improve the quality of data in your database. +The installer serves the API and MCP server on port 8530 by default. Change it with `--api-port`. -### Community -Talk and learn with other data practitioners who are building with DataKitchen. Share knowledge, get help, and contribute to our open-source project. +Setup instructions: [docs.datakitchen.io/testgen/mcp](https://docs.datakitchen.io/testgen/mcp/) -Join our community here: +## Open source vs Enterprise -* 👋 [Join us on Slack](https://data-observability-slack.datakitchen.io/join), this is also how you get support (see above) +Both editions run the same core engine, with every test type, unlimited tables, and unlimited data volume. Enterprise adds multi-user access, unlimited connections and projects, enterprise security, and DataKitchen support. -* 🌟 [Star us on GitHub](https://github.com/DataKitchen/data-observability-installer) +| | Open source | Enterprise | +|---|---|---| +| License | Apache 2.0 | Commercial | +| Users | 1 | Unlimited, $100 per user per month | +| Database connections | 1 | Unlimited, $100 per connection per month | +| Projects | 1 | Unlimited | +| Enterprise security | | Included | +| Support | Community Slack | DataKitchen support | -* 🐦 [Follow us on Twitter](https://twitter.com/i/flow/login?redirect_after_login=%2Fdatakitchen_io) +See [datakitchen.io/pricing](https://datakitchen.io/pricing/) for current pricing. + +## Documentation -* 🕴️ [Follow us on LinkedIn](https://www.linkedin.com/company/datakitchen) +* [DataOps TestGen documentation](https://docs.datakitchen.io/testgen/what-is-testgen/) +* [DataOps TestGen product page](https://datakitchen.io/products/dataops-testgen/) +* [Configuration (environment variables)](docs/configuration.md) +* [Local development](docs/local_development.md) -* 📺 [Get Free DataOps Fundamentals Certification](https://info.datakitchen.io/training-certification-dataops-fundamentals) +## Community -* 📚 [Read our blog posts](https://datakitchen.io/blog/) +Talk with other data practitioners who build with DataKitchen. Share what you know, get help, and contribute. -* 🗃 [Sign The DataOps Manifesto](https://DataOpsManifesto.org) +* [Join us on Slack](https://data-observability-slack.datakitchen.io/join). This is also how you get support: post in the `#support` channel. +* [Star us on GitHub](https://github.com/DataKitchen/dataops-testgen) +* [Follow us on X](https://x.com/datakitchen_io) +* [Follow us on LinkedIn](https://www.linkedin.com/company/datakitchen) +* [Get free DataOps certification](https://datakitchen.io/certifications/) +* [Read our blog](https://datakitchen.io/blog/) +* [Sign The DataOps Manifesto](https://DataOpsManifesto.org) +* [Sign The Data Journey Manifesto](https://DataJourneyManifesto.org) -* 🗃 [Sign The Data Journey Manifesto](https://DataJourneyManifesto.org) +## Contributing +For details on contributing or running the project for development, see the [contributing guide](CONTRIBUTING.md). -### Contributing -For details on contributing or running the project for development, check out our [contributing guide](CONTRIBUTING.md). +## License -### License -DataKitchen's DataOps Data Quality TestGen is Apache 2.0 licensed. +DataOps TestGen is Apache 2.0 licensed. See [LICENSE](LICENSE). From e324b6ac3e452387562e87348f4a95cc1f092d04 Mon Sep 17 00:00:00 2001 From: cbergh Date: Wed, 21 Oct 2026 15:55:10 -0400 Subject: [PATCH 2/3] docs(readme): list Enterprise-only capabilities instead of claiming every test type --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 17a3ff38..1968b8d1 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ DataOps TestGen is an open-source data quality tool that profiles tables, generates tests from the profiling results, and runs them on a schedule with freshness, volume, and schema monitoring. -The open-source edition includes the full web UI, not just a library. You also get a CLI and an MCP server, so you can drive TestGen from a terminal or an AI assistant. It is Apache 2.0 licensed and runs the same engine and UI as TestGen Enterprise. +The open-source edition includes the full web UI, not just a library. You also get a CLI and an MCP server, so you can drive TestGen from a terminal or an AI assistant. It is Apache 2.0 licensed and runs the same core engine and UI as TestGen Enterprise. [Interactive Product Tour](https://datakitchen.storylane.io/share/byag8vimd5tn) | [Product page](https://datakitchen.io/products/dataops-testgen/) | [Documentation](https://docs.datakitchen.io/testgen/what-is-testgen/) @@ -90,7 +90,7 @@ Setup instructions: [docs.datakitchen.io/testgen/mcp](https://docs.datakitchen.i ## Open source vs Enterprise -Both editions run the same core engine, with every test type, unlimited tables, and unlimited data volume. Enterprise adds multi-user access, unlimited connections and projects, enterprise security, and DataKitchen support. +The core profiling, testing, and monitoring engine is free and open source, with unlimited tables and unlimited data volume. Enterprise adds team access, security, Commercial Pharma tests, source-data controls for AI, and support. | | Open source | Enterprise | |---|---|---| @@ -99,6 +99,8 @@ Both editions run the same core engine, with every test type, unlimited tables, | Database connections | 1 | Unlimited, $100 per connection per month | | Projects | 1 | Unlimited | | Enterprise security | | Included | +| Commercial Pharma test types (12) | | Included | +| Hide source data from AI clients (MCP) | | Included | | Support | Community Slack | DataKitchen support | See [datakitchen.io/pricing](https://datakitchen.io/pricing/) for current pricing. From 389fac59efe647ff2d3411148c7d3f9a2819a720 Mon Sep 17 00:00:00 2001 From: Aarthy Adityan Date: Wed, 21 Oct 2026 16:54:13 -0400 Subject: [PATCH 3/3] docs(readme): replace CLI mention with API --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1968b8d1..f91361ea 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ DataOps TestGen is an open-source data quality tool that profiles tables, generates tests from the profiling results, and runs them on a schedule with freshness, volume, and schema monitoring. -The open-source edition includes the full web UI, not just a library. You also get a CLI and an MCP server, so you can drive TestGen from a terminal or an AI assistant. It is Apache 2.0 licensed and runs the same core engine and UI as TestGen Enterprise. +The open-source edition includes the full web UI, not just a library. You also get a REST API and an MCP server, so you can drive TestGen from your pipelines or an AI assistant. It is Apache 2.0 licensed and runs the same core engine and UI as TestGen Enterprise. [Interactive Product Tour](https://datakitchen.storylane.io/share/byag8vimd5tn) | [Product page](https://datakitchen.io/products/dataops-testgen/) | [Documentation](https://docs.datakitchen.io/testgen/what-is-testgen/)