Repository navigation
Add lifecycle.depends_on to order resources without references #7120
+2,767
−128
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * Added `lifecycle.depends_on` to order resources without referencing their fields. ([#7120](https://github.com/databricks/cli/pull/7120)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| bundle: | ||
| name: test-bundle | ||
|
|
||
| resources: | ||
| jobs: | ||
| a_job: | ||
| name: a_job | ||
| lifecycle: | ||
| depends_on: | ||
| - resource: a_job | ||
| - resource: missing | ||
| b_job: | ||
| name: b_job | ||
| lifecycle: | ||
| depends_on: | ||
| - resource: a.b.c | ||
| # lifecycle is a separate type on job_runs. | ||
| job_runs: | ||
| a_run: | ||
| job_id: ${resources.jobs.b_job.id} | ||
| lifecycle: | ||
| triggers: | ||
| - on_bundle_deploy: true | ||
| depends_on: | ||
| - resource: missing_for_run |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["", "true"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| Error: invalid lifecycle.depends_on: no such resource: missing_for_run | ||
| at resources.job_runs.a_run.lifecycle.depends_on[0].resource | ||
| in databricks.yml:25:23 | ||
|
|
||
| Error: invalid lifecycle.depends_on: resource "a_job" cannot depend on itself | ||
| at resources.jobs.a_job.lifecycle.depends_on[0].resource | ||
| in databricks.yml:10:23 | ||
|
|
||
| Error: invalid lifecycle.depends_on: no such resource: missing | ||
| at resources.jobs.a_job.lifecycle.depends_on[1].resource | ||
| in databricks.yml:11:23 | ||
|
|
||
| Error: invalid lifecycle.depends_on: no such resource: a.b.c | ||
| at resources.jobs.b_job.lifecycle.depends_on[0].resource | ||
| in databricks.yml:16:23 | ||
|
|
||
| Name: test-bundle | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default | ||
|
|
||
| Found 4 errors |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| musterr $CLI bundle validate |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| Ignore = [".databricks"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,17 @@ | ||||||||||||||||
| bundle: | ||||||||||||||||
| name: test-bundle | ||||||||||||||||
|
|
||||||||||||||||
| resources: | ||||||||||||||||
| jobs: | ||||||||||||||||
| # No references between the jobs: the order comes from lifecycle.depends_on only. | ||||||||||||||||
| a_job: | ||||||||||||||||
| name: a_job | ||||||||||||||||
| lifecycle: | ||||||||||||||||
| depends_on: | ||||||||||||||||
| - resource: z_job | ||||||||||||||||
| - resource: pipelines.z_pipeline | ||||||||||||||||
|
Comment on lines
+10
to
+12
Member
There was a problem hiding this comment. might be nice to be able to do this too:
Suggested change
Contributor
Author
There was a problem hiding this comment. why, what would it help with? I find it confusing because according to our current $ references rules, it literally pastes the definition of y_job into depends_on block.
Member
There was a problem hiding this comment.
ah, I didn't know we paste verbatim. then nvm.
quickly see from looking at the yml what part of this is static vs dynamic content
Contributor
Author
There was a problem hiding this comment. not sure what you mean -- it is static, the key of the resource we depend on is a literal.
Member
There was a problem hiding this comment. poor choice of words, not static vs static but maybe "belongs to the resource no matter what else changes" (its name; other lifecycle fields) vs "value may need to be changed due to changes external to this resource" (table name from another resource; key of another resource). This is purely visual and of course lifecycle:
depends_on:
- resources: jobs.foo # -> jobs.barvs lifecycle:
depends_on:
- ${resources.jobs.foo} # -> ${resources.jobs.bar}will require the exact same refactor if someone renames the key of (& rebinds) the depended-on resource. I personally would just find the config easier to (re)view with a visual cue that this is a "special" value (a reference to a key, not just a string). |
||||||||||||||||
| z_job: | ||||||||||||||||
| name: z_job | ||||||||||||||||
| pipelines: | ||||||||||||||||
| z_pipeline: | ||||||||||||||||
| name: z_pipeline | ||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["", "true"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
|
|
||
| === Plan records the edges | ||
| >>> [CLI] bundle plan -o json | ||
| { | ||
| "resources.jobs.a_job": [ | ||
| { | ||
| "node": "resources.jobs.z_job", | ||
| "label": "depends_on" | ||
| }, | ||
| { | ||
| "node": "resources.pipelines.z_pipeline", | ||
| "label": "depends_on" | ||
| } | ||
| ], | ||
| "resources.jobs.z_job": null, | ||
| "resources.pipelines.z_pipeline": null | ||
| } | ||
|
|
||
| === Deploy creates dependencies first | ||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files... | ||
| Created jobs.a_job | ||
| Created jobs.z_job | ||
| Created pipelines.z_pipeline | ||
| Files: 1 uploaded, 0 deleted | ||
| Resources: 3 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| >>> print_requests.py //jobs //pipelines | ||
| POST /api/2.2/jobs/create a_job | ||
|
|
||
| === Destroy deletes dependents first | ||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.jobs.a_job | ||
| delete resources.jobs.z_job | ||
| delete resources.pipelines.z_pipeline | ||
|
|
||
| This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the | ||
| Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: | ||
| delete resources.pipelines.z_pipeline | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default | ||
|
|
||
| Destroy: 3 deleted | ||
| a_job deleted first |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| title "Plan records the edges" | ||
| trace $CLI bundle plan -o json | jq '.plan | map_values(.depends_on)' | ||
|
|
||
| title "Deploy creates dependencies first" | ||
| trace $CLI bundle deploy | ||
| # z_job and z_pipeline are unordered relative to each other, so only check that a_job is last. | ||
| trace print_requests.py //jobs //pipelines | jq -r '.method + " " + .path + " " + (.body.name // "")' | tail -n 1 | ||
|
|
||
| rm -f out.requests.txt | ||
|
|
||
| title "Destroy deletes dependents first" | ||
| a_job_id=$($CLI bundle summary -o json | jq -r .resources.jobs.a_job.id) | ||
| trace $CLI bundle destroy --auto-approve | ||
| first_deleted=$(print_requests.py //jobs/delete | jq -r .body.job_id | head -n 1) | ||
| test "$first_deleted" = "$a_job_id" && echo "a_job deleted first" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Cloud = false | ||
| RecordRequests = true | ||
| Ignore = [".databricks"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| bundle: {name: test-bundle} | ||
|
|
||
| resources: | ||
| jobs: | ||
| test_job: | ||
| name: "Test Auto-Migrate Job" | ||
| # Added after the terraform deployment; terraform has no lifecycle.depends_on. | ||
| other_job: | ||
| name: "Other Job" | ||
| lifecycle: | ||
| depends_on: | ||
| - resource: test_job |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = [""] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
|
|
||
| === Replay a prior terraform deployment | ||
| >>> replay_tfstate.py terraform-requests.json terraform.tfstate | ||
|
|
||
| === Deploy migrates to direct and honors lifecycle.depends_on | ||
| >>> [CLI] bundle deploy | ||
| Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765). | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files... | ||
| Migrated 1 resource to direct deployment engine. | ||
| Created jobs.other_job | ||
| Files: 1 uploaded, 0 deleted | ||
| Resources: 1 created, 0 changed, 0 deleted, 1 unchanged | ||
|
|
||
| >>> print_migration_telemetry | ||
| direct_migrated_via_default true | ||
|
|
||
| === The edge is recorded in the plan | ||
| >>> [CLI] bundle plan -o json | ||
| { | ||
| "resources.jobs.other_job": [ | ||
| { | ||
| "node": "resources.jobs.test_job", | ||
| "label": "depends_on" | ||
| } | ||
| ], | ||
| "resources.jobs.test_job": null | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| export DATABRICKS_BUNDLE_ENGINE= | ||
|
|
||
| title "Replay a prior terraform deployment" | ||
| trace replay_tfstate.py terraform-requests.json terraform.tfstate | ||
| replace_ids.py | ||
| rm -f "$OUT_REQUESTS" | ||
|
|
||
| title "Deploy migrates to direct and honors lifecycle.depends_on" | ||
| trace $CLI bundle deploy | ||
| trace print_migration_telemetry | ||
|
|
||
| title "The edge is recorded in the plan" | ||
| trace $CLI bundle plan -o json | jq '.plan | map_values(.depends_on)' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| [ | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.2/jobs/create", | ||
| "body": { | ||
| "deployment": { | ||
| "kind": "BUNDLE", | ||
| "metadata_file_path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/state/metadata.json" | ||
| }, | ||
| "edit_mode": "UI_LOCKED", | ||
| "format": "MULTI_TASK", | ||
| "max_concurrent_runs": 1, | ||
| "name": "Test Auto-Migrate Job", | ||
| "queue": { | ||
| "enabled": true | ||
| } | ||
| } | ||
| } | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| { | ||
| "version": 4, | ||
| "terraform_version": "1.5.5", | ||
| "serial": 1, | ||
| "lineage": "c39488aa-8f36-06fd-a7cb-3f0c95b4c506", | ||
| "outputs": {}, | ||
| "resources": [ | ||
| { | ||
| "mode": "managed", | ||
| "type": "databricks_job", | ||
| "name": "test_job", | ||
| "provider": "provider[\"registry.terraform.io/databricks/databricks\"]", | ||
| "instances": [ | ||
| { | ||
| "schema_version": 2, | ||
| "attributes": { | ||
| "always_running": false, | ||
| "budget_policy_id": null, | ||
| "continuous": [], | ||
| "control_run_state": false, | ||
| "dbt_task": [], | ||
| "deployment": [ | ||
| { | ||
| "deployment_id": "", | ||
| "kind": "BUNDLE", | ||
| "metadata_file_path": "/Workspace/Users/tester@databricks.com/.bundle/test-bundle/default/state/metadata.json", | ||
| "version_id": "" | ||
| } | ||
| ], | ||
| "description": null, | ||
| "edit_mode": "UI_LOCKED", | ||
| "email_notifications": [ | ||
| { | ||
| "no_alert_for_skipped_runs": false, | ||
| "on_duration_warning_threshold_exceeded": [], | ||
| "on_failure": [], | ||
| "on_start": [], | ||
| "on_streaming_backlog_exceeded": [], | ||
| "on_success": [] | ||
| } | ||
| ], | ||
| "environment": [], | ||
| "existing_cluster_id": null, | ||
| "format": "MULTI_TASK", | ||
| "git_source": [], | ||
| "health": [], | ||
| "id": "8790612527183335000", | ||
| "job_cluster": [], | ||
| "library": [], | ||
| "max_concurrent_runs": 1, | ||
| "max_retries": 0, | ||
| "min_retry_interval_millis": 0, | ||
| "name": "Test Auto-Migrate Job", | ||
| "new_cluster": [], | ||
| "notebook_task": [], | ||
| "notification_settings": [], | ||
| "parameter": [], | ||
| "parent_path": null, | ||
| "performance_target": null, | ||
| "pipeline_task": [], | ||
| "provider_config": [ | ||
| { | ||
| "workspace_id": "900800700600" | ||
| } | ||
| ], | ||
| "python_wheel_task": [], | ||
| "queue": [ | ||
| { | ||
| "enabled": true | ||
| } | ||
| ], | ||
| "retry_on_timeout": false, | ||
| "run_as": [ | ||
| { | ||
| "group_name": "", | ||
| "service_principal_name": "", | ||
| "user_name": "tester@databricks.com" | ||
| } | ||
| ], | ||
| "run_job_task": [], | ||
| "schedule": [], | ||
| "spark_jar_task": [], | ||
| "spark_python_task": [], | ||
| "spark_submit_task": [], | ||
| "tags": null, | ||
| "task": [], | ||
| "timeout_seconds": 0, | ||
| "timeouts": null, | ||
| "trigger": [], | ||
| "triggers": [], | ||
| "url": "[DATABRICKS_URL]/#job/8790612527183335000", | ||
| "usage_policy_id": null, | ||
| "webhook_notifications": [ | ||
| { | ||
| "on_duration_warning_threshold_exceeded": [], | ||
| "on_failure": [], | ||
| "on_start": [], | ||
| "on_streaming_backlog_exceeded": [], | ||
| "on_success": [] | ||
| } | ||
| ] | ||
| }, | ||
| "sensitive_attributes": [], | ||
| "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJ1cGRhdGUiOjE4MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjIifQ==" | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "check_results": null | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looking at this example I would conclude that
z_jobonly works because it's under the sameresources.jobsresource type and z_pipeline needs thepipeline.prefix. But I assume that's not true since we require unique keys. Can you add the other permutations (jobs.z_jobandz_pipeline) as well?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see what you mean, I think it's fine like this because we don't implement resolution of these keys in this PR, it's standard since #7128
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but that PR didn't add acceptance test coverage for that behaviour. I find these example databricks.yml always give a good insight into currently supported behaviour. Would be great to add some (can be a follow-up PR)