Further performance improvements for build_asset_reconciliation_sensor.
Dagster now allows you to backfill asset selections that include mapped partition definitions, such as a daily asset which rolls up into a weekly asset, as long as the root assets in your selection share a partition definition.
Dagit now includes information about the cause of an asset’s staleness.
Improved the error message for non-matching cron schedules in TimeWindowPartitionMappings with offsets. (Thanks Sean Han!)
[dagster-aws] The EcsRunLauncher now allows you to configure the runtimePlatform field for the task definitions of the runs that it launches, allowing it to launch runs using Windows Docker images.
[dagster-azure] Add support for DefaultAzureCredential for adls2_resource (Thanks Martin Picard!)
[dagster-databricks] Added op factories to create ops for running existing Databricks jobs (create_databricks_run_now_op), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op). See the new Databricks guide for more details.
[dagster-duckdb-polars] Added a dagster-duckdb-polars library that includes a DuckDBPolarsTypeHandler for use with build_duckdb_io_manager, which allows loading / storing Polars DataFrames from/to DuckDB. (Thanks Pezhman Zarabadi-Poor!)
[dagster-gcp-pyspark] New PySpark TypeHandler for the BigQuery I/O manager. Store and load your PySpark DataFrames in BigQuery using bigquery_pyspark_io_manager.
[dagster-snowflake][dagster-duckdb] The Snowflake and DuckDB IO managers can now load multiple partitions in a single step - e.g. when a non-partitioned asset depends on a partitioned asset or a single partition of an asset depends on multiple partitions of an upstream asset. Loading occurs using a single SQL query and returns a single DataFrame.
[dagster-k8s] The Helm chart now supports the full kubernetes env var spec for user code deployments. Example:
dagster-user-deployments: deployments: - name: my-code env: - name: FOO valueFrom: fieldFre: fieldPath: metadata.uid
If includeConfigInLaunchedRuns is enabled, these env vars will also be applied to the containers for launched runs.
AssetSelection which matched no assets was passed into define_asset_job, the resulting job would target all assets in the repository. This has been fixed.build_schedule_from_partitioned_job.fs_io_manager to store multipartitioned materializations in directory levels by dimension. This resolves a bug on windows where multipartitioned materializations could not be stored with the fs_io_manager.context.partition_key would raise an error when executing on a partition range within a single run via Dagit.Definitions reloaded messages on fresh page loads.location information.ProgrammingError was imported from the wrong library, this has been fixed. Thanks @herbert-allium!Output objects rather than using Dagster’s auto-generated versions.get_asset_provenance method on OpExecutionContext allows fetching logical version provenance for an arbitrary asset key.examples/assets_dynamic_partitions folderThe new @graph_asset and @graph_multi_asset decorators make it more ergonomic to define graph-backed assets.
Dagster will auto-infer dependency relationships between single-dimensionally partitioned assets and multipartitioned assets, when the single-dimensional partitions definition is a dimension of the MultiPartitionsDefinition.
A new Test sensor / Test schedule button that allows you to perform a dry-run of your sensor / schedule. Check out the docs on this functionality here for sensors and here for schedules.
[dagit] Added (back) tag autocompletion in the runs filter, now with improved query performance.
[dagit] The Dagster libraries and their versions that were used when loading definitions can now be viewed in the actions menu for each code location.
New bigquery_pandas_io_manager can store and load Pandas dataframes in BigQuery.
[dagster-snowflake, dagster-duckdb] SnowflakeIOManagers and DuckDBIOManagers can now default to loading inputs as a specified type if a type annotation does not exist for the input.
[dagster-dbt] Added the ability to use the “state:” selector
[dagster-k8s] The Helm chart now supports the full kubernetes env var spec for Dagit and the Daemon. E.g.
dagit: env: - name: “FOO” valueFrom: fieldRef: fieldPath: metadata.uid
ruff, pyright).FreshnessPolicy object now supports a cron_schedule_timezone argument.AssetsDefinition.from_graph now supports a freshness_policies_by_output_name parameter.@asset_sensor will now display an informative SkipReason when no new materializations have been created since the last sensor tick.AssetsDefinition now has a to_source_asset method, which returns a representation of this asset as a SourceAsset.from dagster import asset, job, op @asset def emails_to_send(): ... @op def send_emails(emails) -> None: ... @job def send_emails_job(): send_emails(emails_to_send.to_source_asset())
--dagit-host/-h argument to the dagster dev command to allow customization of the host where Dagit runs.input_manager_key to an asset’s input, incorrect resource config could be used when loading that input. This has been fixed.runK8sConfig in the Dagster Helm chart would not pass configuration through to pods launched using the k8s_job_executor.execute_k8s_job op downstream of a dynamic output would result in k8s jobs with duplicate names being created. This has been fixed.asset_key, asset_partitions, and asset_partitions_defs arguments on Out.@multi_asset no longer accepts Out values in the dictionary passed to its outs argument. This was experimental and deprecated. Instead, use AssetOut.top_level_resources argument to the repository decorator has been renamed to _top_level_resources to emphasize that it should not be set manually.load_asset_values now accepts resource configuration (thanks @Nintorac!)UPathIOManager, paths with the "." character in them would be incorrectly truncated, which could result in multiple distinct objects being written to the same path. This has been fixed. (Thanks @spenczar!)PartitionsDefinitions (e.g. HourlyPartitionsDefinition, DailyPartitionsDefinition) may now have a FreshnessPolicy.load_assets_from_dbt_project or load_assets_from_dbt_manifest with dbt-core>=1.4, AssetMaterialization events will be emitted as the dbt command executes, rather than waiting for dbt to complete before emitting events.make_ephemeral_airflow_db_resource which returns a ResourceDefinition for a local only airflow database for use in migrated airflow DAGsdagster instance migrate.observable_source_asset incorrectly required its function to have a context argument.dagster-airbytedagster-dbt integration was incompatible with dbt-core>=1.4. This has been fixed.load_assets_from_dbt_cloud_job will now avoid unnecessarily generating docs when compiling a manifest for the job. Compile runs will no longer be kicked off for jobs not managed by this integration.context.asset_partition_key returned a string instead of a MultiPartitionKey. This has been fixed.k8s_job_executor would sometimes unexpectedly fail due to transient 401 errors in certain kubernetes clusters.load_assets_from_airflow_dag no longer creates airflow db resource definitions, as a user you will need to provide them on Definitions directlypartitions_fn argument of the DynamicPartitionsDefinition class is now deprecated and will be removed in 2.0.0.asset (rather than using @asset in decorator form) to create an asset. Thanks @ns-finkelstein!dagster-airflow library as been moved to 1.x.x to denote the stability of its api's going forward.make_schedules_and_jobs_from_airflow_dag_bag has been added to allow for more fine grained composition of your transformed airflow DAGs into Dagster.retries and retry_delay configuration are now converted to op RetryPolicies with all make_dagster_* apis.0 5 * * mon#1 to execute on a certain day of the week each month executed every week instead.use_airflow_template_context, mock_xcom and use_ephemeral_airflow_db params have been dropped, by default all make_dagster_* apis now use a run-scoped airflow db, similiar to how use_ephemeral_airflow_db worked.make_airflow_dag has been removed.make_airflow_dag_for_operator has been removed.make_airflow_dag_containerized has been removed.airflow_operator_to_op has been removed.make_dagster_repo_from_airflow_dags_path has been removed.make_dagster_repo_from_airflow_dag_bag has been removed.make_dagster_repo_from_airflow_example_dags has been removed.${dag_id}__${task_id} from airflow_${task_id}_${unique_int}.${dag_id} from airflow_${dag_id}.