Dagit now allows re-executing arbitrary step subset via step selector syntax, regardless of
whether the previous pipeline failed or not.
Added a search filter for the root Assets page
Adds tooltip explanations for disabled run actions
The last output of the cron job command created by the scheduler is now stored in a file. A new
dagster schedule logs {schedule_name} command will show the log file for a given schedule. This
helps uncover errors like missing environment variables and import errors.
The Dagit schedule page will now show inconsistency errors between schedule state and the cron
tab that were previously only displayed by the dagster schedule debug command. As before, these
errors can be resolve using dagster schedule up
Bugfix
Fixes an issue with config schema validation on Arrays
Fixes an issue with initializing K8sRunLauncher when configured via dagster.yaml
Fixes a race condition in Airflow injection logic that happens when multiple Operators try to
create PipelineRun entries simultaneously.
Fixed an issue with schedules that had invalid config not logging the appropriate error.
dagster pipeline backfill command no longer takes a mode flag. Instead, it uses the mode
specified on the PartitionSetDefinition. Similarly, the runs created from the backfill also use
the solid_subset specified on the PartitionSetDefinition
BugFix
Fixes a bug where using solid subsets when launching pipeline runs would fail config validation.
(dagster-gcp) allow multiple "bq_solid_for_queries" solids to co-exist in a pipeline
Improve scheduler state reconciliation with dagster-cron scheduler. dagster schedule debug
command will display issues related to missing crob jobs, extraneous cron jobs, and duplicate cron
jobs. Running dagster schedule up will fix any issues.
New
The dagster-airflow package now supports loading Airflow dags without depending on initialized
Airflow db
Improvements to the longitudinal partitioned schedule view, including live updates, run filtering,
and better default states.
Added user warning for dagster library packages that are out of sync with the core dagster
package.
Fixed an issue with strict snapshot ID matching when loading historical snapshots, which caused
errors on the Runs page when viewing historical runs.
Fixed an issue where dagster_celery had introduced a spurious dependency on dagster_k8s
(#2435)
Fixed an issue where our Airflow, Celery, and Dask integrations required S3 or GCS storage and
prevented use of filesystem storage. Filesystem storage is now also permitted, to enable use of
these integrations with distributed filesystems like NFS (#2436).
RepositoryDefinition now takes schedule_defs and partition_set_defs directly. The loading
scheme for these definitions via repository.yaml under the scheduler: and partitions: keys
is deprecated and expected to be removed in 0.8.0.
Mark published modules as python 3.8 compatible.
The dagster-airflow package supports loading all Airflow DAGs within a directory path, file path,
or Airflow DagBag.
The dagster-airflow package supports loading all 23 DAGs in Airflow example_dags folder and
execution of 17 of them (see: make_dagster_repo_from_airflow_example_dags).
The dagster-celery CLI tools now allow you to pass additional arguments through to the underlying
celery CLI, e.g., running dagster-celery worker start -n my-worker -- --uid=42 will pass the
--uid flag to celery.
It is now possible to create a PresetDefinition that has no environment defined.
Added dagster schedule debug command to help debug scheduler state.
The SystemCronScheduler now verifies that a cron job has been successfully been added to the
crontab when turning a schedule on, and shows an error message if unsuccessful.
Breaking Changes
A dagster instance migrate is required for this release to support the new experimental assets
view.
Runs created prior to 0.7.8 will no longer render their execution plans as DAGs. We are only
rendering execution plans that have been persisted. Logs are still available.
Path is no longer valid in config schemas. Use str or dagster.String instead.
Removed the @pyspark_solid decorator - its functionality, which was experimental, is subsumed by
requiring a StepLauncher resource (e.g. emr_pyspark_step_launcher) on the solid.
Dagit
Merged "re-execute", "single-step re-execute", "resume/retry" buttons into one "re-execute" button
with three dropdown selections on the Run page.
Experimental
Added new asset_key string parameter to Materializations and created a new “Assets” tab in Dagit
to view pipelines and runs associated with these keys. The API and UI of these asset-based are
likely to change, but feedback is welcome and will be used to inform these changes.
Added an emr_pyspark_step_launcher that enables launching PySpark solids in EMR. The
"simple_pyspark" example demonstrates how it’s used.
Bugfix
Fixed an issue when running Jupyter notebooks in a Python 2 kernel through dagstermill with
Dagster running in Python 3.
Improved error messages produced when dagstermill spins up an in-notebook context.
Fixed an issue with retrieving step events from CompositeSolidResult objects.