Changelog#

0.8.7#

Breaking Changes

Loading python modules reliant on the working directory being on the PYTHONPATH is no longer supported. The dagster and dagit CLI commands no longer add the working directory to the PYTHONPATH when resolving modules, which may break some imports. Explicitly installed python packages can be specified in workspaces using the python_package workspace yaml config option. The python_module config option is deprecated and will be removed in a future release.

New

Dagit can be hosted on a sub-path by passing --path-prefix to the dagit CLI. #2073
The date_partition_range util function now accepts an optional inclusive boolean argument. By default, the function does not return include the partition for which the end time of the date range is greater than the current time. If inclusive=True, then the list of partitions returned will include the extra partition.
MultiDependency or fan-in inputs will now only cause the solid step to skip if all of the fanned-in inputs upstream outputs were skipped

Bugfixes

Fixed accidental breaking change with input_hydration_config arguments
Fixed an issue with yaml merging (thanks @shasha79!)
Invoking alias on a solid output will produce a useful error message (thanks @iKintosh!)
Restored missing run pagination controls
Fixed error resolving partition-based schedules created via dagster schedule decorators (e.g. daily_schedule) for certain workspace.yaml formats

0.8.6#

Breaking Changes

The dagster-celery module has been broken apart to manage dependencies more coherently. There are now three modules: dagster-celery, dagster-celery-k8s, and dagster-celery-docker.
Related to above, the dagster-celery worker start command now takes a required -A parameter which must point to the app.py file within the appropriate module. E.g if you are using the celery_k8s_job_executor then you must use the -A dagster_celery_k8s.app option when using the celery or dagster-celery cli tools. Similar for the celery_docker_executor: -A dagster_celery_docker.app must be used.
Renamed the input_hydration_config and output_materialization_config decorators to dagster_type_ and dagster_type_materializer respectively. Renamed DagsterType's input_hydration_config and output_materialization_config arguments to loader and materializer respectively.

New

New pipeline scoped runs tab in Dagit
Add the following Dask Job Queue clusters: moab, sge, lsf, slurm, oar (thanks @DavidKatz-il!)

K8s resource-requirements for run coordinator pods can be specified using the dagster-k8s/ resource_requirements tag on pipeline definitions:


@pipeline(
    tags={
        'dagster-k8s/resource_requirements': {
            'requests': {'cpu': '250m', 'memory': '64Mi'},
            'limits': {'cpu': '500m', 'memory': '2560Mi'},
        }
    },
)
def foo_bar_pipeline():

Added better error messaging in dagit for partition set and schedule configuration errors
An initial version of the CeleryDockerExecutor was added (thanks @mrdrprofuroboros!). The celery workers will launch tasks in docker containers.
Experimental: Great Expectations integration is currently under development in the new library dagster-ge. Example usage can be found here

0.8.5#

Breaking Changes

Python 3.5 is no longer under test.
Engine and ExecutorConfig have been deleted in favor of Executor. Instead of the @executor decorator decorating a function that returns an ExecutorConfig it should now decorate a function that returns an Executor.

New

The python built-in dict can be used as an alias for Permissive() within a config schema declaration.
Use StringSource in the S3ComputeLogManager configuration schema to support using environment variables in the configuration (Thanks @mrdrprofuroboros!)
Improve Backfill CLI help text
Add options to spark_df_output_schema (Thanks @DavidKatz-il!)
Helm: Added support for overriding the PostgreSQL image/version used in the init container checks.
Update celery k8s helm chart to include liveness checks for celery workers and flower
Support step level retries to celery k8s executor

Bugfixes

Improve error message shown when a RepositoryDefinition returns objects that are not one of the allowed definition types (Thanks @sd2k!)
Show error message when $DAGSTER_HOME environment variable is not an absolute path (Thanks @AndersonReyes!)
Update default value for staging_prefix in the DatabricksPySparkStepLauncher configuration to be an absolute path (Thanks @sd2k!)
Improve error message shown when Databricks logs can't be retrieved (Thanks @sd2k!)
Fix errors in documentation fo input_hydration_config (Thanks @joeyfreund!)

0.8.4#

Bugfix

Reverted changed in 0.8.3 that caused error during run launch in certain circumstances
Updated partition graphs on schedule page to select most recent run
Forced reload of partitions for partition sets to ensure not serving stale data

New

Added reload button to dagit to reload current repository
Added option to wipe a single asset key by using dagster asset wipe <asset_key>
Simplified schedule page, removing ticks table, adding tags for last tick attempt
Better debugging tools for launch errors

0.8.3#

Breaking Changes

Previously, the gcs_resource returned a GCSResource wrapper which had a single client property that returned a google.cloud.storage.client.Client. Now, the gcs_resource returns the client directly.
To update solids that use the gcp_resource, change:
```
context.resources.gcs.client
```
To:
```
context.resources.gcs
```

New

Introduced a new Python API reexecute_pipeline to reexecute an existing pipeline run.
Performance improvements in Pipeline Overview and other pages.
Long metadata entries in the asset details view are now scrollable.
Added a project field to the gcs_resource in dagster_gcp.
Added new CLI command dagster asset wipe to remove all existing asset keys.

Bugfix

Several Dagit bugfixes and performance improvements
Fixes pipeline execution issue with custom run launchers that call executeRunInProcess.
Updates dagster schedule up output to be repository location scoped