Libraries
You can integrate Dagster with external services using our libraries and libraries supported by the community.
Airbyte
2 items
Airlift
3 items
Anthropic
The dagster-anthropic library allows you to easily interact with the Anthropic REST API using the Anthropic Python API to build AI steps into your Dagster pipelines. You can also log Anthropic API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption.
AWS
10 items
Azure Data Lake Storage Gen 2
Dagster helps you use Azure Storage Accounts as part of your data pipeline. Azure Data Lake Storage Gen 2 (ADLS2) is our primary focus but we also provide utilities for Azure Blob Storage.
Census
Community / Partner supported
With the dagster-census integration you can execute a Census sync and poll until that sync completes, raising an error if it's unsuccessful.
Chroma
The dagster-chroma library allows you to easily interact with Chroma's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets.
Cube
Community / Partner supported
With the dagster_cube integration you can setup Cube and Dagster to work together so that Dagster can push changes from upstream data sources to Cube using its integration API.
Databricks
The dagster-databricks integration library provides the PipesDatabricksClient resource, enabling you to launch Databricks jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Databricks code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
Datadog
While Dagster provides comprehensive monitoring and observability of the pipelines it orchestrates, many teams look to centralize all their monitoring across apps, processes and infrastructure using Datadog's 'Cloud Monitoring as a Service'. The dagster-datadog integration allows you to publish metrics to Datadog from within Dagster ops.
dbt
6 items
Delta Lake
2 items
DingTalk
Community / Partner supported
The community-supported dagster-dingtalk package provides an integration with DingTalk.
dlt
1 item
Docker
The dagster-docker integration library provides the PipesDockerClient resource, enabling you to launch Docker containers and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Docker containers while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
DuckDB
2 items
Embedded ELT
The dagster-embedded-elt package provides a framework for building ELT pipelines with Dagster through helpful asset decorators and resources. It includes the dagster-dlt and dagster-sling packages, which you can also use on their own. To get started,
Fivetran
This guide provides instructions for using Dagster with Fivetran using the dagster-fivetran library. Your Fivetran connector tables can be represented as assets in the Dagster asset graph, allowing you to track lineage and dependencies between Fivetran assets and data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Fivetran connectors, allowing you to trigger syncs for these on a cadence or based on upstream data changes.
GCP
4 items
Gemini
The dagster-gemini library allows you to easily interact with the Gemini REST API using the Gemini Python API to build AI steps into your Dagster pipelines. You can also log Gemini API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption.
GitHub
This library provides an integration with GitHub Apps by providing a thin wrapper on the GitHub v4 GraphQL API. This allows for automating operations within your GitHub repositories and with the tighter permissions scopes that GitHub Apps allow for vs using a personal token.
HashiCorp Nomad
Community / Partner supported
The community-supported dagster-nomad package provides an integration with HashiCorp Nomad.
HashiCorp Vault
Community / Partner supported
Package for integrating HashiCorp Vault into Dagster so that you can securely manage tokens and passwords.
Hex
Community / Partner supported
The community-supported dagster-hex package provides an integration with HashiCorp Nomad.
Hightouch
Community / Partner supported
With this integration you can trigger Hightouch syncs and monitor them from within Dagster. Fine-tune when Hightouch syncs kick-off, visualize their dependencies, and monitor the steps in your data activation workflow.
Libraries
You can integrate Dagster with external services using our libraries and libraries supported by the community.
Jupyter Notebooks
2 items
Kubernetes
The dagster-k8s integration library provides the PipesK8sClient resource, enabling you to launch Kubernetes pods and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Kubernetes pods while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
LakeFS
Community / Partner supported
By integrating with lakeFS, a big data scale version control system, you can leverage the versioning capabilities of lakeFS to track changes to your data. This integration allows you to have a complete lineage of your data, from the initial raw data to the transformed and processed data, making it easier to understand and reproduce data transformations.
Looker
1 item
Meltano
Community / Partner supported
The dagster-meltano library allows you to run Meltano using Dagster. Design and configure ingestion jobs using the popular Singer.io specification.
Microsoft Teams
By configuring this resource, you can post messages to MS Teams from any Dagster op or asset.
Modal
Community / Partner supported
The community-supported dagster-modal package provides an integration with Modal.
MSSQL Bulk Copy Tool
The community-supported dagster-mssql-bcp package is a custom Dagster I/O manager for loading data into SQL Server using the bcp utility.
Open Metadata
Community / Partner supported
With this integration you can create a Open Metadata service to ingest metadata produced by the Dagster application. View the Ingestion Pipeline running from the Open Metadata Service Page.
OpenAI
1 item
PagerDuty
This library provides an integration between Dagster and PagerDuty to support creating alerts from your Dagster code.
Pandas
1 item
Pandera
1 item
Perian
Community / Partner supported
The dagster-perian integration allows you to easily dockerize your codebase and execute it on the PERIAN platform, PERIAN's serverless GPU environment.
Power BI
1 item
Prometheus
This integration allows you to push metrics to the Prometheus gateway from within a Dagster pipeline.
Ray
Community / Partner supported
The community-supported dagster-ray package allows orchestrating distributed Ray compute from Dagster pipelines.
SDF
Community / Partner supported
SDF can integrate seamlessly with your existing Dagster projects, providing the best-in-class transformation layer while enabling you to schedule, orchestrate, and monitor your dags in Dagster.
Secoda
Community / Partner supported
Connect Dagster to Secoda and see metadata related to your Dagster assets, asset groups and jobs right in Secoda. Simplify your team's access, and remove the need to switch between tools.
Bash / Shell
Dagster comes with a native PipesSubprocessClient resource that enables you to launch shell commands directly from Dagster assets and ops. This integration allows you to pass parameters to external shell scripts while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated external execution. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
Sigma
1 item
Slack
This library provides an integration with Slack to support posting messages in your company's Slack workspace.
Sling
This integration allows you to use Sling to extract and load data from popular data sources to destinations with high performance and ease.
Snowflake
3 items
Spark
Spark jobs typically execute on infrastructure that's specialized for Spark. Spark applications are typically not containerized or executed on Kubernetes.
SSH/SFTP
This integration provides a resource for SSH remote execution using Paramiko. It allows you to establish secure connections to networked resources and execute commands remotely. The integration also provides an SFTP client for secure file transfers between the local and remote systems.
Tableau
This integration is currently experimental.
Twilio
Use your Twilio Account SID and Auth Token to build Twilio tasks right into your Dagster pipeline.
Weights & Biases
Community / Partner supported
Use Dagster and Weights & Biases (W&B) to orchestrate your MLOps pipelines and maintain ML assets. The integration with W&B makes it easy within Dagster to:
Weaviate
The dagster-weaviate library allows you to easily interact with Weaviate's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets.