Setup

Create MongoDB Sinks

https://www.mongodb.com/docs/atlas/sample-data/sample-mflix/

  1. Create project
  2. Create cluster
  3. Connect to mongoDB driver

Create Snowflake Destinations

use role accountadmin;

create warehouse if not exists dagster_wh with warehouse_size='x-small';
create database if not exists dagster_db;
create role if not exists dagster_role;

grant usage on warehouse dagster_wh to role dagster_role;
grant role dagster_role to user jayzern;
grant all on database dagster_db to role dagster_role;

Create Dagster Project

https://docs.dagster.io/getting-started/install

https://docs.dagster.io/integrations/embedded-elt/dlt

conda create -n dagster python=3.11
conda activate dagster

pip install dagster
pip install dagster-embedded-elt

# Scaffold a project. Or create a template
dagster project scaffold --name dagster_tutorial

Tour of Dagster

https://docs.dagster.io/guides/understanding-dagster-project-files