dbt 基本试用
dbt 是一个很不错的进行etl 中的t 处理的工具,灵活简单,我们需要写的就是select 语句dbt 帮助我们进行处理
测试集成了graphql 以及使用docker 运行
安装
pip install dbt
创建简单demo
- 使用dbt 处理
dbt init demoapp
- 配置profile(pg 数据库)
~/.dbt/profiles.yml default: target: dev outputs: dev: type: postgres host: 127.0.0.1 user: root pass: password port: 5432 dbname: dbt schema: dbt threads: 3 pg: target: dev outputs: dev: type: postgres host: 127.0.0.1 user: root pass: password port: 5433 dbname: dbt schema: dbt
- graphql 集成
docker-compose 文件
version: "3" services: graphql-engine: image: hasura/graphql-engine:v1.0.0-alpha24 ports: - "8080:8080" depends_on: - database environment: - "POSTGRES_PASSWORD:password" command: > /bin/sh -c " graphql-engine --database-url postgres://root:password@database:5432/dbt serve --enable-console; " database: image: postgres environment: POSTGRES_USER: "root" POSTGRES_PASSWORD: "password" POSTGRES_DB: "dbt" ports: - "5432:5432" database2: image: postgres environment: POSTGRES_USER: "root" POSTGRES_PASSWORD: "password" POSTGRES_DB: "dbt" ports: - "5433:5432"
- 修改简单model
models/example/my_first_dbt_model.sql -- Welcome to your first dbt model! -- Did you know that you can also configure models directly within -- the SQL file? This will override configurations stated in dbt_project.yml -- Try changing 'view' to 'table', then re-running dbt {{ config(materialized='view') }} select 1 as id, 'v1' as version,'default' as v,'default' as v2 union all select 2,'v2','default','default' union all select 3,'v3','default','default' union all select 4,'v4','default','default' union all select 5,'v5','default','default' union all select 6,'v6','default','default' union all select 7,'v7','default','default' union all select 8,'v8','default','default' union all select 9,'v9','default','default'
运行
- docker-compose 启动
docker-compose up -d
- 运行dbt
dbt fun
- 效果
- 支持graphql 配置
配置即可
- 数据查询