活动

天翼云最新优惠活动，涵盖免费试用，产品折扣等，助您降本增效！
热门活动
免费活动
  • 活动
  • 智算服务
  • 产品
  • 解决方案
  • 应用商城
  • 合作伙伴
  • 开发者
  • 支持与服务
  • 了解天翼云
searchusermenu
  • 发布文章
  • 消息中心
点赞
收藏
评论
分享
原创

Atlas系列之简介和源代码编译

大数据
2023-05-25 08:25:44
70
0

1，Atlas简介

       Apache Atlas【详见Apache Atlas官网https://atlas.apache.org】 是托管于 Apache 旗下的一款元数据管理和治理的产品，目前在大数据领域应用颇为广泛，可以很好的帮助企业管理数据资产，并对这些资产进行分类和治理，为数据分析，数据治理提供高质量的元数据信息。Atlas 支持从Hadoop、HBase 、Hive、Sqoop、Storm、Kafka 中提取和管理元数据，同时也可以通过 Rest Api 的方式自行定义元数据模型，生成元数据。

      如下图所示，各种Atlas Hook捕获到元数据信息之后封装成元数据事件推送到消息队列Kafka中，Atlas Server端的元数据解析服务从Kafka中监听读取元数据事件，完成解析之后，将元数据模型存入HBase和ElasticSearch等大数据存储组件中，Atlas后端和前端服务提供给终端用户操作。 

2，Atlas源代码编译

（1）git clone项目

git clone https://github.com/apache/atlas.git


（2）新建开发分支

 /d/workspace/idea/atlas (master)$ git tag | grep -i 2.1
release-2.1.0-rc3

-----基于最新的发布tag版本来新建dev_2.1分支
 /d/workspace/idea/atlas (master)
$ git checkout -b dev_2.1 release-2.1.0-rc3
Switched to a new branch 'dev_2.1'

 /d/workspace/idea/atlas (dev_2.1)
$ git branch
* dev_2.1
master

（3）编译，会出现有些jar包无法自动下载

 /d/workspace/idea/atlas (dev_2.1)
$ export MAVEN_OPTS="-Xms2g -Xmx2g"

 /d/workspace/idea/atlas (dev_2.1)
$ mvn clean package -Pdist -DskipTests -Drat.skip=true
[INFO] Scanning for projects...
......
......
[ERROR] Failed to execute goal on project sqoop-bridge-shim: Could not resolve dependencies for project org.apache.atlas:sqoop-bridge-shim:jar:2.1.0: Could not find artifact org.apache.sqoop:sqoop:jar:1.4.6.2.3.99.0-195 in aliyunmaven (https://maven.aliyun.com/repository/public) -> [Help 1]


（4）自己下载jar，然后本地maven安装

mvn install:install-file -DgroupId=org.apache.sqoop -DartifactId=sqoop -Dversion=1.4.6.2.3.99.0-195 -Dpackaging=jar -Dfile=D:\workspace\jar\sqoop-1.4.6.2.3.99.0-195.jar


（5）再次编译成功

 /d/workspace/idea/atlas (dev_2.1)
$ mvn clean package -Pdist -DskipTests -Drat.skip=true

[INFO] Scanning for projects...
......
......
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Atlas Server Build Tools 1.0 ................ SUCCESS [  1.203 s]
[INFO] apache-atlas 2.1.0 ................................. SUCCESS [  7.617 s]
[INFO] Apache Atlas Test Utility Tools 2.1.0 .............. SUCCESS [  7.486 s]
[INFO] Apache Atlas Integration 2.1.0 ..................... SUCCESS [ 11.345 s]
[INFO] Apache Atlas Common 2.1.0 .......................... SUCCESS [  3.757 s]
[INFO] Apache Atlas Client 2.1.0 .......................... SUCCESS [  0.260 s]
[INFO] atlas-client-common 2.1.0 .......................... SUCCESS [  1.732 s]
[INFO] atlas-client-v1 2.1.0 .............................. SUCCESS [  2.659 s]
[INFO] Apache Atlas Server API 2.1.0 ...................... SUCCESS [  3.250 s]
[INFO] Apache Atlas Notification 2.1.0 .................... SUCCESS [  4.613 s]
[INFO] atlas-client-v2 2.1.0 .............................. SUCCESS [  1.618 s]
[INFO] Apache Atlas Graph Database Projects 2.1.0 ......... SUCCESS [  0.282 s]
[INFO] Apache Atlas Graph Database API 2.1.0 .............. SUCCESS [  1.760 s]
[INFO] Graph Database Common Code 2.1.0 ................... SUCCESS [  1.540 s]
[INFO] Apache Atlas JanusGraph-HBase2 Module 2.1.0 ........ SUCCESS [  1.720 s]
[INFO] Apache Atlas JanusGraph DB Impl 2.1.0 .............. SUCCESS [  8.114 s]
[INFO] Apache Atlas Graph DB Dependencies 2.1.0 ........... SUCCESS [  1.519 s]
[INFO] Apache Atlas Authorization 2.1.0 ................... SUCCESS [  2.235 s]
[INFO] Apache Atlas Repository 2.1.0 ...................... SUCCESS [ 16.830 s]
[INFO] Apache Atlas UI 2.1.0 .............................. SUCCESS [01:29 min]
[INFO] Apache Atlas New UI 2.1.0 .......................... SUCCESS [01:30 min]
[INFO] Apache Atlas Web Application 2.1.0 ................. SUCCESS [01:20 min]
[INFO] Apache Atlas Documentation 2.1.0 ................... SUCCESS [  1.952 s]
[INFO] Apache Atlas FileSystem Model 2.1.0 ................ SUCCESS [  2.288 s]
[INFO] Apache Atlas Plugin Classloader 2.1.0 .............. SUCCESS [  1.498 s]
[INFO] Apache Atlas Hive Bridge Shim 2.1.0 ................ SUCCESS [  2.205 s]
[INFO] Apache Atlas Hive Bridge 2.1.0 ..................... SUCCESS [  6.984 s]
[INFO] Apache Atlas Falcon Bridge Shim 2.1.0 .............. SUCCESS [  1.012 s]
[INFO] Apache Atlas Falcon Bridge 2.1.0 ................... SUCCESS [  2.582 s]
[INFO] Apache Atlas Sqoop Bridge Shim 2.1.0 ............... SUCCESS [  0.293 s]
[INFO] Apache Atlas Sqoop Bridge 2.1.0 .................... SUCCESS [  4.927 s]
[INFO] Apache Atlas Storm Bridge Shim 2.1.0 ............... SUCCESS [  0.521 s]
[INFO] Apache Atlas Storm Bridge 2.1.0 .................... SUCCESS [  5.640 s]
[INFO] Apache Atlas Hbase Bridge Shim 2.1.0 ............... SUCCESS [  1.497 s]
[INFO] Apache Atlas Hbase Bridge 2.1.0 .................... SUCCESS [  5.023 s]
[INFO] Apache HBase - Testing Util 2.1.0 .................. SUCCESS [  2.781 s]
[INFO] Apache Atlas Kafka Bridge 2.1.0 .................... SUCCESS [  2.116 s]
[INFO] Apache Atlas classification updater 2.1.0 .......... SUCCESS [  1.027 s]
[INFO] Apache Atlas Impala Hook API 2.1.0 ................. SUCCESS [  0.278 s]
[INFO] Apache Atlas Impala Bridge Shim 2.1.0 .............. SUCCESS [  0.362 s]
[INFO] Apache Atlas Impala Bridge 2.1.0 ................... SUCCESS [  4.231 s]
[INFO] Apache Atlas Distribution 2.1.0 .................... SUCCESS [ 54.008 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:22 min
[INFO] Finished at: 2021-07-29T20:22:25+08:00
[INFO] ------------------------------------------------------------------------

 

（6）编译后的文件

/d/workspace/idea/atlas (dev_2.1)$ ll distro/target/
total 787336
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 META-INF/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:22 apache-atlas-2.1.0-bin/
-rw-r--r-- 1 lenovo 197121 385714601 Jul 29 20:22 apache-atlas-2.1.0-bin.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:22 apache-atlas-2.1.0-classification-updater/
-rw-r--r-- 1 lenovo 197121     28500 Jul 29 20:22 apache-atlas-2.1.0-classification-updater.zip
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-falcon-hook/
-rw-r--r-- 1 lenovo 197121   9270727 Jul 29 20:21 apache-atlas-2.1.0-falcon-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-hbase-hook/
-rw-r--r-- 1 lenovo 197121  11177060 Jul 29 20:21 apache-atlas-2.1.0-hbase-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-hive-hook/
-rw-r--r-- 1 lenovo 197121  11264626 Jul 29 20:21 apache-atlas-2.1.0-hive-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-impala-hook/
-rw-r--r-- 1 lenovo 197121  11227154 Jul 29 20:21 apache-atlas-2.1.0-impala-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-kafka-hook/
-rw-r--r-- 1 lenovo 197121   9281575 Jul 29 20:21 apache-atlas-2.1.0-kafka-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-server/
-rw-r--r-- 1 lenovo 197121 285773897 Jul 29 20:21 apache-atlas-2.1.0-server.tar.gz
-rw-r--r-- 1 lenovo 197121  14088161 Jul 29 20:22 apache-atlas-2.1.0-sources.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-sqoop-hook/
-rw-r--r-- 1 lenovo 197121   9256666 Jul 29 20:21 apache-atlas-2.1.0-sqoop-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-storm-hook/
-rw-r--r-- 1 lenovo 197121  59015081 Jul 29 20:21 apache-atlas-2.1.0-storm-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 archive-tmp/
-rw-r--r-- 1 lenovo 197121     96870 Jul 29 20:21 atlas-distro-2.1.0.jar
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 bin/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 conf/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 maven-archiver/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 maven-shared-archive-resources/
-rw-r--r-- 1 lenovo 197121      3357 Jul 29 20:21 rat.txt
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 test-classes/


我们使用apache-atlas-2.1.0-server.tar.gz来部署Atlas Server服务

版权声明：本文内容系天翼云实名用户自发贡献，版权归原作者所有，天翼云开发者社区不拥有其著作权，亦不承担相应法律责任，未经许可，不得转载。

如有疑问或争议，请联系ctyunbbs@chinatelecom.cn删除。

0条评论
作者已关闭评论
s****m
3文章数
0粉丝数
s****m
3 文章 | 0 粉丝
Ta的热门文章查看更多
Atlas系列之简介和源代码编译Atlas系列之Server服务端部署Atlas Hook部署之HiveMetastore服务
s****m
3文章数
0粉丝数
s****m
3 文章 | 0 粉丝
原创

Atlas系列之简介和源代码编译

大数据
2023-05-25 08:25:44
70
0

1，Atlas简介

       Apache Atlas【详见Apache Atlas官网https://atlas.apache.org】 是托管于 Apache 旗下的一款元数据管理和治理的产品，目前在大数据领域应用颇为广泛，可以很好的帮助企业管理数据资产，并对这些资产进行分类和治理，为数据分析，数据治理提供高质量的元数据信息。Atlas 支持从Hadoop、HBase 、Hive、Sqoop、Storm、Kafka 中提取和管理元数据，同时也可以通过 Rest Api 的方式自行定义元数据模型，生成元数据。

      如下图所示，各种Atlas Hook捕获到元数据信息之后封装成元数据事件推送到消息队列Kafka中，Atlas Server端的元数据解析服务从Kafka中监听读取元数据事件，完成解析之后，将元数据模型存入HBase和ElasticSearch等大数据存储组件中，Atlas后端和前端服务提供给终端用户操作。 

2，Atlas源代码编译

（1）git clone项目

git clone https://github.com/apache/atlas.git


（2）新建开发分支

 /d/workspace/idea/atlas (master)$ git tag | grep -i 2.1
release-2.1.0-rc3

-----基于最新的发布tag版本来新建dev_2.1分支
 /d/workspace/idea/atlas (master)
$ git checkout -b dev_2.1 release-2.1.0-rc3
Switched to a new branch 'dev_2.1'

 /d/workspace/idea/atlas (dev_2.1)
$ git branch
* dev_2.1
master

（3）编译，会出现有些jar包无法自动下载

 /d/workspace/idea/atlas (dev_2.1)
$ export MAVEN_OPTS="-Xms2g -Xmx2g"

 /d/workspace/idea/atlas (dev_2.1)
$ mvn clean package -Pdist -DskipTests -Drat.skip=true
[INFO] Scanning for projects...
......
......
[ERROR] Failed to execute goal on project sqoop-bridge-shim: Could not resolve dependencies for project org.apache.atlas:sqoop-bridge-shim:jar:2.1.0: Could not find artifact org.apache.sqoop:sqoop:jar:1.4.6.2.3.99.0-195 in aliyunmaven (https://maven.aliyun.com/repository/public) -> [Help 1]


（4）自己下载jar，然后本地maven安装

mvn install:install-file -DgroupId=org.apache.sqoop -DartifactId=sqoop -Dversion=1.4.6.2.3.99.0-195 -Dpackaging=jar -Dfile=D:\workspace\jar\sqoop-1.4.6.2.3.99.0-195.jar


（5）再次编译成功

 /d/workspace/idea/atlas (dev_2.1)
$ mvn clean package -Pdist -DskipTests -Drat.skip=true

[INFO] Scanning for projects...
......
......
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Atlas Server Build Tools 1.0 ................ SUCCESS [  1.203 s]
[INFO] apache-atlas 2.1.0 ................................. SUCCESS [  7.617 s]
[INFO] Apache Atlas Test Utility Tools 2.1.0 .............. SUCCESS [  7.486 s]
[INFO] Apache Atlas Integration 2.1.0 ..................... SUCCESS [ 11.345 s]
[INFO] Apache Atlas Common 2.1.0 .......................... SUCCESS [  3.757 s]
[INFO] Apache Atlas Client 2.1.0 .......................... SUCCESS [  0.260 s]
[INFO] atlas-client-common 2.1.0 .......................... SUCCESS [  1.732 s]
[INFO] atlas-client-v1 2.1.0 .............................. SUCCESS [  2.659 s]
[INFO] Apache Atlas Server API 2.1.0 ...................... SUCCESS [  3.250 s]
[INFO] Apache Atlas Notification 2.1.0 .................... SUCCESS [  4.613 s]
[INFO] atlas-client-v2 2.1.0 .............................. SUCCESS [  1.618 s]
[INFO] Apache Atlas Graph Database Projects 2.1.0 ......... SUCCESS [  0.282 s]
[INFO] Apache Atlas Graph Database API 2.1.0 .............. SUCCESS [  1.760 s]
[INFO] Graph Database Common Code 2.1.0 ................... SUCCESS [  1.540 s]
[INFO] Apache Atlas JanusGraph-HBase2 Module 2.1.0 ........ SUCCESS [  1.720 s]
[INFO] Apache Atlas JanusGraph DB Impl 2.1.0 .............. SUCCESS [  8.114 s]
[INFO] Apache Atlas Graph DB Dependencies 2.1.0 ........... SUCCESS [  1.519 s]
[INFO] Apache Atlas Authorization 2.1.0 ................... SUCCESS [  2.235 s]
[INFO] Apache Atlas Repository 2.1.0 ...................... SUCCESS [ 16.830 s]
[INFO] Apache Atlas UI 2.1.0 .............................. SUCCESS [01:29 min]
[INFO] Apache Atlas New UI 2.1.0 .......................... SUCCESS [01:30 min]
[INFO] Apache Atlas Web Application 2.1.0 ................. SUCCESS [01:20 min]
[INFO] Apache Atlas Documentation 2.1.0 ................... SUCCESS [  1.952 s]
[INFO] Apache Atlas FileSystem Model 2.1.0 ................ SUCCESS [  2.288 s]
[INFO] Apache Atlas Plugin Classloader 2.1.0 .............. SUCCESS [  1.498 s]
[INFO] Apache Atlas Hive Bridge Shim 2.1.0 ................ SUCCESS [  2.205 s]
[INFO] Apache Atlas Hive Bridge 2.1.0 ..................... SUCCESS [  6.984 s]
[INFO] Apache Atlas Falcon Bridge Shim 2.1.0 .............. SUCCESS [  1.012 s]
[INFO] Apache Atlas Falcon Bridge 2.1.0 ................... SUCCESS [  2.582 s]
[INFO] Apache Atlas Sqoop Bridge Shim 2.1.0 ............... SUCCESS [  0.293 s]
[INFO] Apache Atlas Sqoop Bridge 2.1.0 .................... SUCCESS [  4.927 s]
[INFO] Apache Atlas Storm Bridge Shim 2.1.0 ............... SUCCESS [  0.521 s]
[INFO] Apache Atlas Storm Bridge 2.1.0 .................... SUCCESS [  5.640 s]
[INFO] Apache Atlas Hbase Bridge Shim 2.1.0 ............... SUCCESS [  1.497 s]
[INFO] Apache Atlas Hbase Bridge 2.1.0 .................... SUCCESS [  5.023 s]
[INFO] Apache HBase - Testing Util 2.1.0 .................. SUCCESS [  2.781 s]
[INFO] Apache Atlas Kafka Bridge 2.1.0 .................... SUCCESS [  2.116 s]
[INFO] Apache Atlas classification updater 2.1.0 .......... SUCCESS [  1.027 s]
[INFO] Apache Atlas Impala Hook API 2.1.0 ................. SUCCESS [  0.278 s]
[INFO] Apache Atlas Impala Bridge Shim 2.1.0 .............. SUCCESS [  0.362 s]
[INFO] Apache Atlas Impala Bridge 2.1.0 ................... SUCCESS [  4.231 s]
[INFO] Apache Atlas Distribution 2.1.0 .................... SUCCESS [ 54.008 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:22 min
[INFO] Finished at: 2021-07-29T20:22:25+08:00
[INFO] ------------------------------------------------------------------------

 

（6）编译后的文件

/d/workspace/idea/atlas (dev_2.1)$ ll distro/target/
total 787336
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 META-INF/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:22 apache-atlas-2.1.0-bin/
-rw-r--r-- 1 lenovo 197121 385714601 Jul 29 20:22 apache-atlas-2.1.0-bin.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:22 apache-atlas-2.1.0-classification-updater/
-rw-r--r-- 1 lenovo 197121     28500 Jul 29 20:22 apache-atlas-2.1.0-classification-updater.zip
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-falcon-hook/
-rw-r--r-- 1 lenovo 197121   9270727 Jul 29 20:21 apache-atlas-2.1.0-falcon-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-hbase-hook/
-rw-r--r-- 1 lenovo 197121  11177060 Jul 29 20:21 apache-atlas-2.1.0-hbase-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-hive-hook/
-rw-r--r-- 1 lenovo 197121  11264626 Jul 29 20:21 apache-atlas-2.1.0-hive-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-impala-hook/
-rw-r--r-- 1 lenovo 197121  11227154 Jul 29 20:21 apache-atlas-2.1.0-impala-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-kafka-hook/
-rw-r--r-- 1 lenovo 197121   9281575 Jul 29 20:21 apache-atlas-2.1.0-kafka-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-server/
-rw-r--r-- 1 lenovo 197121 285773897 Jul 29 20:21 apache-atlas-2.1.0-server.tar.gz
-rw-r--r-- 1 lenovo 197121  14088161 Jul 29 20:22 apache-atlas-2.1.0-sources.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-sqoop-hook/
-rw-r--r-- 1 lenovo 197121   9256666 Jul 29 20:21 apache-atlas-2.1.0-sqoop-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 apache-atlas-2.1.0-storm-hook/
-rw-r--r-- 1 lenovo 197121  59015081 Jul 29 20:21 apache-atlas-2.1.0-storm-hook.tar.gz
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 archive-tmp/
-rw-r--r-- 1 lenovo 197121     96870 Jul 29 20:21 atlas-distro-2.1.0.jar
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 bin/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 conf/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 maven-archiver/
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 maven-shared-archive-resources/
-rw-r--r-- 1 lenovo 197121      3357 Jul 29 20:21 rat.txt
drwxr-xr-x 1 lenovo 197121         0 Jul 29 20:21 test-classes/


我们使用apache-atlas-2.1.0-server.tar.gz来部署Atlas Server服务

版权声明：本文内容系天翼云实名用户自发贡献，版权归原作者所有，天翼云开发者社区不拥有其著作权，亦不承担相应法律责任，未经许可，不得转载。

如有疑问或争议，请联系ctyunbbs@chinatelecom.cn删除。

文章来自个人专栏
文章 | 订阅
0条评论
作者已关闭评论
作者已关闭评论
0
0
售前咨询热线
400-810-9889转1
关注天翼云
  • 权益商城
  • 天翼云APP
  • 天翼云微信公众号
服务与支持
账户管理
快速入口
云网生态
了解天翼云
热门产品
热门推荐
更多推荐
友情链接
©2025 天翼云科技有限公司版权所有 增值电信业务经营许可证A2.B1.B2-20090001
公司地址：北京市东城区青龙胡同甲1号、3号2幢2层205-32室
备案京公网安备11010802043424号京ICP备 2021034386号