Azkaban安装部署

准备⼯作

编译

这⾥选⽤azkaban3.51.0这个版本⾃⼰进⾏重新编译，编译完成之后得到我们需要的安装包进⾏安装

cd /opt/lagou/software/

wget https://github.com/azkaban/azkaban/archive/3.51.0.tar.gz

tar -zxvf 3.51.0.tar.gz -C ../servers/

cd /opt/lagou/servers/azkaban-3.51.0/

yum -y install git

yum -y install gcc-c++

./gradlew build installDist -x test

Gradle是⼀个基于Apache Ant和Apache Maven的项⽬⾃动化构建⼯具。-x test 跳过测试。（注意联⽹下载jar可能会失败、慢）

上传编译后的安装⽂件

上传到linux122节点/opt/lagou/software

solo-server模式部署

安装启动

1	[root@Linux122 ~]# mkdir /opt/lagou/servers/azkaban

1. 解压
azkaban 的solo server使⽤的是⼀个单节点的模式来进⾏启动服务的，只需要⼀个azkaban-soloserver-0.1.0-SNAPSHOT.tar.gz的安装包即可启动，所有的数据信息都是保存在H2这个azkaban默认的数据当中
1
[root@Linux122 ~]# tar -zxvf /opt/lagou/software/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz -C /opt/lagou/servers/azkaban

修改配置⽂件

修改时区配置⽂件

[root@Linux122 ~]# cd /opt/lagou/servers/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT/conf

[root@Linux122 conf]# vim azkaban.properties

default.timezone.id=Asia/Shanghai

修改commonprivate.properties配置⽂件

[root@Linux122 conf]# cd /opt/lagou/servers/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT/plugins/jobtypes

[root@Linux122 jobtypes]# vim commonprivate.properties

execute.as.user=false
memCheck.enabled=false

azkaban默认需要3G的内存，剩余内存不⾜则会报异常。

启动solo-server

1
2
3

[root@Linux122 jobtypes]# cd /opt/lagou/servers/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT

bin/start-solo.sh

1. 浏览器⻚⾯访问
浏览器⻚⾯访问：http://Linux122:8081/index

⽤户名：azkaban

密码： azkaban

停止solo-server

1
2
3

[root@Linux122 jobtypes]# cd /opt/lagou/servers/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT

bin/shutdown-solo.sh

使⽤

需求

使⽤azkaban调度我们的shell脚本，执⾏linux的shell命令
具体步骤

创建普通⽂本⽂件 foo.job，⽂件内容如下
1
2
type=command
command=echo 'hello world'
打成压缩包

上传压缩包到Azkaban,创建project

指定project名称和描述信息

进入project,上传我们的压缩包

查看⼯作流计划并执⾏

multiple-executor模式部署

安装所需软件

Azkaban Web服务安装包

azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
Azkaban执⾏服务安装包

azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
sql脚本

azkaban-db-0.1.0-SNAPSHOT.tar.gz
节点规划

HOST ⻆⾊

Linux123 mysql,exec-server

Linux122 web-server

Linux121 exec-server

HOST	⻆⾊
Linux123	mysql,exec-server
Linux122	web-server
Linux121	exec-server

数据库准备

进入Linux123节点,创建文件夹

1	[root@Linux123 ~]# mkdir /opt/lagou/servers/azkaban

上传并解压 azkaban-db-0.1.0-SNAPSHOT.tar.gz

1
2
3

[root@Linux123 ~]#cd /opt/lagou/software

[root@Linux123 software]# tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz -C /opt/lagou/servers/azkaban

进⼊mysql的客户端执⾏以下命令

mysql -uroot -p

mysql> SET GLOBAL validate_password_length=5;

mysql> SET GLOBAL validate_password_policy=0;

mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';

mysql> GRANT all privileges ON azkaban.* to 'azkaban'@'%' identified by 'azkaban' WITH GRANT OPTION;

mysql> CREATE DATABASE azkaban;

mysql> use azkaban;

mysql> source /opt/lagou/servers/azkaban/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql;

mysql> show tables;
+--------------------------+
| Tables_in_azkaban        |
+--------------------------+
| QRTZ_BLOB_TRIGGERS       |
| QRTZ_CALENDARS           |
| QRTZ_CRON_TRIGGERS       |
| QRTZ_FIRED_TRIGGERS      |
| QRTZ_JOB_DETAILS         |
| QRTZ_LOCKS               |
| QRTZ_PAUSED_TRIGGER_GRPS |
| QRTZ_SCHEDULER_STATE     |
| QRTZ_SIMPLE_TRIGGERS     |
| QRTZ_SIMPROP_TRIGGERS    |
| QRTZ_TRIGGERS            |
| active_executing_flows   |
| active_sla               |
| execution_dependencies   |
| execution_flows          |
| execution_jobs           |
| execution_logs           |
| executor_events          |
| executors                |
| project_events           |
| project_files            |
| project_flow_files       |
| project_flows            |
| project_permissions      |
| project_properties       |
| project_versions         |
| projects                 |
| properties               |
| triggers                 |
+--------------------------+
29 rows in set (0.00 sec)

配置Azkaban-web-server

进⼊linux122节点,解压azkaban-web-server

1
2
3

[root@Linux122 ~]# cd /opt/lagou/software

[root@Linux122 software]# tar -zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz –C /opt/lagou/servers/azkaban/

进⼊azkaban-web-server根⽬录下

[root@Linux122 software]# cd /opt/lagou/servers/azkaban/azkaban-web-server-0.1.0-SNAPSHOT

#⽣成ssl证书：
[root@Linux122 azkaban-web-server-0.1.0-SNAPSHOT]# keytool -keystore keystore -alias jetty -genkey -keyalg RSA

注意：运⾏此命令后,会提示输⼊当前⽣成keystore的密码及相应信息,输⼊的密码请记住(所有密码统⼀以azkaban输⼊)

修改 azkaban-web-server的配置⽂件

[root@Linux122 ~]# cd /opt/lagou/servers/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/conf

[root@Linux122 conf]# vim azkaban.properties

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai # 时区注意后⾯不要有空格

# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml

# Azkaban Jetty server properties. 开启使⽤ssl 并且知道端⼝
jetty.use.ssl=true
jetty.port=8443
jetty.maxThreads=25

# KeyStore for SSL ssl相关配置 注意密码和证书路径
jetty.keystore=keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=keystore
jetty.trustpassword=azkaban

# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=Linux123
mysql.database=azkaban
mysql.user=root
mysql.password=12345678
mysql.numconnections=100

#Multiple Executor 设置为false
azkaban.use.multiple.executors=true

#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

添加属性

[root@Linux122 ~]# cd /opt/lagou/servers/azkaban/azkaban-web-server-0.1.0-SNAPSHOT

[root@Linux122 azkaban-web-server-0.1.0-SNAPSHOT]# mkdir -p plugins/jobtypes

[root@Linux122 azkaban-web-server-0.1.0-SNAPSHOT]# cd plugins/jobtypes/

[root@Linux122 jobtypes]# vim commonprivate.properties
azkaban.native.lib=false
execute.as.user=false
memCheck.enabled=false

配置Azkaban-exec-server

在Linux123节点，上传azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz安装包到/opt/lagou/software

解压azkaban-exec-server

1
2
3

[root@Linux123 ~]# cd /opt/lagou/software

[root@Linux123 software]# tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C /opt/lagou/servers/azkaban/

修改azkaban-exec-server的配置⽂件

[root@Linux123 software]# cd /opt/lagou/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/conf/

[root@Linux123 conf]# vim azkaban.properties

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Where the Azkaban web server is located
azkaban.webserver.url=https://Linux122:8443
# Azkaban mysql settings by default. Users should configure their own username
and password.
database.type=mysql
mysql.port=3306
mysql.host=Linux123
mysql.database=azkaban
mysql.user=root
mysql.password=12345678
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

分发exec-server到Linux121节点

1 2	# Linux121创建安装目录 [root@Linux121 ~]# mkdir /opt/lagou/servers/azkaban

1
2
3

[root@Linux123 conf]# cd /opt/lagou/servers/azkaban

[root@Linux123 azkaban]# scp -r azkaban-exec-server-0.1.0-SNAPSHOT Linux121:$PWD

启动服务

先启动exec-server

#启动exec-server
[root@Linux123 azkaban]# cd /opt/lagou/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT
[root@Linux123 azkaban-exec-server-0.1.0-SNAPSHOT]# bin/start-exec.sh

[root@Linux121 ~]# cd /opt/lagou/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT
[root@Linux121 azkaban-exec-server-0.1.0-SNAPSHOT]# bin/start-exec.sh

查看启动日志

[root@Linux123 azkaban-exec-server-0.1.0-SNAPSHOT]# tail -1000f executorServerLog__2022-07-07+10\:22\:46.out

2022/07/07 10:19:17.506 +0100 ERROR [MySQLDataSource] [Azkaban] Failed to find write-enabled DB connection. Wait 15 seconds and retry.No.Attempt = 4
java.sql.SQLException: Cannot create PoolableConnectionFactory (Access denied for user 'root'@'Linux123' (using password: YES))
        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2294)
        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2039)
        at azkaban.db.MySQLDataSource.getConnection(MySQLDataSource.java:76)
        at org.apache.commons.dbutils.AbstractQueryRunner.prepareConnection(AbstractQueryRunner.java:175)
        at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:286)
        at azkaban.db.DatabaseOperator.query(DatabaseOperator.java:68)
        at azkaban.executor.ExecutorDao.fetchExecutor(ExecutorDao.java:63)
        at azkaban.executor.JdbcExecutorLoader.fetchExecutor(JdbcExecutorLoader.java:272)
        at azkaban.execapp.AzkabanExecutorServer.insertExecutorEntryIntoDB(AzkabanExecutorServer.java:264)
        at azkaban.execapp.AzkabanExecutorServer.start(AzkabanExecutorServer.java:242)
        at azkaban.execapp.AzkabanExecutorServer.launch(AzkabanExecutorServer.java:152)
        at azkaban.execapp.AzkabanExecutorServer.main(AzkabanExecutorServer.java:148)
Caused by: java.sql.SQLException: Access denied for user 'root'@'Linux123' (using password: YES)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1086)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:928)
        at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1750)
        at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1290)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2493)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
        at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
        at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
        at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2304)
        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2290)
        ... 11 more

此时是mysql的root用户的host是localhost

# 进入 mysql
mysql -uroot -p

# 使用mysql这个数据库
use mysql;

# 查询mysql的user表信息
select User,Host from user;
+---------------+-----------+
| User          | Host      |
+---------------+-----------+
| azkaban       | %         |
| hive          | %         |
| mysql.session | localhost |
| mysql.sys     | localhost |
| root          | localhost |
+---------------+-----------+

# 修改user表
update user set Host='%'' where User='root';

# 刷新
flush privileges;

# 退出即可
quit;

再次启动exec-server,查看启动日志如下:

2022/07/07 10:22:47.359 +0100 INFO [JobCallbackRequestMaker] [Azkaban] Initialization for azkaban.execapp.event.JobCallbackRequestMaker is completed
2022/07/07 10:22:47.360 +0100 INFO [JobCallbackManager] [Azkaban] Initialization completed azkaban.execapp.event.JobCallbackManager
2022/07/07 10:22:47.360 +0100 INFO [JobCallbackManager] [Azkaban] azkabanHostName Linux122:8443
2022/07/07 10:22:47.360 +0100 INFO [AzkabanExecutorServer] [Azkaban] Registering MBeans...
2022/07/07 10:22:47.367 +0100 INFO [AzkabanExecutorServer] [Azkaban] Bean azkaban.jmx.JmxJettyServer registered.
2022/07/07 10:22:47.368 +0100 INFO [AzkabanExecutorServer] [Azkaban] Bean azkaban.execapp.jmx.JmxFlowRunnerManager registered.
2022/07/07 10:22:47.370 +0100 INFO [AzkabanExecutorServer] [Azkaban] Bean azkaban.execapp.jmx.JmxJobMBeanManager registered.
2022/07/07 10:22:47.370 +0100 INFO [AzkabanExecutorServer] [Azkaban] Bean azkaban.execapp.jmx.JmxJobCallback registered.
2022/07/07 10:22:47.370 +0100 INFO [AzkabanExecutorServer] [Azkaban] No value for property: jmx.attribute.processor.class was found
2022/07/07 10:22:47.370 +0100 INFO [log] [Azkaban] jetty-6.1.26
2022/07/07 10:22:47.391 +0100 INFO [log] [Azkaban] Started SocketConnector@0.0.0.0:12321
2022/07/07 10:22:47.620 +0100 INFO [AzkabanExecutorServer] [Azkaban] Started Executor Server on Linux123:12321
2022/07/07 10:22:47.631 +0100 INFO [AzkabanExecutorServer] [Azkaban] Setting timezone to Asia/Shanghai
2022/07/07 10:23:47.120 +0100 INFO [FlowRunnerManager] [Azkaban] # of executing flows: 0
2022/07/07 10:24:47.124 +0100 INFO [FlowRunnerManager] [Azkaban] # of executing flows: 0
2022/07/07 10:24:47.125 +0100 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished
2022/07/07 10:25:47.126 +0100 INFO [FlowRunnerManager] [Azkaban] # of executing flows: 0
2022/07/07 10:26:47.128 +0100 INFO [FlowRunnerManager] [Azkaban] # of executing flows: 0
2022/07/07 10:26:47.128 +0100 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished

启动web-server

1
2
3

#启动web-server
[root@Linux122 jobtypes]# cd /opt/lagou/servers/azkaban/azkaban-web-server-0.1.0-SNAPSHOT
[root@Linux122 azkaban-web-server-0.1.0-SNAPSHOT]# bin/start-web.sh

启动webServer之后通过安装包根⽬录下查看对应启动⽇志

Caused by: azkaban.executor.ExecutorManagerException: No active executor found
      at azkaban.executor.ExecutorManager.setupExecutors(ExecutorManager.java:253)
      at azkaban.executor.ExecutorManager.<init>(ExecutorManager.java:131)
      at azkaban.executor.ExecutorManager$$FastClassByGuice$$e1c1dfed.newInstance(<generated>)
      at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
      at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
      at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)

此时需要⼿动激活executor

[root@Linux121 azkaban-exec-server-0.1.0-SNAPSHOT]# cd /opt/lagou/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT
[root@Linux121 azkaban-exec-server-0.1.0-SNAPSHOT]# curl -G "Linux121:$(<./executor.port)/executor?action=activate" && echo
{"status":"success"}

[root@Linux123 azkaban-exec-server-0.1.0-SNAPSHOT]# cd /opt/lagou/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT
[root@Linux123 azkaban-exec-server-0.1.0-SNAPSHOT]# curl -G "Linux123:$(<./executor.port)/executor?action=activate" && echo
{"status":"success"}

访问地址：https://linux122:8443