Компиляция и установка Apache Ranger

Большие данные

Заметки для разработчиков с открытым исходным кодом: DevOps, микросервисы, распределенные, большие данные, высокая доступность, блокчейн, официальные документы, алгоритмы, взлом, шаблоны проектирования, вопросы для интервью. попроси звезду⭐️

Apache Ranger — это инфраструктура централизованного управления безопасностью в области больших данных, целью которой является обеспечение централизованного управления безопасностью компонентов Hadoop путем формулирования политик. Пользователи могут безопасно получать доступ к данным в кластере через Ranger.

окрестности

Рамка Версия
Ubuntu 16.04
JAVA OpenJDK-8-jdk
Python 2.7
Maven 3.6.3
Ranger 2.0.0
Hadoop 3.1.3
Solr 8.5.2

компиляция исходного кода

Подготовка окружающей среды

  • Скачать исходный код Рейнджера
wget https://downloads.apache.org/ranger/2.0.0/apache-ranger-2.0.0.tar.gz
或使用镜像
wget https://mirrors.tuna.tsinghua.edu.cn/apache/ranger/2.0.0/apache-ranger-2.0.0.tar.gz
  • Скачать Maven
wget https://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
  • Установите среду компиляции java, python и c++
apt install openjdk-8-jdk
apt install python2.7
apt install gcc g++
ln -s /usr/bin/python2.7 /usr/bin/python
  • Настроить переменные среды
export MAVEN_HOME=/opt/app/apache-maven-3.6.3
export PATH=$PATH:$MAVEN_HOME/bin
export MAVEN_OPTS=-Xmx2048m
  • используйте источник ali maven

/opt/app/apache-maven-3.6.2/conf/settings.xml

<mirrors>
   <mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>central</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
   </mirror>

   <mirror>
     <id>CN</id>
     <name>OSChina Central</name>
     <url>http://maven.oschina.net/content/groups/public/</url>
     <mirrorOf>central</mirrorOf>
   </mirror>

   <mirror>
     <id>alimaven</id>
     <mirrorOf>central</mirrorOf>
     <name>aliyun maven</name>
     <url>https://maven.aliyun.com/nexus/content/repositories/central/</url>
   </mirror>

   <mirror>
     <id>jboss-public-repository-group</id>
     <mirrorOf>central</mirrorOf>
     <name>JBoss Public Repository Group</name>
     <url>https://repository.jboss.org/nexus/content/groups/public</url>
   </mirror>
</mirrors>

компилировать

tar xf apache-ranger-2.0.0.tar.gz
cd apache-ranger-2.0.0
mvn clean compile package assembly:assembly install -DskipTests -Drat.skip=true

编译成功打开 target 文件夹
apache-ranger-2.0.0/target# ll
total 1605220
drwxr-xr-x  5 root root      4096 Jun 29 11:04 ./
drwxr-xr-x 59 root root      4096 Jun 29 03:20 ../
drwxr-xr-x  2 root root      4096 Jun 29 03:20 antrun/
drwxr-xr-x  2 root root      4096 Jun 29 11:04 archive-tmp/
drwxr-xr-x  3 root root      4096 Jun 29 03:20 maven-shared-archive-resources/
-rw-r--r--  1 root root        30 Jun 29 11:04 .plxarc
-rw-r--r--  1 root root 248635508 Jun 29 11:01 ranger-2.0.0-admin.tar.gz
-rw-r--r--  1 root root 249666712 Jun 29 11:02 ranger-2.0.0-admin.zip
-rw-r--r--  1 root root  27787895 Jun 29 11:03 ranger-2.0.0-atlas-plugin.tar.gz
-rw-r--r--  1 root root  27832012 Jun 29 11:03 ranger-2.0.0-atlas-plugin.zip
-rw-r--r--  1 root root  31555304 Jun 29 11:04 ranger-2.0.0-elasticsearch-plugin.tar.gz
-rw-r--r--  1 root root  31605841 Jun 29 11:04 ranger-2.0.0-elasticsearch-plugin.zip
-rw-r--r--  1 root root  26638377 Jun 29 10:58 ranger-2.0.0-hbase-plugin.tar.gz
-rw-r--r--  1 root root  26665982 Jun 29 10:58 ranger-2.0.0-hbase-plugin.zip
-rw-r--r--  1 root root  23971392 Jun 29 10:58 ranger-2.0.0-hdfs-plugin.tar.gz
-rw-r--r--  1 root root  23997623 Jun 29 10:58 ranger-2.0.0-hdfs-plugin.zip
-rw-r--r--  1 root root  23825995 Jun 29 10:58 ranger-2.0.0-hive-plugin.tar.gz
-rw-r--r--  1 root root  23854522 Jun 29 10:58 ranger-2.0.0-hive-plugin.zip
-rw-r--r--  1 root root  39930681 Jun 29 10:59 ranger-2.0.0-kafka-plugin.tar.gz
-rw-r--r--  1 root root  39983878 Jun 29 10:59 ranger-2.0.0-kafka-plugin.zip
-rw-r--r--  1 root root  90982883 Jun 29 11:02 ranger-2.0.0-kms.tar.gz
-rw-r--r--  1 root root  91106270 Jun 29 11:02 ranger-2.0.0-kms.zip
-rw-r--r--  1 root root  28380704 Jun 29 10:58 ranger-2.0.0-knox-plugin.tar.gz
-rw-r--r--  1 root root  28411022 Jun 29 10:58 ranger-2.0.0-knox-plugin.zip
-rw-r--r--  1 root root  23940874 Jun 29 11:03 ranger-2.0.0-kylin-plugin.tar.gz
-rw-r--r--  1 root root  23980053 Jun 29 11:03 ranger-2.0.0-kylin-plugin.zip
-rw-r--r--  1 root root     34223 Jun 29 11:02 ranger-2.0.0-migration-util.tar.gz
-rw-r--r--  1 root root     37740 Jun 29 11:02 ranger-2.0.0-migration-util.zip
-rw-r--r--  1 root root  26388071 Jun 29 11:00 ranger-2.0.0-ozone-plugin.tar.gz
-rw-r--r--  1 root root  26421136 Jun 29 11:00 ranger-2.0.0-ozone-plugin.zip
-rw-r--r--  1 root root  40302042 Jun 29 11:04 ranger-2.0.0-presto-plugin.tar.gz
-rw-r--r--  1 root root  40341626 Jun 29 11:04 ranger-2.0.0-presto-plugin.zip
-rw-r--r--  1 root root  22232050 Jun 29 11:02 ranger-2.0.0-ranger-tools.tar.gz
-rw-r--r--  1 root root  22248747 Jun 29 11:02 ranger-2.0.0-ranger-tools.zip
-rw-r--r--  1 root root     42667 Jun 29 11:02 ranger-2.0.0-solr_audit_conf.tar.gz
-rw-r--r--  1 root root     45636 Jun 29 11:02 ranger-2.0.0-solr_audit_conf.zip
-rw-r--r--  1 root root  26964416 Jun 29 11:00 ranger-2.0.0-solr-plugin.tar.gz
-rw-r--r--  1 root root  27010058 Jun 29 11:00 ranger-2.0.0-solr-plugin.zip
-rw-r--r--  1 root root  23952732 Jun 29 11:03 ranger-2.0.0-sqoop-plugin.tar.gz
-rw-r--r--  1 root root  23986050 Jun 29 11:03 ranger-2.0.0-sqoop-plugin.zip
-rw-r--r--  1 root root   4081388 Jun 29 11:03 ranger-2.0.0-src.tar.gz
-rw-r--r--  1 root root   6257752 Jun 29 11:03 ranger-2.0.0-src.zip
-rw-r--r--  1 root root  37230628 Jun 29 10:59 ranger-2.0.0-storm-plugin.tar.gz
-rw-r--r--  1 root root  37268719 Jun 29 10:59 ranger-2.0.0-storm-plugin.zip
-rw-r--r--  1 root root  32772036 Jun 29 11:02 ranger-2.0.0-tagsync.tar.gz
-rw-r--r--  1 root root  32782070 Jun 29 11:02 ranger-2.0.0-tagsync.zip
-rw-r--r--  1 root root  16256778 Jun 29 11:02 ranger-2.0.0-usersync.tar.gz
-rw-r--r--  1 root root  16280886 Jun 29 11:02 ranger-2.0.0-usersync.zip
-rw-r--r--  1 root root  23953849 Jun 29 10:59 ranger-2.0.0-yarn-plugin.tar.gz
-rw-r--r--  1 root root  23992362 Jun 29 10:59 ranger-2.0.0-yarn-plugin.zip
-rw-r--r--  1 root root         5 Jun 29 11:04 version

Произошла ошибка при компиляции

  • Среда компиляции C++ не установлена
[INFO] Unix Native Authenticator .......................... FAILURE [  1.387 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:51 min
[INFO] Finished at: 2020-06-030T12:00:43-02:00
[INFO] Final Memory: 96M/420M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:compile (default-compile) on project credValidator: Error executing command line. Exit code:127 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:compile (default-compile) on project credValidator: Error executing command line. Exit code:127
  • питон не установлен
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-version-annotation) on project ranger-util: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" executable="python">... @ 4:48 in /data/apache-ranger-2.0.0/ranger-util/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-version-annotation) on project ranger-util: An Ant BuildException has occured: exec returned: 1
around Ant part ...<exec failonerror="true" executable="python">... @ 4:48 in /data/apache-ranger-2.0.0/ranger-util/target/antrun/build-main.xml

Установить рейнджер-админ

распаковать

tar zxvf ranger-2.0.0-admin.tar.gz -C /opt/ranger
cd /opt/ranger

Исправлятьranger-2.0.0-admin/install.properties

# 数据库连接
SQL_CONNECTOR_JAR=/opt/ranger/mysql-connector-java-8.0.18.jar
db_root_user=root
db_root_password=root
db_host=localhost

db_name=ranger
db_user=ranger
db_password=rangeradmin

# 日志审计
audit_store=solr
audit_solr_urls=http://localhost:6083/solr/ranger_audits
audit_solr_user=solr

воплощать в жизньranger-2.0.0-admin/setup.shустановить

Вывод в случае успеха.install.propertiesтребуется в конфигурацииsolr,mysql,mysql驱动包Если нет, обратитесь к следующему

2020-06-30 07:55:44,211  [I] Checking connection passed.
Installation of Ranger PolicyManager Web Application is completed.

воплощать в жизньranger-2.0.0-admin/set_globals.shНастроить программную ссылку

воплощать в жизньranger-admin startилиranger-2.0.0-admin/ews/ranger-admin-services.sh startзапустить службу

доступ через браузерhttp://ip:6080/ admin/admin

установить Solr

Исправлятьranger-2.0.0-admin/contrib/solr_for_audit_setup/install.properties

SOLR_INSTALL=true

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
SOLR_DOWNLOAD_URL=https://mirrors.tuna.tsinghua.edu.cn/apache/lucene/solr/8.5.2/solr-8.5.2.tgz

SOLR_INSTALL_FOLDER=/data/solr   #安装目录，这个随意
SOLR_RANGER_HOME=/data/solr/ranger_audit_server
SOLR_RANGER_PORT=6083   #默认端口
SOLR_DEPLOYMENT=standalone  #部署模式(单节点部署)，还有solrcloud(集群)模式
SOLR_RANGER_DATA_FOLDER=/data/solr/ranger_audit_server/data   #数据存放目录

воплощать в жизньranger-2.0.0-admin/contrib/solr_for_audit_setup/setup.sh

Tue Jun 30 06:58:31 UTC 2020|INFO|Installed Solr in /opt/solr
Tue Jun 30 06:58:31 UTC 2020|INFO|Configuring standalone instance
Tue Jun 30 06:58:31 UTC 2020|INFO|Copying Ranger Audit Server configuration to /opt/solr/ranger_audit_server
Tue Jun 30 06:58:31 UTC 2020|INFO|Creating group solr
Tue Jun 30 06:58:31 UTC 2020|INFO|Creating user solr
Tue Jun 30 06:58:31 UTC 2020|INFO|Done configuring Solr for Apache Ranger Audit
Tue Jun 30 06:58:31 UTC 2020|INFO|Solr HOME for Ranger Audit is /opt/solr/ranger_audit_server
Tue Jun 30 06:58:31 UTC 2020|INFO|Data folder for Audit logs is /opt/solr/ranger_audit_server/data
Tue Jun 30 06:58:31 UTC 2020|INFO|To start Solr run /opt/solr/ranger_audit_server/scripts/start_solr.sh
Tue Jun 30 06:58:31 UTC 2020|INFO|To stop Solr run /opt/solr/ranger_audit_server/scripts/stop_solr.sh
Tue Jun 30 06:58:31 UTC 2020|INFO|After starting Solr for RangerAudit, it will listen at 6083. E.g http://weihai-2:6083
Tue Jun 30 06:58:31 UTC 2020|INFO|Configure Ranger to use the following URL http://weihai-2:6083/solr/ranger_audits
Tue Jun 30 06:58:31 UTC 2020|INFO| ** NOTE: If Solr is Secured then solrclient JAAS configuration has to be added to Ranger Admin and Ranger Plugin properties
Tue Jun 30 06:58:31 UTC 2020|INFO| ** Refer documentation on how to configure Ranger for audit to Secure Solr
########## Done ###################
Created file /opt/solr/ranger_audit_server/install_notes.txt with instructions to start and stop
###################################

старт/стоп решение

/opt/solr/ranger_audit_server/scripts/start_solr.sh
/opt/solr/ranger_audit_server/scripts/stop_solr.sh

Установить MySQL

  • Загрузите пакет драйверов mysqlhttps://downloads.mysql.com/archives/c-j/

  • скачать mysql-сервер

    • Загрузка с официального сайта
    https://dev.mysql.com/downloads/mysql/

    или

    • Обновить источник APT
    wget https://dev.mysql.com/get/mysql-apt-config_0.8.15-1_all.de
pkg -i mysql-apt-config_0.8.15-1_all.deb

apt search mysql-server
Sorting... Done
Full Text Search... Done
auth2db/xenial 0.2.5-2+dfsg-5ubuntu1 all
  Powerful and eye-candy IDS logger, log viewer and alert generator

mysql-server/unknown,now 8.0.20-1ubuntu16.04 amd64 [installed]
  MySQL Server meta package depending on latest version
  
  
apt install mysql-server

  • настроить mysql

create user 'ranger'@'%' identified by 'rangeradmin';
create database ranger;
grant all privileges on ranger.* to 'ranger'@'%';
set global log_bin_trust_function_creators=TRUE;
SET GLOBAL innodb_lock_wait_timeout=500;
Категории