技术标签: hadoop
加载环境变量
由于编译impala的时候下载了很多相关依赖配置,但是有些可能需要使用我们自己的环境依赖需要进行部分修改
source /app/impala/bin/impala-config.sh
修改自定义环境变量
由于impala依赖的项目过多,所以环境变量配置的不对,完全起不来,所以这么给出需要配置的环境变量,有自己搭建的自行修改,这里面最麻烦的就是CLASSPATH,我后面给个脚本输出一下这个CLASSPATH,这步很重要,因为假如没有这步,你会一直报java core dump
source /app/impala/bin/impala-config.sh
获取所有依赖
for f in /app/impala/fe/target/dependency/*.jar; do
export CLASSPATH=$CLASSPATH:$f
done
echo $CLASSPATH
修改环境变量文件,将以下内容拷贝到一个sh文件,然后执行再启动impala
export IMPALA_HOME=/app/impala
export HADOOP_HOME=/app/hadoop-3.1.2
export HADOOP_CONF_DIR=/etc/impala/conf.dist
export HADOOP_INCLUDE_DIR=/app/hadoop-3.1.2/include
export HADOOP_LIB_DIR=/app/hadoop-3.1.2/lib
export MINI_DFS_BASE_DATA_DIR=/app/impala/hdfs-data
export HIVE_HOME=/app/hive
export HIVE_CONF_DIR=/etc/impala/conf.dist/
export HIVE_SRC_DIR=/app/impala/toolchain/cdh_components-1814051/hive-2.1.1-cdh6.x-SNAPSHOT/src
export HBASE_HOME=/app/impala/toolchain/cdh_components-1814051/hbase-2.1.0-cdh6.x-SNAPSHOT/
export HBASE_CONF_DIR=/app/impala/fe/src/test/resources
export SENTRY_HOME=/app/impala/toolchain/cdh_components-1814051/sentry-2.1.0-cdh6.x-SNAPSHOT
export SENTRY_CONF_DIR=/app/impala/fe/src/test/resources
export RANGER_HOME=/app/impala/toolchain/cdp_components-1617729/ranger-2.0.0.7.0.2.0-212-admin
export RANGER_CONF_DIR=/app/impala/fe/src/test/resources
export THRIFT_HOME=/app/impala/toolchain/thrift-0.9.3-p7
export HADOOP_LZO=/app/impala/../hadoop-lzo
export IMPALA_LZO=/app/impala/../Impala-lzo
export LIBHDFS_OPTS="-Djava.library.path=/app/impala/toolchain/cdh_components-1814051/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native/ -XX:ErrorFile=/app/impala/logs/hs_err_pid%p.log -XX:MaxPermSize=128m -Djava.library.path=/app/impala/toolchain/cdh_components-1814051/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native/ -XX:ErrorFile=/app/impala/logs/hs_err_pid%p.log -XX:MaxPermSize=128m"
export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64
export LD_LIBRARY_PATH=:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server:/app/impala/toolchain/cdh_components-1814051/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native:/app/impala/toolchain/snappy-1.1.4/lib:/app/impala/../Impala-lzo/build:/app/impala/toolchain/gcc-4.9.2/lib64:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server:/app/impala/toolchain/cdh_components-1814051/hadoop-3.0.0-cdh6.x-SNAPSHOT//lib/native:/app/impala/toolchain/snappy-1.1.4/lib:/app/impala/../Impala-lzo/build:/app/impala/toolchain/gcc-4.9.2/lib64
export LD_PRELOAD=:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjsig.so:/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjsig.so
export MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
export IMPALA_CATALOG_SERVICE_HOST=sit1
export IMPALA_STATE_STORE_HOST=sit1
export IMPALA_STATE_STORE_PORT=24000
export IMPALA_BACKEND_PORT=22000
export IMPALA_LOG_DIR=/app/impala/logs/impala
export IMPALA_CATALOG_ARGS=" --log_dir=${IMPALA_LOG_DIR} "
export IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}"
export IMPALA_SERVER_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-be_port=${IMPALA_BACKEND_PORT}"
export ENABLE_CORE_DUMPS=false
export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64
export CLASSPATH=./:/app/impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar:/app/hive//lib:/usr/java/jdk1.8.0_201-amd64/lib:/usr/java/jdk1.8.0_201-amd64/jre/lib:/app/spark//lib:/app/hadoop-3.1.2/share/hadoop/common:/app/hadoop-3.1.2/share/hadoop/yarn:/app/spark//lib:/app/impala/fe/target/dependency/accessors-smart-1.2.jar:/app/impala/fe/target/dependency/activation-1.1.jar:/app/impala/fe/target/dependency/aggdesigner-algorithm-6.0.jar:/app/impala/fe/target/dependency/aircompressor-0.15.jar:/app/impala/fe/target/dependency/annotations-17.0.0.jar:/app/impala/fe/target/dependency/antlr-2.7.7.jar:/app/impala/fe/target/dependency/antlr-runtime-3.3.jar:/app/impala/fe/target/dependency/aopalliance-1.0.jar:/app/impala/fe/target/dependency/aopalliance-repackaged-2.5.0-b32.jar:/app/impala/fe/target/dependency/apacheds-jdbm1-2.0.0-M2.jar:/app/impala/fe/target/dependency/asm-5.0.4.jar:/app/impala/fe/target/dependency/async-1.4.1.jar:/app/impala/fe/target/dependency/audience-annotations-0.5.0.jar:/app/impala/fe/target/dependency/avro-1.8.2-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/aws-java-sdk-bundle-1.11.375.jar:/app/impala/fe/target/dependency/azure-data-lake-store-sdk-2.2.9.jar:/app/impala/fe/target/dependency/azure-keyvault-core-0.8.0.jar:/app/impala/fe/target/dependency/azure-storage-5.4.0.jar:/app/impala/fe/target/dependency/bcpkix-jdk15on-1.60.jar:/app/impala/fe/target/dependency/bcprov-jdk15on-1.59.jar:/app/impala/fe/target/dependency/bonecp-0.8.0.RELEASE.jar:/app/impala/fe/target/dependency/calcite-core-1.12.0.jar:/app/impala/fe/target/dependency/calcite-druid-1.12.0.jar:/app/impala/fe/target/dependency/calcite-linq4j-1.12.0.jar:/app/impala/fe/target/dependency/commonj.sdo-2.1.1.jar:/app/impala/fe/target/dependency/commons-beanutils-1.9.4.jar:/app/impala/fe/target/dependency/commons-cli-1.2.jar:/app/impala/fe/target/dependency/commons-codec-1.11.jar:/app/impala/fe/target/dependency/commons-collections-3.2.2.jar:/app/impala/fe/target/dependency/commons-compiler-2.7.6.jar:/app/impala/fe/target/dependency/commons-compress-1.19.jar:/app/impala/fe/target/dependency/commons-configuration2-2.1.1.jar:/app/impala/fe/target/dependency/commons-crypto-1.0.0.jar:/app/impala/fe/target/dependency/commons-daemon-1.0.13.jar:/app/impala/fe/target/dependency/commons-dbcp-1.4.jar:/app/impala/fe/target/dependency/commons-el-1.0.jar:/app/impala/fe/target/dependency/commons-io-2.6.jar:/app/impala/fe/target/dependency/commons-lang-2.6.jar:/app/impala/fe/target/dependency/commons-lang3-3.7.jar:/app/impala/fe/target/dependency/commons-logging-1.1.3.jar:/app/impala/fe/target/dependency/commons-math-2.2.jar:/app/impala/fe/target/dependency/commons-math3-3.1.1.jar:/app/impala/fe/target/dependency/commons-net-3.1.jar:/app/impala/fe/target/dependency/commons-pool-1.5.4.jar:/app/impala/fe/target/dependency/commons-pool2-2.4.2.jar:/app/impala/fe/target/dependency/curator-client-2.12.0.jar:/app/impala/fe/target/dependency/curator-framework-2.12.0.jar:/app/impala/fe/target/dependency/curator-recipes-2.12.0.jar:/app/impala/fe/target/dependency/curator-test-2.11.1.jar:/app/impala/fe/target/dependency/curator-x-discovery-2.11.1.jar:/app/impala/fe/target/dependency/datanucleus-api-jdo-4.2.5.jar:/app/impala/fe/target/dependency/datanucleus-core-4.1.17.jar:/app/impala/fe/target/dependency/datanucleus-rdbms-4.1.17.jar:/app/impala/fe/target/dependency/derby-10.14.2.0.jar:/app/impala/fe/target/dependency/disruptor-3.3.6.jar:/app/impala/fe/target/dependency/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/app/impala/fe/target/dependency/eclipselink-2.5.2.jar:/app/impala/fe/target/dependency/ehcache-3.3.1.jar:/app/impala/fe/target/dependency/error_prone_annotations-2.3.1.jar:/app/impala/fe/target/dependency/fastutil-6.5.6.jar:/app/impala/fe/target/dependency/findbugs-annotations-1.3.9-1.jar:/app/impala/fe/target/dependency/flatbuffers-java-1.6.0.1.jar:/app/impala/fe/target/dependency/fluent-hc-4.3.2.jar:/app/impala/fe/target/dependency/fst-2.50.jar:/app/impala/fe/target/dependency/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/app/impala/fe/target/dependency/gethostname4j-0.0.2.jar:/app/impala/fe/target/dependency/groovy-all-2.4.11.jar:/app/impala/fe/target/dependency/gson-2.2.4.jar:/app/impala/fe/target/dependency/guava-14.0.1.jar:/app/impala/fe/target/dependency/guice-4.0.jar:/app/impala/fe/target/dependency/guice-assistedinject-3.0.jar:/app/impala/fe/target/dependency/guice-servlet-4.0.jar:/app/impala/fe/target/dependency/hadoop-annotations-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-auth-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-aws-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-azure-3.0.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hadoop-azure-datalake-3.0.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hadoop-client-api-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-client-runtime-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-common-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-distcp-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-hdfs-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-hdfs-client-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-lzo-0.4.20-SNAPSHOT.jar:/app/impala/fe/target/dependency/hadoop-mapreduce-client-app-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-mapreduce-client-common-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-mapreduce-client-core-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-yarn-api-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-yarn-client-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-yarn-common-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-yarn-server-applicationhistoryservice-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-yarn-server-common-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-yarn-server-resourcemanager-3.1.2.jar:/app/impala/fe/target/dependency/hadoop-yarn-server-web-proxy-3.1.2.jar:/app/impala/fe/target/dependency/hbase-client-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-common-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-hadoop2-compat-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-hadoop2-compat-2.1.0-cdh6.x-SNAPSHOT-tests.jar:/app/impala/fe/target/dependency/hbase-hadoop-compat-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-http-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-mapreduce-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-metrics-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-metrics-api-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-procedure-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-protocol-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-protocol-shaded-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-replication-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-server-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/hbase-shaded-miscellaneous-2.2.1.jar:/app/impala/fe/target/dependency/hbase-shaded-netty-2.2.1.jar:/app/impala/fe/target/dependency/hbase-shaded-protobuf-2.2.1.jar:/app/impala/fe/target/dependency/hbase-zookeeper-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/HikariCP-2.6.1.jar:/app/impala/fe/target/dependency/HikariCP-java7-2.4.12.jar:/app/impala/fe/target/dependency/hive-common-2.3.7.jar:/app/impala/fe/target/dependency/hive-exec-2.3.7.jar:/app/impala/fe/target/dependency/hive-hbase-handler-2.3.7.jar:/app/impala/fe/target/dependency/hive-llap-client-2.3.7.jar:/app/impala/fe/target/dependency/hive-llap-common-2.3.7.jar:/app/impala/fe/target/dependency/hive-llap-tez-2.3.7.jar:/app/impala/fe/target/dependency/hive-metastore-2.3.7.jar:/app/impala/fe/target/dependency/hive-orc-2.2.0.jar:/app/impala/fe/target/dependency/hive-serde-2.3.7.jar:/app/impala/fe/target/dependency/hive-service-2.3.7.jar:/app/impala/fe/target/dependency/hive-service-rpc-2.3.7.jar:/app/impala/fe/target/dependency/hive-shims-0.23-2.3.7.jar:/app/impala/fe/target/dependency/hive-shims-2.3.7.jar:/app/impala/fe/target/dependency/hive-shims-common-2.3.7.jar:/app/impala/fe/target/dependency/hive-shims-scheduler-2.3.7.jar:/app/impala/fe/target/dependency/hive-storage-api-2.4.0.jar:/app/impala/fe/target/dependency/hk2-api-2.5.0-b32.jar:/app/impala/fe/target/dependency/hk2-locator-2.5.0-b32.jar:/app/impala/fe/target/dependency/hk2-utils-2.5.0-b32.jar:/app/impala/fe/target/dependency/htrace-core4-4.1.0-incubating.jar:/app/impala/fe/target/dependency/httpclient-4.5.3.jar:/app/impala/fe/target/dependency/httpcore-4.4.9.jar:/app/impala/fe/target/dependency/httpmime-4.5.3.jar:/app/impala/fe/target/dependency/hudi-common-0.5.0-incubating.jar:/app/impala/fe/target/dependency/hudi-hadoop-mr-0.5.0-incubating.jar:/app/impala/fe/target/dependency/impala-data-source-api-1.0-SNAPSHOT.jar:/app/impala/fe/target/dependency/ivy-2.4.0.jar:/app/impala/fe/target/dependency/jackson-annotations-2.9.10.jar:/app/impala/fe/target/dependency/jackson-core-2.10.0.jar:/app/impala/fe/target/dependency/jackson-core-asl-1.9.13.jar:/app/impala/fe/target/dependency/jackson-databind-2.10.0.jar:/app/impala/fe/target/dependency/jackson-jaxrs-1.9.13.jar:/app/impala/fe/target/dependency/jackson-jaxrs-base-2.9.10.jar:/app/impala/fe/target/dependency/jackson-jaxrs-json-provider-2.9.10.jar:/app/impala/fe/target/dependency/jackson-mapper-asl-1.9.13-cloudera.1.jar:/app/impala/fe/target/dependency/jackson-module-jaxb-annotations-2.9.10.jar:/app/impala/fe/target/dependency/jackson-xc-1.9.2.jar:/app/impala/fe/target/dependency/jamon-runtime-2.3.1.jar:/app/impala/fe/target/dependency/janino-2.7.6.jar:/app/impala/fe/target/dependency/jasper-compiler-5.5.23.jar:/app/impala/fe/target/dependency/jasper-runtime-5.5.23.jar:/app/impala/fe/target/dependency/java-cup-0.11-a-czt02-cdh.jar:/app/impala/fe/target/dependency/java-cup-runtime-0.11-a-czt01-cdh.jar:/app/impala/fe/target/dependency/javassist-3.18.1-GA.jar:/app/impala/fe/target/dependency/java-util-1.9.0.jar:/app/impala/fe/target/dependency/javax.activation-api-1.2.0.jar:/app/impala/fe/target/dependency/javax.annotation-api-1.2.jar:/app/impala/fe/target/dependency/javax.el-3.0.1-b08.jar:/app/impala/fe/target/dependency/javax.inject-1.jar:/app/impala/fe/target/dependency/javax.inject-2.5.0-b32.jar:/app/impala/fe/target/dependency/javax.jdo-3.2.0-m3.jar:/app/impala/fe/target/dependency/javax.json-1.0.2.jar:/app/impala/fe/target/dependency/javax.persistence-2.1.0.jar:/app/impala/fe/target/dependency/javax.servlet-api-3.1.0.jar:/app/impala/fe/target/dependency/javax.servlet.jsp-2.3.2.jar:/app/impala/fe/target/dependency/javax.servlet.jsp-api-2.3.1.jar:/app/impala/fe/target/dependency/javax.ws.rs-api-2.0.1.jar:/app/impala/fe/target/dependency/javolution-5.5.1.jar:/app/impala/fe/target/dependency/jaxb-api-2.2.11.jar:/app/impala/fe/target/dependency/jaxb-impl-2.2.3-1.jar:/app/impala/fe/target/dependency/jcip-annotations-1.0-1.jar:/app/impala/fe/target/dependency/jcl-over-slf4j-1.7.24.jar:/app/impala/fe/target/dependency/jcodings-1.0.18.jar:/app/impala/fe/target/dependency/jdo-api-3.0.1.jar:/app/impala/fe/target/dependency/jersey-client-1.19.jar:/app/impala/fe/target/dependency/jersey-client-2.25.1.jar:/app/impala/fe/target/dependency/jersey-common-2.25.1.jar:/app/impala/fe/target/dependency/jersey-container-servlet-core-2.25.1.jar:/app/impala/fe/target/dependency/jersey-core-1.19.jar:/app/impala/fe/target/dependency/jersey-guava-2.25.1.jar:/app/impala/fe/target/dependency/jersey-guice-1.19.jar:/app/impala/fe/target/dependency/jersey-json-1.19.jar:/app/impala/fe/target/dependency/jersey-media-jaxb-2.25.1.jar:/app/impala/fe/target/dependency/jersey-server-1.19.jar:/app/impala/fe/target/dependency/jersey-server-2.25.1.jar:/app/impala/fe/target/dependency/jersey-servlet-1.19.jar:/app/impala/fe/target/dependency/jettison-1.1.jar:/app/impala/fe/target/dependency/jetty-util-9.3.25.v20180904.jar:/app/impala/fe/target/dependency/jline-2.12.jar:/app/impala/fe/target/dependency/jna-5.2.0.jar:/app/impala/fe/target/dependency/jna-platform-5.2.0.jar:/app/impala/fe/target/dependency/joda-time-2.9.9.jar:/app/impala/fe/target/dependency/joni-2.1.11.jar:/app/impala/fe/target/dependency/jpam-1.1.jar:/app/impala/fe/target/dependency/jsch-0.1.54.jar:/app/impala/fe/target/dependency/json-20090211.jar:/app/impala/fe/target/dependency/json-io-2.5.1.jar:/app/impala/fe/target/dependency/json-simple-1.1.1.jar:/app/impala/fe/target/dependency/json-smart-2.3.jar:/app/impala/fe/target/dependency/jsp-api-2.0.jar:/app/impala/fe/target/dependency/jsp-api-2.1.jar:/app/impala/fe/target/dependency/jsr305-3.0.0.jar:/app/impala/fe/target/dependency/jsr311-api-1.1.1.jar:/app/impala/fe/target/dependency/jta-1.1.jar:/app/impala/fe/target/dependency/kerb-admin-1.0.0.jar:/app/impala/fe/target/dependency/kerb-client-1.0.0.jar:/app/impala/fe/target/dependency/kerb-common-1.0.0.jar:/app/impala/fe/target/dependency/kerb-core-1.0.0.jar:/app/impala/fe/target/dependency/kerb-crypto-1.0.0.jar:/app/impala/fe/target/dependency/kerb-identity-1.0.0.jar:/app/impala/fe/target/dependency/kerb-server-1.0.0.jar:/app/impala/fe/target/dependency/kerb-simplekdc-1.0.0.jar:/app/impala/fe/target/dependency/kerb-util-1.0.0.jar:/app/impala/fe/target/dependency/kerby-asn1-1.0.0.jar:/app/impala/fe/target/dependency/kerby-config-1.0.0.jar:/app/impala/fe/target/dependency/kerby-pkix-1.0.0.jar:/app/impala/fe/target/dependency/kerby-util-1.0.0.jar:/app/impala/fe/target/dependency/kerby-xdr-1.0.0.jar:/app/impala/fe/target/dependency/kryo-shaded-4.0.2.jar:/app/impala/fe/target/dependency/kudu-client-1.12.0.jar:/app/impala/fe/target/dependency/libfb303-0.9.3.jar:/app/impala/fe/target/dependency/libthrift-0.9.3-1.jar:/app/impala/fe/target/dependency/log4j-1.2.17.jar:/app/impala/fe/target/dependency/log4j-api-2.8.2.jar:/app/impala/fe/target/dependency/log4j-core-2.8.2.jar:/app/impala/fe/target/dependency/log4j-web-2.8.2.jar:/app/impala/fe/target/dependency/logredactor-2.0.7.jar:/app/impala/fe/target/dependency/mail-1.4.jar:/app/impala/fe/target/dependency/metrics-core-3.2.2.jar:/app/impala/fe/target/dependency/metrics-json-3.1.0.jar:/app/impala/fe/target/dependency/metrics-jvm-3.1.0.jar:/app/impala/fe/target/dependency/minlog-1.3.0.jar:/app/impala/fe/target/dependency/mssql-jdbc-6.2.1.jre7.jar:/app/impala/fe/target/dependency/mysql-connector-java-8.0.19.jar:/app/impala/fe/target/dependency/netty-3.10.6.Final.jar:/app/impala/fe/target/dependency/netty-all-4.1.42.Final.jar:/app/impala/fe/target/dependency/nimbus-jose-jwt-4.41.1.jar:/app/impala/fe/target/dependency/noggit-0.8.jar:/app/impala/fe/target/dependency/objenesis-2.6.jar:/app/impala/fe/target/dependency/okhttp-2.7.5.jar:/app/impala/fe/target/dependency/okio-1.6.0.jar:/app/impala/fe/target/dependency/opencsv-2.3.jar:/app/impala/fe/target/dependency/orc-core-1.6.2.jar:/app/impala/fe/target/dependency/orc-shims-1.6.2.jar:/app/impala/fe/target/dependency/oro-2.0.8.jar:/app/impala/fe/target/dependency/osgi-resource-locator-1.0.1.jar:/app/impala/fe/target/dependency/paranamer-2.8.jar:/app/impala/fe/target/dependency/parquet-hadoop-bundle-1.10.99-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/postgresql-42.2.5.jar:/app/impala/fe/target/dependency/protobuf-java-2.5.0.jar:/app/impala/fe/target/dependency/query-event-hook-api-1.0-SNAPSHOT.jar:/app/impala/fe/target/dependency/ranger-plugins-audit-2.0.0.jar:/app/impala/fe/target/dependency/ranger-plugins-common-2.0.0.jar:/app/impala/fe/target/dependency/ranger-plugins-cred-2.0.0.jar:/app/impala/fe/target/dependency/re2j-1.1.jar:/app/impala/fe/target/dependency/rocksdbjni-5.17.2.jar:/app/impala/fe/target/dependency/sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-binding-hive-common-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-binding-hive-conf-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-binding-hive-follower-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-core-model-db-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-core-model-indexer-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-core-model-kafka-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-core-model-solr-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-hdfs-common-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-policy-common-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-policy-engine-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-provider-cache-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-provider-common-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-provider-db-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-provider-file-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-service-client-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/sentry-shaded-miscellaneous-2.1.0-cdh6.x-SNAPSHOT.jar:/app/impala/fe/target/dependency/shiro-cache-1.4.0.jar:/app/impala/fe/target/dependency/shiro-config-core-1.4.0.jar:/app/impala/fe/target/dependency/shiro-config-ogdl-1.4.0.jar:/app/impala/fe/target/dependency/shiro-core-1.4.0.jar:/app/impala/fe/target/dependency/shiro-crypto-cipher-1.4.0.jar:/app/impala/fe/target/dependency/shiro-crypto-core-1.4.0.jar:/app/impala/fe/target/dependency/shiro-crypto-hash-1.4.0.jar:/app/impala/fe/target/dependency/shiro-event-1.4.0.jar:/app/impala/fe/target/dependency/shiro-lang-1.4.0.jar:/app/impala/fe/target/dependency/sizeof-0.3.0.jar:/app/impala/fe/target/dependency/slf4j-api-1.7.25.jar:/app/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar:/app/impala/fe/target/dependency/snappy-0.2.jar:/app/impala/fe/target/dependency/snappy-java-1.1.4.jar:/app/impala/fe/target/dependency/solr-solrj-7.7.1.jar:/app/impala/fe/target/dependency/ST4-4.0.4.jar:/app/impala/fe/target/dependency/stax2-api-3.1.4.jar:/app/impala/fe/target/dependency/stax-api-1.0.1.jar:/app/impala/fe/target/dependency/stringtemplate-3.2.1.jar:/app/impala/fe/target/dependency/tephra-api-0.6.0.jar:/app/impala/fe/target/dependency/tephra-core-0.6.0.jar:/app/impala/fe/target/dependency/tephra-hbase-compat-1.0-0.6.0.jar:/app/impala/fe/target/dependency/twill-api-0.6.0-incubating.jar:/app/impala/fe/target/dependency/twill-common-0.6.0-incubating.jar:/app/impala/fe/target/dependency/twill-core-0.6.0-incubating.jar:/app/impala/fe/target/dependency/twill-discovery-api-0.6.0-incubating.jar:/app/impala/fe/target/dependency/twill-discovery-core-0.6.0-incubating.jar:/app/impala/fe/target/dependency/twill-zookeeper-0.6.0-incubating.jar:/app/impala/fe/target/dependency/validation-api-1.1.0.Final.jar:/app/impala/fe/target/dependency/velocity-1.5.jar:/app/impala/fe/target/dependency/wildfly-openssl-1.0.4.Final.jar:/app/impala/fe/target/dependency/woodstox-core-5.0.3.jar:/app/impala/fe/target/dependency/woodstox-core-asl-4.4.1.jar:/app/impala/fe/target/dependency/xz-1.8.jar:/app/impala/fe/target/dependency/yarn-extras-0.1-SNAPSHOT.jar:/app/impala/fe/target/dependency/zookeeper-3.4.5-cdh6.x-SNAPSHOT.jar
拷贝mysql连接jar
可以加上上述classpath里
启动各个进程
export CLASSPATH=/app/impala/../hadoop-lzo/build/hadoop-lzo-0.4.15.jar:/app/impala/fe/src/test/resources:/app/impala/fe/target/classes:/app/impala/fe/target/dependency:.:/usr/java/jdk1.8.0_201-amd64/lib/dt.jar:/usr/java/jdk1.8.0_201-amd64/lib/tools.jar:/app/impala/fe/target/dependency/*.jar
source /app/impala/config.sh
/app/impala/be/build/debug/service/statestored ${IMPALA_STATE_STORE_ARGS}
/app/impala/be/build/debug/service/catalogd ${IMPALA_CATALOG_ARGS}
#注意指令有变化
# /app/impala/be/build/debug/service/impalad -log_dir=/app/impala/logs/impala -state_store_host=sit1 -catalog_service_host=sit1 -state_store_port=24000 -be_port=22000
/app/impala/be/build/debug/service/impalad ${IMPALA_SERVER_ARGS}
测试impala
/app/impala/shell/build/impala-shell-3.4.0-RELEASE/impala-shell
报错
Exception thrown when executing query : SELECT ‘org.apache.hadoop.hive.metastore.model.MVersionTable’ AS
NUCLEUS_TYPE
,A0
.SCHEMA_VERSION
,A1
.SCHEMA_VERSION
,A0
.VERSION_COMMENT
,A0
.VER_ID
FROMVERSION
A0
INNER JOINCDH_VERSION
A1
ONA0
.VER_ID
=A1
.VER_ID
这是由于impala默认的hive和hadoop版本是CDH的,所以需要更换上面的impala的相关依赖替换上面的hadoop、hive、habase开头的相关包
报错
Event: 0.319 loading class org/apache/hadoop/hdfs/web/SWebHdfsFileSystem
Event: 0.319 loading class org/apache/hadoop/hdfs/web/SWebHdfsFileSystem done
Event: 0.329 loading class org/apache/hadoop/hive/ql/io/NullScanFileSystem
Event: 0.329 loading class org/apache/hadoop/hive/ql/io/NullScanFileSystem done
Event: 0.330 loading class org/apache/hadoop/security/token/Token
Event: 0.330 loading class org/apache/hadoop/security/token/Token done
Event: 0.332 loading class org/apache/hadoop/hive/ql/io/ProxyLocalFileSystem
Event: 0.332 loading class org/apache/hadoop/hive/ql/io/ProxyLocalFileSystem done
Event: 0.333 loading class org/apache/hadoop/shaded/com/ctc/wstx/io/InputBootstrapper
Event: 0.333 loading class org/apache/hadoop/shaded/com/ctc/wstx/io/InputBootstrapper done
这是由于hadoop3.1.2除了需要impala那些个同名jar文件外还需要引入hadoop-client-runtime-3.1.2.jar
报错
E0312 14:10:09.454614 30004 impala-server.cc:371] Currently configured default filesystem: ProxyLocalFileSystem. fs.defaultFS (file:///) is not supported.
这个问题太恶心了,网上几乎没有解答对的,这个问题的解决办法就是,当你启动impala的时候,当前目录必须是hadoop配置文件的目录,要不就报上面那个错误,不知道为什么指定了hadoop的目录不可以
查询报错
CAUSED BY: NoClassDefFoundError: org/apache/hadoop/fs/adl/AdlFileSystem
文章浏览阅读461次。本次教程用宇宙模拟器space engine 0.980版本教大家如何创建自定义太阳系系统,包括创建恒星,行星,卫星,小行星,彗星等天体,最后教如何制作插件,打包压缩成pak文件。把链接复制到浏览器地址栏上,按回车键就可以看了关于宇宙模拟器Space Engine的天体(星球,星系,星云,星团等)脚本参数,如何创建天体(星球,星系,星云,星团等)的脚本,请看这些教程space engine打包创建..._space engine
文章浏览阅读10w+次,点赞5次,收藏17次。更新了Chrome,发现网页全部打不开了。都显示:“喔唷 崩溃啦”。点击任何按钮,例如,设置,选项,历史记录等,也“喔唷 崩溃啦” 卸载重装也无效解决方案找到 路径C:\Windows\System32\drivers\bd0001.sys删除(没有强制删除工具重命名也行,随便改个名字)重启电脑chrome就正常了原因可能是注册列表被一些卫士类优化工具或杀毒..._为什么googlechrome设置也打不开
文章浏览阅读1.5k次。安装thrift,我是mac机器,mac下使用homebrew安装thrift很方便,具体看另一篇博客。首先的定义接口文件: service HelloWorldService { string sayHello(1:string username)}因为thrift是支持跨平台的,所以这里接口的定义,thrift定义了自己的规范。 然后使用thrift的工具..._thrift调用例子 java
文章浏览阅读261次。我不确定术语“通配符”是否可以解释我的观点,但有时在一些现成的脚本中,我们可以调用一个非定义的函数,如find_by_age(23),其中age可以是映射到数据库表记录的任何其他内容.所以我可以调用find_by_name,find_by_email,find_by_id等等.那么我们怎么能以程序或面向对象的方式做这样的事情呢?解决方法:你正在寻找的术语是魔法.基本上是这样的:class Foo ..._php路径通配符
文章浏览阅读904次。ANR-WatchDog-ohos一个简单的监测程序,可检测到鸿蒙系统的 ANR(Application Not Response-应用程序无响应)错误并引发有意义的异常项目名称:ANR-WatchDog-ohos所属系列:鸿蒙的第三方组件适配移植功能:可检测到鸿蒙系统的ANR错误并引发有意义的异常项目移植状态:全部完成调用差异:无开发版本:sdk5,DevEco Studio2.1 beta3项..._com.github.anrwatchdog.anrerror: application not responding for at least 400
文章浏览阅读441次。赵辉《Visual+C++_MATLAB图像处理与识别实用案例精选》程序代码说明P0201:MATLAB赋值P0202:MATLAB中的for循环P0203:MATLAB中的for循环和if条件P0205:MATLAB图像处理的基本操作P0206:MATLAB高级图像处理操作P0207:根据RGB图像创建一幅灰度图像P0208:二值图像的取反操作P0209:用imshow函数显示图像P0210:在..._matlab命令行检测网络
文章浏览阅读3.1k次。经常会有机友提问,乐视MAX(乐视 X900+)手机支不支持一键刷机?由于奇兔刷机已经支持多达上千款安卓手机一键刷机,所以有时候小编也无法及时回答上来,最简单的办法就是把手机连上奇兔刷机,即可看到手机是否支持一键刷机。一键刷机并非每次都能一次成功,如果一次不成功可以多试几次,刷机失败不会对您的手机造成任何影响。奇兔市场小编给大家分享下乐视MAX(乐视 X900+)一键刷机教程,虽然使用方法很简单,..._乐视x900刷miui10
文章浏览阅读2.5k次,点赞7次,收藏49次。文章目录1.创建数据库与表2. 创建java项目并且导入jar包(buildPath)2.1 创建数据库连接文件 DBhelper.java2.2 创建vo (stu.java), 与数据库一致2.3 创建service (StuService.java)3. 编写功能类 FromDbToExcel.java注意事项:4. 编写功能类 FromExcelToDb.java注意事项5. 效果展示1.创建数据库与表2. 创建java项目并且导入jar包(buildPath)2.1 创建数据库连接文件_mysql导出多张表结构 excel java代码
文章浏览阅读352次。首先介绍阻塞与非阻塞:阻塞是个什么概念呢?比如某个时候你在等快递,但是你不知道快递什么时候过来,而且你没有别的事可以干(或者说接下来的事要等快递来了才能做);那么你可以去睡觉了,因为你知道快递把货送来时一定会给你打个电话(假定一定能叫醒你)。非阻塞忙轮询。接着上面等快递的例子,如果用忙轮询的方法,那么你需要知道快递员的手机号,然后每分钟给他挂个电话:“你到了没?”很明显一般人不会用第二..._epoll kqueue
文章浏览阅读509次。关于西门子PLC modbus通讯与运动控制的应用应用的设备昆仑通态1570gi(15寸)西门子plc smart-st20昆仑通态参数的设置与组态的重点组态软件选用型号1570gi的屏幕设置通讯地址 192.168.190报警参数的配置脚本的编写关于组态过程中遇到的问题由于这是我参加工作自己独立的完成的,这个期间真的是太坎坷了,我来说说我遇到的问题吧脚本应用if和if的嵌套IF THEN IF THEN ENDIFENDIF由于画面中关联着报_plc作为从站 modbus协议和运动控制命令冲突
文章浏览阅读972次。先用一台电脑连接IPSAN管理端口进去配置阵列配置完成后把IPSAN的ISCSI接口连接到服务器网口,man iscsiadm 查看可以查看是否安装了 iscsiadmyum -y install*iscsi* 命令安装iscsiadm然后在服务器上输入:iscsiadm -m discovery -t sendtargets -p 169.254.194.201:3260 (169.254.19..._centos实现ipsan卸载
文章浏览阅读698次。点击上方 "程序员小乐"关注,星标或置顶一起成长后台回复“大礼包”有惊喜礼包!关注订阅号「程序员小乐」,收看更多精彩内容每日英文Tough people aren..._cannot use range access on index 'env_index' due to type or collation conver