HBase实战（5）：Spark SQL+Hive +HBASE 使用Spark 操作分布式集群HBASE
本文的操作是使用spark 自带的spark sql工具 通过Hive去操作Hbase的数据。
在spark 集群中提交spark sql运行语句。分别使用了本地模式、集群模式提交，遇到的一些报错是JAR
包没加载全，提交过程中加载HBASE的相关Jar包运行就可以。
1
2
| 1root@master:~# spark-sql --master spark://192.168.189.1:7077 --driver-class-path /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar --jars /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar,/usr/local/apache-hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar,/usr/local/hbase-1.2.0/lib/hbase-client-1.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-common-1.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-protocol-1.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-server-1.2.0.jar,/usr/local/hbase-1.2.0/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.2.0/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-hadoop2-compat-1.2.0.jar,/usr/local/hbase-1.2.0/lib/guava-12.0.1.jar,/usr/local/hbase-1.2.0/lib/protobuf-java-2.5.0.jar --executor-memory 512m --total-executor-cores 4
2 |
操作步骤：
1，基于spark 2.3.0 本地模式直接进入spark sql 客户端，提示出错，找不到“com.mysql.jdbc.Driver”
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| 1root@master:/usr/local/spark-2.3.0-bin-hadoop2.6/bin# spark-sql
2SLF4J: Class path contains multiple SLF4J bindings.
3SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
4SLF4J: Found binding in [jar:file:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
5SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
6SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
72018-06-14 10:22:53 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
82018-06-14 10:22:57 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
92018-06-14 10:22:59 INFO HiveMetaStore:589 - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
102018-06-14 10:22:59 INFO ObjectStore:289 - ObjectStore, initialize called
112018-06-14 10:23:00 INFO Persistence:77 - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
122018-06-14 10:23:00 INFO Persistence:77 - Property datanucleus.cache.level2 unknown - will be ignored
132018-06-14 10:23:00 WARN HiveMetaStore:622 - Retrying creating default database after error: Error creating transactional connection factory
14javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
15 at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
16 at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
17 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| 1 ... 52 more
2Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
3 at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:259)
4 at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:131)
5 at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:85)
6 ... 70 more
7Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
8 at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
9 at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)
10 at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)
11 ... 72 more
122018-06-14 10:23:01 INFO ShutdownHookManager:54 - Shutdown hook called
132018-06-14 10:23:01 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-2fa09172-d8d5-4f4d-ba5f-84faee7c7d52
14root@master:/usr/local/spark-2.3.0-bin-hadoop2.6/bin#
15 |
2，启动Spark集群。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| 1root@master:~# /usr/local/spark-2.3.0-bin-hadoop2.6/sbin/start-all.sh
2starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.3.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
3worker2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.3.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker2.out
4worker1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.3.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker1.out
5worker1: failed to launch: nice -n 0 /usr/local/spark-2.3.0-bin-hadoop2.6/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master:7077
6worker1: full log in /usr/local/spark-2.3.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker1.out
7worker3: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.3.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker3.out
8root@master:~# jps
93075 ResourceManager
103715 HRegionServer
113395 QuorumPeerMain
122932 SecondaryNameNode
134613 Jps
143543 HMaster
152715 NameNode
163823 RunJar
174543 Master
18root@master:~#
19 |
3，在spark集群中使用spark-sql。
1
2
3
4
| 1root@master:~# spark-sql --master spark://192.168.189.1:7077 --driver-class-path /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar --executor-memory 512m --total-executor-cores 4
2
3
4 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
| 12018-06-14 10:31:36 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22018-06-14 10:31:37 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
32018-06-14 10:31:38 INFO HiveMetaStore:589 - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
42018-06-14 10:31:38 INFO ObjectStore:289 - ObjectStore, initialize called
52018-06-14 10:31:39 INFO Persistence:77 - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
62018-06-14 10:31:39 INFO Persistence:77 - Property datanucleus.cache.level2 unknown - will be ignored
72018-06-14 10:31:43 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
82018-06-14 10:31:43 INFO ObjectStore:370 - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
92018-06-14 10:31:45 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
102018-06-14 10:31:45 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
112018-06-14 10:31:49 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
122018-06-14 10:31:49 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
132018-06-14 10:31:50 INFO Query:77 - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
142018-06-14 10:31:50 INFO MetaStoreDirectSql:139 - Using direct SQL, underlying DB is MYSQL
152018-06-14 10:31:51 INFO ObjectStore:272 - Initialized ObjectStore
162018-06-14 10:31:51 INFO HiveMetaStore:663 - Added admin role in metastore
172018-06-14 10:31:51 INFO HiveMetaStore:672 - Added public role in metastore
182018-06-14 10:31:51 INFO HiveMetaStore:712 - No user is added in admin role, since config is empty
192018-06-14 10:31:52 INFO HiveMetaStore:746 - 0: get_all_databases
202018-06-14 10:31:52 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_all_databases
212018-06-14 10:31:52 INFO HiveMetaStore:746 - 0: get_functions: db=default pat=*
222018-06-14 10:31:52 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
232018-06-14 10:31:52 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
242018-06-14 10:31:52 INFO HiveMetaStore:746 - 0: get_functions: db=hive pat=*
252018-06-14 10:31:52 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hive pat=*
262018-06-14 10:31:52 INFO HiveMetaStore:746 - 0: get_functions: db=hivestudy pat=*
272018-06-14 10:31:52 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hivestudy pat=*
282018-06-14 10:31:52 INFO HiveMetaStore:746 - 0: get_functions: db=hrs pat=*
292018-06-14 10:31:52 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hrs pat=*
302018-06-14 10:31:55 INFO SessionState:641 - Created local directory: /tmp/999d5941-24ad-4579-a61c-ba719868dc9b_resources
312018-06-14 10:31:55 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/999d5941-24ad-4579-a61c-ba719868dc9b
322018-06-14 10:31:55 INFO SessionState:641 - Created local directory: /tmp/root/999d5941-24ad-4579-a61c-ba719868dc9b
332018-06-14 10:31:55 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/999d5941-24ad-4579-a61c-ba719868dc9b/_tmp_space.db
342018-06-14 10:31:56 INFO SparkContext:54 - Running Spark version 2.3.0
352018-06-14 10:31:56 INFO SparkContext:54 - Submitted application: SparkSQL::master
362018-06-14 10:31:57 INFO SecurityManager:54 - Changing view acls to: root
372018-06-14 10:31:57 INFO SecurityManager:54 - Changing modify acls to: root
382018-06-14 10:31:57 INFO SecurityManager:54 - Changing view acls groups to:
392018-06-14 10:31:57 INFO SecurityManager:54 - Changing modify acls groups to:
402018-06-14 10:31:57 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
412018-06-14 10:31:58 INFO Utils:54 - Successfully started service 'sparkDriver' on port 49760.
422018-06-14 10:31:58 INFO SparkEnv:54 - Registering MapOutputTracker
432018-06-14 10:31:59 INFO SparkEnv:54 - Registering BlockManagerMaster
442018-06-14 10:31:59 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
452018-06-14 10:31:59 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
462018-06-14 10:31:59 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-e9250684-2aee-4f04-8466-28c951fa52cf
472018-06-14 10:31:59 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
482018-06-14 10:31:59 INFO SparkEnv:54 - Registering OutputCommitCoordinator
492018-06-14 10:32:00 INFO log:192 - Logging initialized @26885ms
502018-06-14 10:32:00 INFO Server:346 - jetty-9.3.z-SNAPSHOT
512018-06-14 10:32:00 INFO Server:414 - Started @27370ms
522018-06-14 10:32:01 INFO AbstractConnector:278 - Started ServerConnector@4aa31ffc{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
532018-06-14 10:32:01 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
542018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a9860{/jobs,null,AVAILABLE,@Spark}
552018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f85217c{/jobs/json,null,AVAILABLE,@Spark}
562018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1fd7a37{/jobs/job,null,AVAILABLE,@Spark}
572018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58a84a12{/jobs/job/json,null,AVAILABLE,@Spark}
582018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e700eba{/stages,null,AVAILABLE,@Spark}
592018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6436e181{/stages/json,null,AVAILABLE,@Spark}
602018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7186b202{/stages/stage,null,AVAILABLE,@Spark}
612018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@36068727{/stages/stage/json,null,AVAILABLE,@Spark}
622018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@72543547{/stages/pool,null,AVAILABLE,@Spark}
632018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3d88e6b9{/stages/pool/json,null,AVAILABLE,@Spark}
642018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22bf9122{/storage,null,AVAILABLE,@Spark}
652018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@208205ed{/storage/json,null,AVAILABLE,@Spark}
662018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73afe2b7{/storage/rdd,null,AVAILABLE,@Spark}
672018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9687f55{/storage/rdd/json,null,AVAILABLE,@Spark}
682018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5700c9db{/environment,null,AVAILABLE,@Spark}
692018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@671d03bb{/environment/json,null,AVAILABLE,@Spark}
702018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6babffb5{/executors,null,AVAILABLE,@Spark}
712018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2173a742{/executors/json,null,AVAILABLE,@Spark}
722018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@706ceca6{/executors/threadDump,null,AVAILABLE,@Spark}
732018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f6329cb{/executors/threadDump/json,null,AVAILABLE,@Spark}
742018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b8137c5{/static,null,AVAILABLE,@Spark}
752018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31973858{/,null,AVAILABLE,@Spark}
762018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@65514add{/api,null,AVAILABLE,@Spark}
772018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@10850d17{/jobs/job/kill,null,AVAILABLE,@Spark}
782018-06-14 10:32:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51e754e1{/stages/stage/kill,null,AVAILABLE,@Spark}
792018-06-14 10:32:01 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
802018-06-14 10:32:02 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://192.168.189.1:7077...
812018-06-14 10:32:02 INFO TransportClientFactory:267 - Successfully created connection to /192.168.189.1:7077 after 164 ms (0 ms spent in bootstraps)
822018-06-14 10:32:03 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180614103203-0000
832018-06-14 10:32:03 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35493.
842018-06-14 10:32:03 INFO NettyBlockTransferService:54 - Server created on master:35493
852018-06-14 10:32:03 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
862018-06-14 10:32:03 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 35493, None)
872018-06-14 10:32:03 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:35493 with 413.9 MB RAM, BlockManagerId(driver, master, 35493, None)
882018-06-14 10:32:03 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 35493, None)
892018-06-14 10:32:03 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 35493, None)
902018-06-14 10:32:04 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614103203-0000/0 on worker-20180614103000-worker1-36930 (worker1:36930) with 1 core(s)
912018-06-14 10:32:04 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614103203-0000/0 on hostPort worker1:36930 with 1 core(s), 512.0 MB RAM
922018-06-14 10:32:04 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614103203-0000/1 on worker-20180614103020-worker3-48018 (worker3:48018) with 1 core(s)
932018-06-14 10:32:04 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614103203-0000/1 on hostPort worker3:48018 with 1 core(s), 512.0 MB RAM
942018-06-14 10:32:04 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614103203-0000/2 on worker-20180614103016-worker2-53644 (worker2:53644) with 1 core(s)
952018-06-14 10:32:04 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614103203-0000/2 on hostPort worker2:53644 with 1 core(s), 512.0 MB RAM
962018-06-14 10:32:05 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614103203-0000/2 is now RUNNING
972018-06-14 10:32:05 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614103203-0000/0 is now RUNNING
982018-06-14 10:32:05 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614103203-0000/1 is now RUNNING
992018-06-14 10:32:07 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@644d1b61{/metrics/json,null,AVAILABLE,@Spark}
1002018-06-14 10:32:07 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
1012018-06-14 10:32:10 INFO SharedState:54 - loading hive config file: file:/usr/local/spark-2.3.0-bin-hadoop2.6/conf/hive-site.xml
1022018-06-14 10:32:10 INFO SharedState:54 - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
1032018-06-14 10:32:10 INFO SharedState:54 - Warehouse path is '/user/hive/warehouse'.
1042018-06-14 10:32:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@46a795de{/SQL,null,AVAILABLE,@Spark}
1052018-06-14 10:32:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@256a0d95{/SQL/json,null,AVAILABLE,@Spark}
1062018-06-14 10:32:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@44f0ff2b{/SQL/execution,null,AVAILABLE,@Spark}
1072018-06-14 10:32:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22ead351{/SQL/execution/json,null,AVAILABLE,@Spark}
1082018-06-14 10:32:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@601eb4af{/static/sql,null,AVAILABLE,@Spark}
1092018-06-14 10:32:13 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
1102018-06-14 10:32:15 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
1112018-06-14 10:32:16 INFO HiveMetaStore:746 - 0: get_database: default
1122018-06-14 10:32:16 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
1132018-06-14 10:32:37 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
114spark-sql> 2018-06-14 10:32:53 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.4:56970) with ID 1
1152018-06-14 10:32:55 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.3:50799) with ID 2
1162018-06-14 10:32:56 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker2:53191 with 117.0 MB RAM, BlockManagerId(2, worker2, 53191, None)
1172018-06-14 10:33:06 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker3:47807 with 117.0 MB RAM, BlockManagerId(1, worker3, 47807, None)
118
119 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| 1spark-sql> show databases;
22018-06-14 10:39:20 INFO HiveMetaStore:746 - 0: get_database: global_temp
32018-06-14 10:39:20 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: global_temp
42018-06-14 10:39:21 WARN ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException
52018-06-14 10:39:30 INFO HiveMetaStore:746 - 0: get_databases: *
62018-06-14 10:39:30 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_databases: *
72018-06-14 10:39:33 INFO CodeGenerator:54 - Code generated in 525.870215 ms
8default
9hive
10hivestudy
11hrs
12Time taken: 12.771 seconds, Fetched 4 row(s)
132018-06-14 10:39:33 INFO SparkSQLCLIDriver:951 - Time taken: 12.771 seconds, Fetched 4 row(s)
14 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
| 1spark-sql> use default;
22018-06-14 10:39:41 INFO HiveMetaStore:746 - 0: get_database: default
32018-06-14 10:39:41 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
4Time taken: 0.102 seconds
52018-06-14 10:39:41 INFO SparkSQLCLIDriver:951 - Time taken: 0.102 seconds
6spark-sql> show tables;
72018-06-14 10:39:46 INFO HiveMetaStore:746 - 0: get_database: default
82018-06-14 10:39:46 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
92018-06-14 10:39:46 INFO HiveMetaStore:746 - 0: get_database: default
102018-06-14 10:39:46 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
112018-06-14 10:39:46 INFO HiveMetaStore:746 - 0: get_tables: db=default pat=*
122018-06-14 10:39:46 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_tables: db=default pat=*
132018-06-14 10:39:47 INFO ContextCleaner:54 - Cleaned accumulator 0
142018-06-14 10:39:47 INFO ContextCleaner:54 - Cleaned accumulator 1
152018-06-14 10:39:47 INFO CodeGenerator:54 - Code generated in 34.691478 ms
16default booktable false
17default hbase_hive.....operation false
18default partition_test false
19default peopledepljion false
20default pokes false
21default src false
22Time taken: 0.442 seconds, Fetched 6 row(s)
232018-06-14 10:39:47 INFO SparkSQLCLIDriver:951 - Time taken: 0.442 seconds, Fetched 6 row(s)
24 |
1
2
| 1spark sql通过hive查询hbase，提示出错。 找不到“org.apache.hadoop.hive.hbase.HBaseStorageHandler”
2 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
| 1spark-sql> select * from hbase_hive_wtb_ow_operation;
22018-06-14 10:42:38 INFO HiveMetaStore:746 - 0: get_table : db=default tbl=hbase_hive_wtb_ow_operation
32018-06-14 10:42:38 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_table : db=default tbl=hbase_hive_wtb_ow_operation
42018-06-14 10:42:39 ERROR log:397 - error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
5java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
6 at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
7 at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:385)
8 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
9 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
10 at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)
11 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:358)
12 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:355)
13 at scala.Option.map(Option.scala:146)
14 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:355)
15 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:353)
16 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:272)
17 at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210)
18 at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209)
19 at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:255)
20 at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:353)
21 at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
22 at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
23 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
24 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
25 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
26 at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
27 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
28 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
29 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
30 at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
31 at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:669)
32 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:660)
33 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:615)
34 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:645)
35 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:638)
36 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
37 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
38 at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
39 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
40 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
41 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
42 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
43 at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
44 at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
45 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
46 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:638)
47 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:584)
48 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
49 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
50 at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
51 at scala.collection.immutable.List.foldLeft(List.scala:84)
52 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
53 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
54 at scala.collection.immutable.List.foreach(List.scala:381)
55 at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
56 at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:123)
57 at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:117)
58 at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:102)
59 at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
60 at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
61 at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
62 at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
63 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
64 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
65 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
66 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
67 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
68 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
69 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
70 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
71 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
72 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
73 at java.lang.reflect.Method.invoke(Method.java:497)
74 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
75 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
76 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
77 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
78 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
79 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
802018-06-14 10:42:39 ERROR Table:608 - Unable to get field from serde: org.apache.hadoop.hive.hbase.HBaseSerDe
81java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found)
82 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:278)
83 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
84 at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)
85 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:358)
86 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:355)
87 at scala.Option.map(Option.scala:146)
88 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:355)
89 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:353)
90 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:272)
91 at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210)
92 at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209)
93 at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:255)
94 at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:353)
95 at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
96 at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
97 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
98 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
99 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
100 at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
101 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
102 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
103 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
104 at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
105 at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:669)
106 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:660)
107 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:615)
108 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:645)
109 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:638)
110 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
111 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
112 at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
113 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
114 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
115 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
116 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
117 at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
118 at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
119 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
120 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:638)
121 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:584)
122 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
123 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
124 at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
125 at scala.collection.immutable.List.foldLeft(List.scala:84)
126 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
127 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
128 at scala.collection.immutable.List.foreach(List.scala:381)
129 at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
130 at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:123)
131 at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:117)
132 at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:102)
133 at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
134 at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
135 at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
136 at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
137 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
138 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
139 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
140 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
141 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
142 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
143 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
144 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
145 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
146 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
147 at java.lang.reflect.Method.invoke(Method.java:497)
148 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
149 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
150 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
151 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
152 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
153 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
154Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found)
155 at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:399)
156 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
157 ... 71 more
1582018-06-14 10:42:39 ERROR SparkSQLDriver:91 - Failed in [select * from hbase_hive_wtb_ow_operation]
159java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
160 at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:292)
161 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:388)
162 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:355)
163 at scala.Option.map(Option.scala:146)
164 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:355)
165 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:353)
166 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:272)
167 at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210)
168 at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209)
169 at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:255)
170 at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:353)
171 at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
172 at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
173 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
174 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
175 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
176 at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
177 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
178 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
179 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
180 at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
181 at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:669)
182 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:660)
183 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:615)
184 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:645)
185 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:638)
186 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
187 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
188 at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
189 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
190 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
191 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
192 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
193 at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
194 at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
195 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
196 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:638)
197 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:584)
198 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
199 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
200 at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
201 at scala.collection.immutable.List.foldLeft(List.scala:84)
202 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
203 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
204 at scala.collection.immutable.List.foreach(List.scala:381)
205 at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
206 at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:123)
207 at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:117)
208 at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:102)
209 at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
210 at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
211 at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
212 at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
213 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
214 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
215 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
216 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
217 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
218 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
219 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
220 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
221 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
222 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
223 at java.lang.reflect.Method.invoke(Method.java:497)
224 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
225 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
226 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
227 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
228 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
229 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
230Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
231 at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:315)
232 at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:287)
233 ... 69 more
234Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HBaseStorageHandler
235 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
236 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
237 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
238 at java.lang.Class.forName0(Native Method)
239 at java.lang.Class.forName(Class.java:348)
240 at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:309)
241 ... 70 more
242java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
243 at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:292)
244 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:388)
245 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:355)
246 at scala.Option.map(Option.scala:146)
247 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:355)
248 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:353)
249 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:272)
250 at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210)
251 at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209)
252 at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:255)
253 at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:353)
254 at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
255 at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
256 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
257 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
258 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
259 at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
260 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
261 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
262 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
263 at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
264 at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:669)
265 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:660)
266 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:615)
267 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:645)
268 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:638)
269 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
270 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
271 at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
272 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
273 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
274 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
275 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
276 at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
277 at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
278 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
279 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:638)
280 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:584)
281 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
282 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
283 at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
284 at scala.collection.immutable.List.foldLeft(List.scala:84)
285 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
286 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
287 at scala.collection.immutable.List.foreach(List.scala:381)
288 at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
289 at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:123)
290 at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:117)
291 at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:102)
292 at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
293 at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
294 at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
295 at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
296 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
297 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
298 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
299 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
300 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
301 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
302 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
303 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
304 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
305 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
306 at java.lang.reflect.Method.invoke(Method.java:497)
307 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
308 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
309 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
310 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
311 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
312 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
313Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
314 at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:315)
315 at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:287)
316 ... 69 more
317Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HBaseStorageHandler
318 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
319 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
320 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
321 at java.lang.Class.forName0(Native Method)
322 at java.lang.Class.forName(Class.java:348)
323 at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:309)
324 ... 70 more
325
326spark-sql>
327 |
检查hive-hbase-handler-1.2.1.jar路径：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
| 1root@master:/usr/local/apache-hive-1.2.1/lib# ls
2accumulo-core-1.6.0.jar commons-httpclient-3.0.1.jar hive-hbase-handler-1.2.1.jar libfb303-0.9.2.jar
3accumulo-fate-1.6.0.jar commons-io-2.4.jar hive-hwi-1.2.1.jar libthrift-0.9.2.jar
4accumulo-start-1.6.0.jar commons-lang-2.6.jar hive-jdbc-1.2.1.jar log4j-1.2.16.jar
5accumulo-trace-1.6.0.jar commons-logging-1.1.3.jar hive-jdbc-1.2.1-standalone.jar mail-1.4.1.jar
6activation-1.1.jar commons-math-2.1.jar hive-metastore-1.2.1.jar maven-scm-api-1.4.jar
7ant-1.9.1.jar commons-pool-1.5.4.jar hive-serde-1.2.1.jar maven-scm-provider-svn-commons-1.4.jar
8ant-launcher-1.9.1.jar commons-vfs2-2.0.jar hive-service-1.2.1.jar maven-scm-provider-svnexe-1.4.jar
9antlr-2.7.7.jar curator-client-2.6.0.jar hive-shims-0.20S-1.2.1.jar mysql-connector-java-5.1.13-bin.jar
10antlr-runtime-3.4.jar curator-framework-2.6.0.jar hive-shims-0.23-1.2.1.jar netty-3.7.0.Final.jar
11apache-curator-2.6.0.pom curator-recipes-2.6.0.jar hive-shims-1.2.1.jar opencsv-2.3.jar
12apache-log4j-extras-1.2.17.jar datanucleus-api-jdo-3.2.6.jar hive-shims-common-1.2.1.jar oro-2.0.8.jar
13asm-commons-3.1.jar datanucleus-core-3.2.10.jar hive-shims-scheduler-1.2.1.jar paranamer-2.3.jar
14asm-tree-3.1.jar datanucleus-rdbms-3.2.9.jar hive-testutils-1.2.1.jar parquet-hadoop-bundle-1.6.0.jar
15avro-1.7.5.jar derby-10.10.2.0.jar httpclient-4.4.jar pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
16bonecp-0.8.0.RELEASE.jar eigenbase-properties-1.1.5.jar httpcore-4.4.jar php
17calcite-avatica-1.2.0-incubating.jar geronimo-annotation_1.0_spec-1.1.1.jar ivy-2.4.0.jar plexus-utils-1.5.6.jar
18calcite-core-1.2.0-incubating.jar geronimo-jaspic_1.0_spec-1.0.jar janino-2.7.6.jar py
19calcite-linq4j-1.2.0-incubating.jar geronimo-jta_1.1_spec-1.1.1.jar jcommander-1.32.jar regexp-1.3.jar
20commons-beanutils-1.7.0.jar groovy-all-2.1.6.jar jdo-api-3.0.1.jar servlet-api-2.5.jar
21commons-beanutils-core-1.8.0.jar guava-14.0.1.jar jetty-all-7.6.0.v20120127.jar snappy-java-1.0.5.jar
22commons-cli-1.2.jar hamcrest-core-1.1.jar jetty-all-server-7.6.0.v20120127.jar ST4-4.0.4.jar
23commons-codec-1.4.jar hive-accumulo-handler-1.2.1.jar jline-2.12.jar stax-api-1.0.1.jar
24commons-collections-3.2.1.jar hive-ant-1.2.1.jar joda-time-2.5.jar stringtemplate-3.2.1.jar
25commons-compiler-2.7.6.jar hive-beeline-1.2.1.jar jpam-1.1.jar super-csv-2.2.0.jar
26commons-compress-1.4.1.jar hive-cli-1.2.1.jar json-20090211.jar tempus-fugit-1.1.jar
27commons-configuration-1.6.jar hive-common-1.2.1.jar jsr305-3.0.0.jar velocity-1.5.jar
28commons-dbcp-1.4.jar hive-contrib-1.2.1.jar jta-1.1.jar xz-1.0.jar
29commons-digester-1.8.jar hive-exec-1.2.1.jar junit-4.11.jar zookeeper-3.4.6.jar
30root@master:/usr/local/apache-hive-1.2.1/lib# ls | grep hbase
31hive-hbase-handler-1.2.1.jar
32root@master:/usr/local/apache-hive-1.2.1/lib#
33 |
加载hive-hbase-handler-1.2.1.jar的路径，重新在spark集群提交spark sql。
1
2
3
4
| 1root@master:~# spark-sql --master spark://192.168.189.1:7077 --driver-class-path /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar:/usr/local/apache-hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar --executor-memory 512m --total-executor-cores 4
2
3
4 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
| 12018-06-14 13:58:33 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22018-06-14 13:58:36 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
32018-06-14 13:58:36 INFO HiveMetaStore:589 - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
42018-06-14 13:58:36 INFO ObjectStore:289 - ObjectStore, initialize called
52018-06-14 13:58:37 INFO Persistence:77 - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
62018-06-14 13:58:37 INFO Persistence:77 - Property datanucleus.cache.level2 unknown - will be ignored
72018-06-14 13:58:39 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
82018-06-14 13:58:39 INFO ObjectStore:370 - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
92018-06-14 13:58:40 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
102018-06-14 13:58:40 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
112018-06-14 13:58:41 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
122018-06-14 13:58:41 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
132018-06-14 13:58:42 INFO Query:77 - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
142018-06-14 13:58:42 INFO MetaStoreDirectSql:139 - Using direct SQL, underlying DB is MYSQL
152018-06-14 13:58:42 INFO ObjectStore:272 - Initialized ObjectStore
162018-06-14 13:58:42 INFO HiveMetaStore:663 - Added admin role in metastore
172018-06-14 13:58:42 INFO HiveMetaStore:672 - Added public role in metastore
182018-06-14 13:58:42 INFO HiveMetaStore:712 - No user is added in admin role, since config is empty
192018-06-14 13:58:43 INFO HiveMetaStore:746 - 0: get_all_databases
202018-06-14 13:58:43 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_all_databases
212018-06-14 13:58:43 INFO HiveMetaStore:746 - 0: get_functions: db=default pat=*
222018-06-14 13:58:43 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
232018-06-14 13:58:43 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
242018-06-14 13:58:43 INFO HiveMetaStore:746 - 0: get_functions: db=hive pat=*
252018-06-14 13:58:43 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hive pat=*
262018-06-14 13:58:43 INFO HiveMetaStore:746 - 0: get_functions: db=hivestudy pat=*
272018-06-14 13:58:43 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hivestudy pat=*
282018-06-14 13:58:43 INFO HiveMetaStore:746 - 0: get_functions: db=hrs pat=*
292018-06-14 13:58:43 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hrs pat=*
302018-06-14 13:58:44 INFO SessionState:641 - Created local directory: /tmp/54a94cff-e4a8-4500-bd56-7eda47f5b72f_resources
312018-06-14 13:58:45 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/54a94cff-e4a8-4500-bd56-7eda47f5b72f
322018-06-14 13:58:45 INFO SessionState:641 - Created local directory: /tmp/root/54a94cff-e4a8-4500-bd56-7eda47f5b72f
332018-06-14 13:58:45 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/54a94cff-e4a8-4500-bd56-7eda47f5b72f/_tmp_space.db
342018-06-14 13:58:45 INFO SparkContext:54 - Running Spark version 2.3.0
352018-06-14 13:58:45 INFO SparkContext:54 - Submitted application: SparkSQL::master
362018-06-14 13:58:46 INFO SecurityManager:54 - Changing view acls to: root
372018-06-14 13:58:46 INFO SecurityManager:54 - Changing modify acls to: root
382018-06-14 13:58:46 INFO SecurityManager:54 - Changing view acls groups to:
392018-06-14 13:58:46 INFO SecurityManager:54 - Changing modify acls groups to:
402018-06-14 13:58:46 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
412018-06-14 13:58:46 INFO Utils:54 - Successfully started service 'sparkDriver' on port 35632.
422018-06-14 13:58:47 INFO SparkEnv:54 - Registering MapOutputTracker
432018-06-14 13:58:47 INFO SparkEnv:54 - Registering BlockManagerMaster
442018-06-14 13:58:47 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
452018-06-14 13:58:47 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
462018-06-14 13:58:47 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-4040f00b-d75a-4d79-be9c-fdd17fc20bae
472018-06-14 13:58:47 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
482018-06-14 13:58:47 INFO SparkEnv:54 - Registering OutputCommitCoordinator
492018-06-14 13:58:47 INFO log:192 - Logging initialized @23015ms
502018-06-14 13:58:48 INFO Server:346 - jetty-9.3.z-SNAPSHOT
512018-06-14 13:58:48 INFO Server:414 - Started @23287ms
522018-06-14 13:58:48 INFO AbstractConnector:278 - Started ServerConnector@230fbd5{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
532018-06-14 13:58:48 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
542018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4ac8768e{/jobs,null,AVAILABLE,@Spark}
552018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58a84a12{/jobs/json,null,AVAILABLE,@Spark}
562018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e700eba{/jobs/job,null,AVAILABLE,@Spark}
572018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7186b202{/jobs/job/json,null,AVAILABLE,@Spark}
582018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b649efa{/stages,null,AVAILABLE,@Spark}
592018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@65ef48f2{/stages/json,null,AVAILABLE,@Spark}
602018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@36068727{/stages/stage,null,AVAILABLE,@Spark}
612018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22bf9122{/stages/stage/json,null,AVAILABLE,@Spark}
622018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@208205ed{/stages/pool,null,AVAILABLE,@Spark}
632018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73afe2b7{/stages/pool/json,null,AVAILABLE,@Spark}
642018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9687f55{/storage,null,AVAILABLE,@Spark}
652018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5700c9db{/storage/json,null,AVAILABLE,@Spark}
662018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@671d03bb{/storage/rdd,null,AVAILABLE,@Spark}
672018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6babffb5{/storage/rdd/json,null,AVAILABLE,@Spark}
682018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2173a742{/environment,null,AVAILABLE,@Spark}
692018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@706ceca6{/environment/json,null,AVAILABLE,@Spark}
702018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f6329cb{/executors,null,AVAILABLE,@Spark}
712018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b8137c5{/executors/json,null,AVAILABLE,@Spark}
722018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77f4c040{/executors/threadDump,null,AVAILABLE,@Spark}
732018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@606a1bc4{/executors/threadDump/json,null,AVAILABLE,@Spark}
742018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6a15b73{/static,null,AVAILABLE,@Spark}
752018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@773014d3{/,null,AVAILABLE,@Spark}
762018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7fedb795{/api,null,AVAILABLE,@Spark}
772018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@34451ed8{/jobs/job/kill,null,AVAILABLE,@Spark}
782018-06-14 13:58:48 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c1050f2{/stages/stage/kill,null,AVAILABLE,@Spark}
792018-06-14 13:58:48 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
802018-06-14 13:58:48 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://192.168.189.1:7077...
812018-06-14 13:58:49 INFO TransportClientFactory:267 - Successfully created connection to /192.168.189.1:7077 after 183 ms (0 ms spent in bootstraps)
822018-06-14 13:58:49 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180614135849-0001
832018-06-14 13:58:49 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614135849-0001/0 on worker-20180614135333-worker2-34319 (worker2:34319) with 1 core(s)
842018-06-14 13:58:49 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614135849-0001/0 on hostPort worker2:34319 with 1 core(s), 512.0 MB RAM
852018-06-14 13:58:49 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614135849-0001/1 on worker-20180614135333-worker3-40134 (worker3:40134) with 1 core(s)
862018-06-14 13:58:49 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614135849-0001/1 on hostPort worker3:40134 with 1 core(s), 512.0 MB RAM
872018-06-14 13:58:49 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614135849-0001/2 on worker-20180614135333-worker1-35690 (worker1:35690) with 1 core(s)
882018-06-14 13:58:49 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614135849-0001/2 on hostPort worker1:35690 with 1 core(s), 512.0 MB RAM
892018-06-14 13:58:49 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49678.
902018-06-14 13:58:49 INFO NettyBlockTransferService:54 - Server created on master:49678
912018-06-14 13:58:49 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
922018-06-14 13:58:49 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614135849-0001/2 is now RUNNING
932018-06-14 13:58:49 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614135849-0001/1 is now RUNNING
942018-06-14 13:58:49 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614135849-0001/0 is now RUNNING
952018-06-14 13:58:49 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 49678, None)
962018-06-14 13:58:49 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:49678 with 413.9 MB RAM, BlockManagerId(driver, master, 49678, None)
972018-06-14 13:58:49 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 49678, None)
982018-06-14 13:58:49 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 49678, None)
992018-06-14 13:58:51 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30bbcf91{/metrics/json,null,AVAILABLE,@Spark}
1002018-06-14 13:58:51 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
1012018-06-14 13:58:52 INFO SharedState:54 - loading hive config file: file:/usr/local/spark-2.3.0-bin-hadoop2.6/conf/hive-site.xml
1022018-06-14 13:58:52 INFO SharedState:54 - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
1032018-06-14 13:58:52 INFO SharedState:54 - Warehouse path is '/user/hive/warehouse'.
1042018-06-14 13:58:52 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@256a0d95{/SQL,null,AVAILABLE,@Spark}
1052018-06-14 13:58:52 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2f3928ac{/SQL/json,null,AVAILABLE,@Spark}
1062018-06-14 13:58:52 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22ead351{/SQL/execution,null,AVAILABLE,@Spark}
1072018-06-14 13:58:52 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@68af87ad{/SQL/execution/json,null,AVAILABLE,@Spark}
1082018-06-14 13:58:52 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@11ede87f{/static/sql,null,AVAILABLE,@Spark}
1092018-06-14 13:58:52 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
1102018-06-14 13:58:52 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
1112018-06-14 13:58:52 INFO HiveMetaStore:746 - 0: get_database: default
1122018-06-14 13:58:52 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
1132018-06-14 13:58:55 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.4:48078) with ID 1
1142018-06-14 13:58:55 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker3:52725 with 117.0 MB RAM, BlockManagerId(1, worker3, 52725, None)
1152018-06-14 13:59:00 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
116spark-sql> 2018-06-14 13:59:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.2:47331) with ID 2
1172018-06-14 13:59:11 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker1:45606 with 117.0 MB RAM, BlockManagerId(2, worker1, 45606, None)
1182018-06-14 13:59:15 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.3:46036) with ID 0
1192018-06-14 13:59:15 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker2:55353 with 117.0 MB RAM, BlockManagerId(0, worker2, 55353, None)
120 |
继续报错：Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
| 1 >
2 > select * from hbase_hive_wtb_ow_operation;
32018-06-14 14:00:25 INFO HiveMetaStore:746 - 0: get_table : db=default tbl=hbase_hive_wtb_ow_operation
42018-06-14 14:00:25 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_table : db=default tbl=hbase_hive_wtb_ow_operation
52018-06-14 14:00:27 ERROR SparkSQLDriver:91 - Failed in [ select * from hbase_hive_wtb_ow_operation]
6java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
7 at org.apache.hadoop.hive.hbase.HBaseSerDe.parseColumnsMapping(HBaseSerDe.java:184)
8 at org.apache.hadoop.hive.hbase.HBaseSerDeParameters.<init>(HBaseSerDeParameters.java:73)
9 at org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:117)
10 at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:53)
11 at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:521)
12 at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:391)
13 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
14 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
15 at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)
16 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:358)
17 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:355)
18 at scala.Option.map(Option.scala:146)
19 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:355)
20 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:353)
21 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:272)
22 at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210)
23 at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209)
24 at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:255)
25 at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:353)
26 at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
27 at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
28 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
29 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
30 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
31 at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
32 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
33 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
34 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
35 at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
36 at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:669)
37 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:660)
38 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:615)
39 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:645)
40 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:638)
41 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
42 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
43 at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
44 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
45 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
46 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
47 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
48 at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
49 at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
50 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
51 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:638)
52 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:584)
53 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
54 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
55 at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
56 at scala.collection.immutable.List.foldLeft(List.scala:84)
57 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
58 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
59 at scala.collection.immutable.List.foreach(List.scala:381)
60 at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
61 at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:123)
62 at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:117)
63 at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:102)
64 at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
65 at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
66 at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
67 at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
68 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
69 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
70 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
71 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
72 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
73 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
74 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
75 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
76 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
77 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
78 at java.lang.reflect.Method.invoke(Method.java:497)
79 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
80 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
81 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
82 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
83 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
84 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
85Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes
86 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
87 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
88 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
89 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
90 ... 78 more
91java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
92 at org.apache.hadoop.hive.hbase.HBaseSerDe.parseColumnsMapping(HBaseSerDe.java:184)
93 at org.apache.hadoop.hive.hbase.HBaseSerDeParameters.<init>(HBaseSerDeParameters.java:73)
94 at org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:117)
95 at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:53)
96 at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:521)
97 at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:391)
98 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
99 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
100 at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)
101 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:358)
102 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:355)
103 at scala.Option.map(Option.scala:146)
104 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:355)
105 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:353)
106 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:272)
107 at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210)
108 at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209)
109 at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:255)
110 at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:353)
111 at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
112 at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
113 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
114 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
115 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
116 at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
117 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
118 at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
119 at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
120 at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
121 at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:669)
122 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:660)
123 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:615)
124 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:645)
125 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:638)
126 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
127 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
128 at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
129 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
130 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
131 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
132 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
133 at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
134 at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
135 at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
136 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:638)
137 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:584)
138 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
139 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
140 at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
141 at scala.collection.immutable.List.foldLeft(List.scala:84)
142 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
143 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
144 at scala.collection.immutable.List.foreach(List.scala:381)
145 at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
146 at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:123)
147 at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:117)
148 at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:102)
149 at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
150 at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
151 at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
152 at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
153 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
154 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
155 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
156 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
157 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
158 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
159 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
160 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
161 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
162 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
163 at java.lang.reflect.Method.invoke(Method.java:497)
164 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
165 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
166 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
167 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
168 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
169 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
170Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes
171 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
172 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
173 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
174 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
175 ... 78 more
176 |
将相关的hbase的jar包都加上，然后在spark集群中运行：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| 1root@master:/usr/local/hbase-1.2.0/lib# pwd
2/usr/local/hbase-1.2.0/lib
3root@master:/usr/local/hbase-1.2.0/lib# ls | grep hbase-client
4hbase-client-1.2.0.jar
5root@master:/usr/local/hbase-1.2.0/lib# ls | grep hbase-common
6hbase-common-1.2.0.jar
7hbase-common-1.2.0-tests.jar
8root@master:/usr/local/hbase-1.2.0/lib# ls | grep hbase-protocol
9hbase-protocol-1.2.0.jar
10root@master:/usr/local/hbase-1.2.0/lib# ls | grep hbase-server
11hbase-server-1.2.0.jar
12hbase-server-1.2.0-tests.jar
13root@master:/usr/local/hbase-1.2.0/lib# ls | grep htrace-core
14htrace-core-3.1.0-incubating.jar
15 |
1
2
3
4
| 1root@master:~# spark-sql --master spark://192.168.189.1:7077 --driver-class-path /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar:/usr/local/apache-hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar:/usr/local/hbase-1.2.0/lib/hbase-client-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-common-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-protocol-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-server-1.2.0.jar:/usr/local/hbase-1.2.0/lib/htrace-core-3.1.0-incubating.jar --executor-memory 512m --total-executor-cores 4
2
3
4 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
| 12018-06-14 14:06:09 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22018-06-14 14:06:13 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
32018-06-14 14:06:14 INFO HiveMetaStore:589 - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
42018-06-14 14:06:14 INFO ObjectStore:289 - ObjectStore, initialize called
52018-06-14 14:06:14 INFO Persistence:77 - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
62018-06-14 14:06:14 INFO Persistence:77 - Property datanucleus.cache.level2 unknown - will be ignored
72018-06-14 14:06:16 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
82018-06-14 14:06:16 INFO ObjectStore:370 - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
92018-06-14 14:06:18 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
102018-06-14 14:06:18 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
112018-06-14 14:06:19 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
122018-06-14 14:06:19 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
132018-06-14 14:06:19 INFO Query:77 - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
142018-06-14 14:06:19 INFO MetaStoreDirectSql:139 - Using direct SQL, underlying DB is MYSQL
152018-06-14 14:06:19 INFO ObjectStore:272 - Initialized ObjectStore
162018-06-14 14:06:20 INFO HiveMetaStore:663 - Added admin role in metastore
172018-06-14 14:06:20 INFO HiveMetaStore:672 - Added public role in metastore
182018-06-14 14:06:20 INFO HiveMetaStore:712 - No user is added in admin role, since config is empty
192018-06-14 14:06:20 INFO HiveMetaStore:746 - 0: get_all_databases
202018-06-14 14:06:20 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_all_databases
212018-06-14 14:06:20 INFO HiveMetaStore:746 - 0: get_functions: db=default pat=*
222018-06-14 14:06:20 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
232018-06-14 14:06:20 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
242018-06-14 14:06:20 INFO HiveMetaStore:746 - 0: get_functions: db=hive pat=*
252018-06-14 14:06:20 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hive pat=*
262018-06-14 14:06:20 INFO HiveMetaStore:746 - 0: get_functions: db=hivestudy pat=*
272018-06-14 14:06:20 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hivestudy pat=*
282018-06-14 14:06:20 INFO HiveMetaStore:746 - 0: get_functions: db=hrs pat=*
292018-06-14 14:06:20 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hrs pat=*
302018-06-14 14:06:22 INFO SessionState:641 - Created local directory: /tmp/c24fa8ce-c4af-46fd-a257-e5f52ab06aa3_resources
312018-06-14 14:06:22 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/c24fa8ce-c4af-46fd-a257-e5f52ab06aa3
322018-06-14 14:06:22 INFO SessionState:641 - Created local directory: /tmp/root/c24fa8ce-c4af-46fd-a257-e5f52ab06aa3
332018-06-14 14:06:22 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/c24fa8ce-c4af-46fd-a257-e5f52ab06aa3/_tmp_space.db
342018-06-14 14:06:23 INFO SparkContext:54 - Running Spark version 2.3.0
352018-06-14 14:06:23 INFO SparkContext:54 - Submitted application: SparkSQL::master
362018-06-14 14:06:23 INFO SecurityManager:54 - Changing view acls to: root
372018-06-14 14:06:23 INFO SecurityManager:54 - Changing modify acls to: root
382018-06-14 14:06:23 INFO SecurityManager:54 - Changing view acls groups to:
392018-06-14 14:06:23 INFO SecurityManager:54 - Changing modify acls groups to:
402018-06-14 14:06:23 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
412018-06-14 14:06:24 INFO Utils:54 - Successfully started service 'sparkDriver' on port 49479.
422018-06-14 14:06:24 INFO SparkEnv:54 - Registering MapOutputTracker
432018-06-14 14:06:24 INFO SparkEnv:54 - Registering BlockManagerMaster
442018-06-14 14:06:24 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
452018-06-14 14:06:24 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
462018-06-14 14:06:25 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-65298030-2240-49f0-b134-b18836a6af8b
472018-06-14 14:06:25 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
482018-06-14 14:06:25 INFO SparkEnv:54 - Registering OutputCommitCoordinator
492018-06-14 14:06:25 INFO log:192 - Logging initialized @24245ms
502018-06-14 14:06:25 INFO Server:346 - jetty-9.3.z-SNAPSHOT
512018-06-14 14:06:25 INFO Server:414 - Started @24612ms
522018-06-14 14:06:25 INFO AbstractConnector:278 - Started ServerConnector@6041203b{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
532018-06-14 14:06:25 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
542018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a33620{/jobs,null,AVAILABLE,@Spark}
552018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@208205ed{/jobs/json,null,AVAILABLE,@Spark}
562018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73afe2b7{/jobs/job,null,AVAILABLE,@Spark}
572018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5700c9db{/jobs/job/json,null,AVAILABLE,@Spark}
582018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@671d03bb{/stages,null,AVAILABLE,@Spark}
592018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6babffb5{/stages/json,null,AVAILABLE,@Spark}
602018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2173a742{/stages/stage,null,AVAILABLE,@Spark}
612018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b8137c5{/stages/stage/json,null,AVAILABLE,@Spark}
622018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77f4c040{/stages/pool,null,AVAILABLE,@Spark}
632018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@606a1bc4{/stages/pool/json,null,AVAILABLE,@Spark}
642018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6a15b73{/storage,null,AVAILABLE,@Spark}
652018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@44dc7b7d{/storage/json,null,AVAILABLE,@Spark}
662018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15fb4566{/storage/rdd,null,AVAILABLE,@Spark}
672018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@25ffd826{/storage/rdd/json,null,AVAILABLE,@Spark}
682018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29896529{/environment,null,AVAILABLE,@Spark}
692018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e29f4f6{/environment/json,null,AVAILABLE,@Spark}
702018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41b64020{/executors,null,AVAILABLE,@Spark}
712018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a538ed8{/executors/json,null,AVAILABLE,@Spark}
722018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@78910096{/executors/threadDump,null,AVAILABLE,@Spark}
732018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39dec536{/executors/threadDump/json,null,AVAILABLE,@Spark}
742018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a1a256d{/static,null,AVAILABLE,@Spark}
752018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@61bb1e4d{/,null,AVAILABLE,@Spark}
762018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c53f292{/api,null,AVAILABLE,@Spark}
772018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41f4039e{/jobs/job/kill,null,AVAILABLE,@Spark}
782018-06-14 14:06:26 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ff00507{/stages/stage/kill,null,AVAILABLE,@Spark}
792018-06-14 14:06:26 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
802018-06-14 14:06:27 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://192.168.189.1:7077...
812018-06-14 14:06:27 INFO TransportClientFactory:267 - Successfully created connection to /192.168.189.1:7077 after 56 ms (0 ms spent in bootstraps)
822018-06-14 14:06:27 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180614140627-0002
832018-06-14 14:06:27 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614140627-0002/0 on worker-20180614135333-worker2-34319 (worker2:34319) with 1 core(s)
842018-06-14 14:06:27 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44524.
852018-06-14 14:06:27 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614140627-0002/0 on hostPort worker2:34319 with 1 core(s), 512.0 MB RAM
862018-06-14 14:06:27 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614140627-0002/1 on worker-20180614135333-worker3-40134 (worker3:40134) with 1 core(s)
872018-06-14 14:06:27 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614140627-0002/1 on hostPort worker3:40134 with 1 core(s), 512.0 MB RAM
882018-06-14 14:06:27 INFO NettyBlockTransferService:54 - Server created on master:44524
892018-06-14 14:06:27 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614140627-0002/2 on worker-20180614135333-worker1-35690 (worker1:35690) with 1 core(s)
902018-06-14 14:06:27 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
912018-06-14 14:06:27 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614140627-0002/2 on hostPort worker1:35690 with 1 core(s), 512.0 MB RAM
922018-06-14 14:06:27 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 44524, None)
932018-06-14 14:06:27 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614140627-0002/0 is now RUNNING
942018-06-14 14:06:27 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614140627-0002/1 is now RUNNING
952018-06-14 14:06:27 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614140627-0002/2 is now RUNNING
962018-06-14 14:06:27 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:44524 with 413.9 MB RAM, BlockManagerId(driver, master, 44524, None)
972018-06-14 14:06:27 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 44524, None)
982018-06-14 14:06:27 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 44524, None)
992018-06-14 14:06:29 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@eed890d{/metrics/json,null,AVAILABLE,@Spark}
1002018-06-14 14:06:29 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
1012018-06-14 14:06:29 INFO SharedState:54 - loading hive config file: file:/usr/local/spark-2.3.0-bin-hadoop2.6/conf/hive-site.xml
1022018-06-14 14:06:29 INFO SharedState:54 - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
1032018-06-14 14:06:29 INFO SharedState:54 - Warehouse path is '/user/hive/warehouse'.
1042018-06-14 14:06:29 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@601eb4af{/SQL,null,AVAILABLE,@Spark}
1052018-06-14 14:06:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@11ede87f{/SQL/json,null,AVAILABLE,@Spark}
1062018-06-14 14:06:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2f9a10df{/SQL/execution,null,AVAILABLE,@Spark}
1072018-06-14 14:06:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@773c2214{/SQL/execution/json,null,AVAILABLE,@Spark}
1082018-06-14 14:06:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@626e0c86{/static/sql,null,AVAILABLE,@Spark}
1092018-06-14 14:06:30 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
1102018-06-14 14:06:30 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
1112018-06-14 14:06:30 INFO HiveMetaStore:746 - 0: get_database: default
1122018-06-14 14:06:30 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
1132018-06-14 14:06:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.4:46864) with ID 1
1142018-06-14 14:06:33 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker3:36552 with 117.0 MB RAM, BlockManagerId(1, worker3, 36552, None)
1152018-06-14 14:06:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.3:34013) with ID 0
1162018-06-14 14:06:35 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker2:60815 with 117.0 MB RAM, BlockManagerId(0, worker2, 60815, None)
1172018-06-14 14:06:36 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
118spark-sql> 2018-06-14 14:06:40 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.2:40473) with ID 2
1192018-06-14 14:06:40 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker1:60898 with 117.0 MB RAM, BlockManagerId(2, worker1, 60898, None)
120
121 |
继续报错：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
| 12018-06-14 14:07:45 INFO ClientCnxn:975 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
22018-06-14 14:07:45 INFO ClientCnxn:852 - Socket connection established to localhost/127.0.0.1:2181, initiating session
32018-06-14 14:07:45 INFO ClientCnxn:1235 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x163fcd5cd600003, negotiated timeout = 40000
42018-06-14 14:07:46 INFO RegionSizeCalculator:91 - Calculating region sizes for table "db_res:wtb_ow_operation".
52018-06-14 14:08:34 ERROR SparkSQLDriver:91 - Failed in [ select * from hbase_hive_wtb_ow_operation]
6org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
7Thu Jun 14 14:08:34 CST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68735: row 'db_res:wtb_ow_operation,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=master,16020,1528955336185, seqNum=0
8
9 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
10 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207)
11 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
12 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
13 at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
14 at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
15 at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
16 at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
17 at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:802)
18 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
19 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
20 at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
21 at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:88)
22 at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
23 at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
24 at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
25 at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:499)
26 at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:432)
27 at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
28 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
29 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
30 at scala.Option.getOrElse(Option.scala:121)
31 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
32 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
33 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
34 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
35 at scala.Option.getOrElse(Option.scala:121)
36 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
37 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
38 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
39 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
40 at scala.Option.getOrElse(Option.scala:121)
41 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
42 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
43 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
44 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
45 at scala.Option.getOrElse(Option.scala:121)
46 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
47 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
48 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
49 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
50 at scala.Option.getOrElse(Option.scala:121)
51 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
52 at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
53 at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
54 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
55 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
56 at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
57 at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
58 at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:297)
59 at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:324)
60 at org.apache.spark.sql.execution.QueryExecution.hiveResultString(QueryExecution.scala:122)
61 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:64)
62 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:64)
63 at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
64 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:63)
65 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
66 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
67 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
68 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
69 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
70 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
71 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
72 at java.lang.reflect.Method.invoke(Method.java:497)
73 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
74 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
75 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
76 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
77 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
78 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
79Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68735: row 'db_res:wtb_ow_operation,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=master,16020,1528955336185, seqNum=0
80 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
81 at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
82 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
83 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
84 at java.lang.Thread.run(Thread.java:745)
85Caused by: java.io.IOException: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
86 at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:330)
87 at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402)
88 at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:203)
89 at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)
90 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
91 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
92 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
93 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
94 ... 4 more
95Caused by: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
96 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:239)
97 at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
98 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
99 at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:394)
100 ... 10 more
101Caused by: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
102 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224)
103 ... 13 more
104Caused by: java.lang.ClassNotFoundException: com.yammer.metrics.core.Gauge
105 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
106 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
107 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
108 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
109 ... 14 more
110org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
111Thu Jun 14 14:08:34 CST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68735: row 'db_res:wtb_ow_operation,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=master,16020,1528955336185, seqNum=0
112
113 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
114 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207)
115 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
116 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
117 at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
118 at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
119 at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
120 at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
121 at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:802)
122 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
123 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
124 at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
125 at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:88)
126 at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
127 at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
128 at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
129 at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:499)
130 at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:432)
131 at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
132 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
133 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
134 at scala.Option.getOrElse(Option.scala:121)
135 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
136 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
137 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
138 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
139 at scala.Option.getOrElse(Option.scala:121)
140 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
141 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
142 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
143 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
144 at scala.Option.getOrElse(Option.scala:121)
145 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
146 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
147 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
148 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
149 at scala.Option.getOrElse(Option.scala:121)
150 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
151 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
152 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
153 at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
154 at scala.Option.getOrElse(Option.scala:121)
155 at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
156 at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
157 at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
158 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
159 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
160 at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
161 at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
162 at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:297)
163 at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:324)
164 at org.apache.spark.sql.execution.QueryExecution.hiveResultString(QueryExecution.scala:122)
165 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:64)
166 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:64)
167 at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
168 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:63)
169 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
170 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
171 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
172 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
173 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
174 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
175 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
176 at java.lang.reflect.Method.invoke(Method.java:497)
177 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
178 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
179 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
180 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
181 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
182 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
183Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68735: row 'db_res:wtb_ow_operation,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=master,16020,1528955336185, seqNum=0
184 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
185 at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
186 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
187 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
188 at java.lang.Thread.run(Thread.java:745)
189Caused by: java.io.IOException: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
190 at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:330)
191 at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402)
192 at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:203)
193 at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)
194 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
195 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
196 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
197 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
198 ... 4 more
199Caused by: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
200 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:239)
201 at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
202 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
203 at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:394)
204 ... 10 more
205Caused by: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
206 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224)
207 ... 13 more
208Caused by: java.lang.ClassNotFoundException: com.yammer.metrics.core.Gauge
209 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
210 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
211 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
212 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
213 ... 14 more
214
215spark-sql>
216 |
加上metrics-core-2.2.0.jar，继续运行
1
2
3
4
| 1root@master:~# spark-sql --master spark://192.168.189.1:7077 --driver-class-path /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar:/usr/local/apache-hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar:/usr/local/hbase-1.2.0/lib/hbase-client-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-common-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-protocol-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-server-1.2.0.jar:/usr/local/hbase-1.2.0/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hbase-1.2.0/lib/metrics-core-2.2.0.jar --executor-memory 512m --total-executor-cores 4
2
3
4 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
| 12018-06-14 14:12:06 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22018-06-14 14:12:07 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
32018-06-14 14:12:08 INFO HiveMetaStore:589 - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
42018-06-14 14:12:08 INFO ObjectStore:289 - ObjectStore, initialize called
52018-06-14 14:12:08 INFO Persistence:77 - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
62018-06-14 14:12:08 INFO Persistence:77 - Property datanucleus.cache.level2 unknown - will be ignored
72018-06-14 14:12:11 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
82018-06-14 14:12:11 INFO ObjectStore:370 - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
92018-06-14 14:12:12 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
102018-06-14 14:12:12 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
112018-06-14 14:12:13 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
122018-06-14 14:12:13 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
132018-06-14 14:12:14 INFO Query:77 - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
142018-06-14 14:12:14 INFO MetaStoreDirectSql:139 - Using direct SQL, underlying DB is MYSQL
152018-06-14 14:12:14 INFO ObjectStore:272 - Initialized ObjectStore
162018-06-14 14:12:14 INFO HiveMetaStore:663 - Added admin role in metastore
172018-06-14 14:12:14 INFO HiveMetaStore:672 - Added public role in metastore
182018-06-14 14:12:14 INFO HiveMetaStore:712 - No user is added in admin role, since config is empty
192018-06-14 14:12:14 INFO HiveMetaStore:746 - 0: get_all_databases
202018-06-14 14:12:14 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_all_databases
212018-06-14 14:12:14 INFO HiveMetaStore:746 - 0: get_functions: db=default pat=*
222018-06-14 14:12:14 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
232018-06-14 14:12:14 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
242018-06-14 14:12:14 INFO HiveMetaStore:746 - 0: get_functions: db=hive pat=*
252018-06-14 14:12:14 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hive pat=*
262018-06-14 14:12:14 INFO HiveMetaStore:746 - 0: get_functions: db=hivestudy pat=*
272018-06-14 14:12:14 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hivestudy pat=*
282018-06-14 14:12:14 INFO HiveMetaStore:746 - 0: get_functions: db=hrs pat=*
292018-06-14 14:12:14 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hrs pat=*
302018-06-14 14:12:16 INFO SessionState:641 - Created local directory: /tmp/7fc6b345-06df-495a-bbdb-8a42fefbabe5_resources
312018-06-14 14:12:16 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/7fc6b345-06df-495a-bbdb-8a42fefbabe5
322018-06-14 14:12:16 INFO SessionState:641 - Created local directory: /tmp/root/7fc6b345-06df-495a-bbdb-8a42fefbabe5
332018-06-14 14:12:16 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/7fc6b345-06df-495a-bbdb-8a42fefbabe5/_tmp_space.db
342018-06-14 14:12:17 INFO SparkContext:54 - Running Spark version 2.3.0
352018-06-14 14:12:17 INFO SparkContext:54 - Submitted application: SparkSQL::master
362018-06-14 14:12:17 INFO SecurityManager:54 - Changing view acls to: root
372018-06-14 14:12:17 INFO SecurityManager:54 - Changing modify acls to: root
382018-06-14 14:12:17 INFO SecurityManager:54 - Changing view acls groups to:
392018-06-14 14:12:17 INFO SecurityManager:54 - Changing modify acls groups to:
402018-06-14 14:12:17 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
412018-06-14 14:12:18 INFO Utils:54 - Successfully started service 'sparkDriver' on port 38605.
422018-06-14 14:12:18 INFO SparkEnv:54 - Registering MapOutputTracker
432018-06-14 14:12:18 INFO SparkEnv:54 - Registering BlockManagerMaster
442018-06-14 14:12:18 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
452018-06-14 14:12:18 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
462018-06-14 14:12:18 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-6428cebd-4b58-47c0-a96c-b36cff9d8123
472018-06-14 14:12:18 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
482018-06-14 14:12:18 INFO SparkEnv:54 - Registering OutputCommitCoordinator
492018-06-14 14:12:19 INFO log:192 - Logging initialized @21860ms
502018-06-14 14:12:19 INFO Server:346 - jetty-9.3.z-SNAPSHOT
512018-06-14 14:12:19 INFO Server:414 - Started @22168ms
522018-06-14 14:12:19 INFO AbstractConnector:278 - Started ServerConnector@5c09ddc8{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
532018-06-14 14:12:19 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
542018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a33620{/jobs,null,AVAILABLE,@Spark}
552018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@208205ed{/jobs/json,null,AVAILABLE,@Spark}
562018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73afe2b7{/jobs/job,null,AVAILABLE,@Spark}
572018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5700c9db{/jobs/job/json,null,AVAILABLE,@Spark}
582018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@671d03bb{/stages,null,AVAILABLE,@Spark}
592018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6babffb5{/stages/json,null,AVAILABLE,@Spark}
602018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2173a742{/stages/stage,null,AVAILABLE,@Spark}
612018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b8137c5{/stages/stage/json,null,AVAILABLE,@Spark}
622018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77f4c040{/stages/pool,null,AVAILABLE,@Spark}
632018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@606a1bc4{/stages/pool/json,null,AVAILABLE,@Spark}
642018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6a15b73{/storage,null,AVAILABLE,@Spark}
652018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@44dc7b7d{/storage/json,null,AVAILABLE,@Spark}
662018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15fb4566{/storage/rdd,null,AVAILABLE,@Spark}
672018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@25ffd826{/storage/rdd/json,null,AVAILABLE,@Spark}
682018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29896529{/environment,null,AVAILABLE,@Spark}
692018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e29f4f6{/environment/json,null,AVAILABLE,@Spark}
702018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41b64020{/executors,null,AVAILABLE,@Spark}
712018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a538ed8{/executors/json,null,AVAILABLE,@Spark}
722018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@78910096{/executors/threadDump,null,AVAILABLE,@Spark}
732018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39dec536{/executors/threadDump/json,null,AVAILABLE,@Spark}
742018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a1a256d{/static,null,AVAILABLE,@Spark}
752018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@61bb1e4d{/,null,AVAILABLE,@Spark}
762018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c53f292{/api,null,AVAILABLE,@Spark}
772018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41f4039e{/jobs/job/kill,null,AVAILABLE,@Spark}
782018-06-14 14:12:19 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ff00507{/stages/stage/kill,null,AVAILABLE,@Spark}
792018-06-14 14:12:19 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
802018-06-14 14:12:20 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://192.168.189.1:7077...
812018-06-14 14:12:20 INFO TransportClientFactory:267 - Successfully created connection to /192.168.189.1:7077 after 55 ms (0 ms spent in bootstraps)
822018-06-14 14:12:20 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180614141220-0003
832018-06-14 14:12:21 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614141220-0003/0 on worker-20180614135333-worker2-34319 (worker2:34319) with 1 core(s)
842018-06-14 14:12:21 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614141220-0003/0 on hostPort worker2:34319 with 1 core(s), 512.0 MB RAM
852018-06-14 14:12:21 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614141220-0003/1 on worker-20180614135333-worker3-40134 (worker3:40134) with 1 core(s)
862018-06-14 14:12:21 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614141220-0003/1 on hostPort worker3:40134 with 1 core(s), 512.0 MB RAM
872018-06-14 14:12:21 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614141220-0003/2 on worker-20180614135333-worker1-35690 (worker1:35690) with 1 core(s)
882018-06-14 14:12:21 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614141220-0003/2 on hostPort worker1:35690 with 1 core(s), 512.0 MB RAM
892018-06-14 14:12:21 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45270.
902018-06-14 14:12:21 INFO NettyBlockTransferService:54 - Server created on master:45270
912018-06-14 14:12:21 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
922018-06-14 14:12:21 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614141220-0003/0 is now RUNNING
932018-06-14 14:12:21 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614141220-0003/1 is now RUNNING
942018-06-14 14:12:21 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614141220-0003/2 is now RUNNING
952018-06-14 14:12:21 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 45270, None)
962018-06-14 14:12:21 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:45270 with 413.9 MB RAM, BlockManagerId(driver, master, 45270, None)
972018-06-14 14:12:21 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 45270, None)
982018-06-14 14:12:21 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 45270, None)
992018-06-14 14:12:21 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6097fca9{/metrics/json,null,AVAILABLE,@Spark}
1002018-06-14 14:12:21 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
1012018-06-14 14:12:22 INFO SharedState:54 - loading hive config file: file:/usr/local/spark-2.3.0-bin-hadoop2.6/conf/hive-site.xml
1022018-06-14 14:12:22 INFO SharedState:54 - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
1032018-06-14 14:12:22 INFO SharedState:54 - Warehouse path is '/user/hive/warehouse'.
1042018-06-14 14:12:22 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7675c171{/SQL,null,AVAILABLE,@Spark}
1052018-06-14 14:12:22 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@44e4cb76{/SQL/json,null,AVAILABLE,@Spark}
1062018-06-14 14:12:22 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15e1f8fe{/SQL/execution,null,AVAILABLE,@Spark}
1072018-06-14 14:12:22 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@110b7837{/SQL/execution/json,null,AVAILABLE,@Spark}
1082018-06-14 14:12:22 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2dd1086{/static/sql,null,AVAILABLE,@Spark}
1092018-06-14 14:12:22 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
1102018-06-14 14:12:22 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
1112018-06-14 14:12:22 INFO HiveMetaStore:746 - 0: get_database: default
1122018-06-14 14:12:22 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
1132018-06-14 14:12:25 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
1142018-06-14 14:12:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.4:48260) with ID 1
1152018-06-14 14:12:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.3:46519) with ID 0
1162018-06-14 14:12:27 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker3:40925 with 117.0 MB RAM, BlockManagerId(1, worker3, 40925, None)
1172018-06-14 14:12:27 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker2:51051 with 117.0 MB RAM, BlockManagerId(0, worker2, 51051, None)
1182018-06-14 14:12:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.2:42532) with ID 2
1192018-06-14 14:12:28 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker1:51809 with 117.0 MB RAM, BlockManagerId(2, worker1, 51809, None)
120spark-sql>
121 |
继续报错：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
| 1 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2Caused by: java.lang.IllegalStateException: unread block data
3 at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
4 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
5 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
6 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
7 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
8 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
9 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
10 at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
11 at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
12 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)
13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
14 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
15 at java.lang.Thread.run(Thread.java:745)
16org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, worker1, executor 2): java.lang.IllegalStateException: unread block data
17 at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
18 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
19 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
20 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
21 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
22 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
23 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
24 at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
25 at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
26 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)
27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
28 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
29 at java.lang.Thread.run(Thread.java:745)
30
31Driver stacktrace:
32 at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1599)
33 at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1587)
34 at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1586)
35 at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
36 at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
37 at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1586)
38 at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
39 at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
40 at scala.Option.foreach(Option.scala:257)
41 at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
42 at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1820)
43 at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1769)
44 at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1758)
45 at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
46 at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
47 at org.apache.spark.SparkContext.runJob(SparkContext.scala:2027)
48 at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
49 at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
50 at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
51 at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
52 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
53 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
54 at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
55 at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
56 at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:297)
57 at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:324)
58 at org.apache.spark.sql.execution.QueryExecution.hiveResultString(QueryExecution.scala:122)
59 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:64)
60 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver$$anonfun$run$1.apply(SparkSQLDriver.scala:64)
61 at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
62 at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:63)
63 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:355)
64 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
65 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:263)
66 at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
67 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
68 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
69 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
70 at java.lang.reflect.Method.invoke(Method.java:497)
71 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
72 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
73 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
74 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
75 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
76 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
77Caused by: java.lang.IllegalStateException: unread block data
78 at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
79 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
80 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
81 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
82 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
83 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
84 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
85 at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
86 at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
87 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)
88 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
89 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
90 at java.lang.Thread.run(Thread.java:745)
91
92spark-sql>
93 |
加上jar包
1
2
3
4
5
6
7
8
9
| 1root@master:/usr/local/hbase-1.2.0/lib# ls | grep compat
2hbase-hadoop2-compat-1.2.0.jar
3hbase-hadoop-compat-1.2.0.jar
4root@master:/usr/local/hbase-1.2.0/lib# ls | grep guava
5guava-12.0.1.jar
6root@master:/usr/local/hbase-1.2.0/lib# ls | grep protobuf
7protobuf-java-2.5.0.jar
8root@master:/usr/local/hbase-1.2.0/lib#
9 |
继续在spark集群上运行
1
2
3
4
| 1root@master:~# spark-sql --master spark://192.168.189.1:7077 --driver-class-path /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar:/usr/local/apache-hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar:/usr/local/hbase-1.2.0/lib/hbase-client-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-common-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-protocol-1.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-server-1.2.0.jar:/usr/local/hbase-1.2.0/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hbase-1.2.0/lib/metrics-core-2.2.0.jar:/usr/local/hbase-1.2.0/lib/hbase-hadoop2-compat-1.2.0.jar:/usr/local/hbase-1.2.0/lib/guava-12.0.1.jar:/usr/local/hbase-1.2.0/lib/protobuf-java-2.5.0.jar --executor-memory 512m --total-executor-cores 4
2
3
4 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
| 12018-06-14 14:20:13 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22018-06-14 14:20:15 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
32018-06-14 14:20:16 INFO HiveMetaStore:589 - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
42018-06-14 14:20:16 INFO ObjectStore:289 - ObjectStore, initialize called
52018-06-14 14:20:16 INFO Persistence:77 - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
62018-06-14 14:20:16 INFO Persistence:77 - Property datanucleus.cache.level2 unknown - will be ignored
72018-06-14 14:20:18 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
82018-06-14 14:20:18 INFO ObjectStore:370 - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
92018-06-14 14:20:20 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
102018-06-14 14:20:20 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
112018-06-14 14:20:21 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
122018-06-14 14:20:21 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
132018-06-14 14:20:21 INFO Query:77 - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
142018-06-14 14:20:21 INFO MetaStoreDirectSql:139 - Using direct SQL, underlying DB is MYSQL
152018-06-14 14:20:21 INFO ObjectStore:272 - Initialized ObjectStore
162018-06-14 14:20:22 INFO HiveMetaStore:663 - Added admin role in metastore
172018-06-14 14:20:22 INFO HiveMetaStore:672 - Added public role in metastore
182018-06-14 14:20:22 INFO HiveMetaStore:712 - No user is added in admin role, since config is empty
192018-06-14 14:20:22 INFO HiveMetaStore:746 - 0: get_all_databases
202018-06-14 14:20:22 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_all_databases
212018-06-14 14:20:22 INFO HiveMetaStore:746 - 0: get_functions: db=default pat=*
222018-06-14 14:20:22 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
232018-06-14 14:20:22 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
242018-06-14 14:20:22 INFO HiveMetaStore:746 - 0: get_functions: db=hive pat=*
252018-06-14 14:20:22 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hive pat=*
262018-06-14 14:20:22 INFO HiveMetaStore:746 - 0: get_functions: db=hivestudy pat=*
272018-06-14 14:20:22 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hivestudy pat=*
282018-06-14 14:20:22 INFO HiveMetaStore:746 - 0: get_functions: db=hrs pat=*
292018-06-14 14:20:22 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hrs pat=*
302018-06-14 14:20:24 INFO SessionState:641 - Created local directory: /tmp/677867b5-b0c3-4955-b8ac-d8f79b7ea29a_resources
312018-06-14 14:20:24 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/677867b5-b0c3-4955-b8ac-d8f79b7ea29a
322018-06-14 14:20:24 INFO SessionState:641 - Created local directory: /tmp/root/677867b5-b0c3-4955-b8ac-d8f79b7ea29a
332018-06-14 14:20:24 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/677867b5-b0c3-4955-b8ac-d8f79b7ea29a/_tmp_space.db
342018-06-14 14:20:24 INFO SparkContext:54 - Running Spark version 2.3.0
352018-06-14 14:20:24 INFO SparkContext:54 - Submitted application: SparkSQL::master
362018-06-14 14:20:25 INFO SecurityManager:54 - Changing view acls to: root
372018-06-14 14:20:25 INFO SecurityManager:54 - Changing modify acls to: root
382018-06-14 14:20:25 INFO SecurityManager:54 - Changing view acls groups to:
392018-06-14 14:20:25 INFO SecurityManager:54 - Changing modify acls groups to:
402018-06-14 14:20:25 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
412018-06-14 14:20:25 INFO Utils:54 - Successfully started service 'sparkDriver' on port 47983.
422018-06-14 14:20:26 INFO SparkEnv:54 - Registering MapOutputTracker
432018-06-14 14:20:26 INFO SparkEnv:54 - Registering BlockManagerMaster
442018-06-14 14:20:26 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
452018-06-14 14:20:26 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
462018-06-14 14:20:26 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-bd3f29b4-7966-4911-aefc-4172805c99c0
472018-06-14 14:20:26 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
482018-06-14 14:20:26 INFO SparkEnv:54 - Registering OutputCommitCoordinator
492018-06-14 14:20:26 INFO log:192 - Logging initialized @22824ms
502018-06-14 14:20:27 INFO Server:346 - jetty-9.3.z-SNAPSHOT
512018-06-14 14:20:27 INFO Server:414 - Started @23183ms
522018-06-14 14:20:27 INFO AbstractConnector:278 - Started ServerConnector@6b649efa{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
532018-06-14 14:20:27 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
542018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c53f292{/jobs,null,AVAILABLE,@Spark}
552018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@45117dd{/jobs/json,null,AVAILABLE,@Spark}
562018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7741771e{/jobs/job,null,AVAILABLE,@Spark}
572018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6cae2e4d{/jobs/job/json,null,AVAILABLE,@Spark}
582018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@604d23fa{/stages,null,AVAILABLE,@Spark}
592018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1cc8416a{/stages/json,null,AVAILABLE,@Spark}
602018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@331ff3ac{/stages/stage,null,AVAILABLE,@Spark}
612018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@79980d8d{/stages/stage/json,null,AVAILABLE,@Spark}
622018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@35d60381{/stages/pool,null,AVAILABLE,@Spark}
632018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@68ab6ab0{/stages/pool/json,null,AVAILABLE,@Spark}
642018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@61ffd148{/storage,null,AVAILABLE,@Spark}
652018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58324c9f{/storage/json,null,AVAILABLE,@Spark}
662018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5d7d8613{/storage/rdd,null,AVAILABLE,@Spark}
672018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6eb089e6{/storage/rdd/json,null,AVAILABLE,@Spark}
682018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@715a70e9{/environment,null,AVAILABLE,@Spark}
692018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3bc69ce9{/environment/json,null,AVAILABLE,@Spark}
702018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@abad89c{/executors,null,AVAILABLE,@Spark}
712018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@602f8f94{/executors/json,null,AVAILABLE,@Spark}
722018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42507640{/executors/threadDump,null,AVAILABLE,@Spark}
732018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4dfe8b37{/executors/threadDump/json,null,AVAILABLE,@Spark}
742018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@11dcd42c{/static,null,AVAILABLE,@Spark}
752018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@11c713b7{/,null,AVAILABLE,@Spark}
762018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2251b3bc{/api,null,AVAILABLE,@Spark}
772018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3ab35b9c{/jobs/job/kill,null,AVAILABLE,@Spark}
782018-06-14 14:20:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7741d346{/stages/stage/kill,null,AVAILABLE,@Spark}
792018-06-14 14:20:27 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
802018-06-14 14:20:27 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://192.168.189.1:7077...
812018-06-14 14:20:28 INFO TransportClientFactory:267 - Successfully created connection to /192.168.189.1:7077 after 102 ms (0 ms spent in bootstraps)
822018-06-14 14:20:28 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180614142028-0004
832018-06-14 14:20:28 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54513.
842018-06-14 14:20:28 INFO NettyBlockTransferService:54 - Server created on master:54513
852018-06-14 14:20:28 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
862018-06-14 14:20:28 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614142028-0004/0 on worker-20180614135333-worker2-34319 (worker2:34319) with 1 core(s)
872018-06-14 14:20:28 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614142028-0004/0 on hostPort worker2:34319 with 1 core(s), 512.0 MB RAM
882018-06-14 14:20:28 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614142028-0004/1 on worker-20180614135333-worker3-40134 (worker3:40134) with 1 core(s)
892018-06-14 14:20:28 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614142028-0004/1 on hostPort worker3:40134 with 1 core(s), 512.0 MB RAM
902018-06-14 14:20:28 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614142028-0004/2 on worker-20180614135333-worker1-35690 (worker1:35690) with 1 core(s)
912018-06-14 14:20:28 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614142028-0004/2 on hostPort worker1:35690 with 1 core(s), 512.0 MB RAM
922018-06-14 14:20:28 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 54513, None)
932018-06-14 14:20:28 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614142028-0004/2 is now RUNNING
942018-06-14 14:20:28 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614142028-0004/1 is now RUNNING
952018-06-14 14:20:28 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614142028-0004/0 is now RUNNING
962018-06-14 14:20:28 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:54513 with 413.9 MB RAM, BlockManagerId(driver, master, 54513, None)
972018-06-14 14:20:28 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 54513, None)
982018-06-14 14:20:28 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 54513, None)
992018-06-14 14:20:29 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6401188a{/metrics/json,null,AVAILABLE,@Spark}
1002018-06-14 14:20:29 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
1012018-06-14 14:20:29 INFO SharedState:54 - loading hive config file: file:/usr/local/spark-2.3.0-bin-hadoop2.6/conf/hive-site.xml
1022018-06-14 14:20:30 INFO SharedState:54 - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
1032018-06-14 14:20:30 INFO SharedState:54 - Warehouse path is '/user/hive/warehouse'.
1042018-06-14 14:20:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@24876a7e{/SQL,null,AVAILABLE,@Spark}
1052018-06-14 14:20:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1625789b{/SQL/json,null,AVAILABLE,@Spark}
1062018-06-14 14:20:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2eda2062{/SQL/execution,null,AVAILABLE,@Spark}
1072018-06-14 14:20:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a9ec80e{/SQL/execution/json,null,AVAILABLE,@Spark}
1082018-06-14 14:20:30 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@14b31e37{/static/sql,null,AVAILABLE,@Spark}
1092018-06-14 14:20:30 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
1102018-06-14 14:20:30 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
1112018-06-14 14:20:30 INFO HiveMetaStore:746 - 0: get_database: default
1122018-06-14 14:20:30 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
1132018-06-14 14:20:33 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
1142018-06-14 14:20:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.2:44278) with ID 2
1152018-06-14 14:20:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.4:37646) with ID 1
1162018-06-14 14:20:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.3:45076) with ID 0
1172018-06-14 14:20:35 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker1:36965 with 117.0 MB RAM, BlockManagerId(2, worker1, 36965, None)
1182018-06-14 14:20:35 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker3:44100 with 117.0 MB RAM, BlockManagerId(1, worker3, 44100, None)
1192018-06-14 14:20:35 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker2:40910 with 117.0 MB RAM, BlockManagerId(0, worker2, 40910, None)
120spark-sql>
121 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| 1 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
2 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
3 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
4 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
5 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
6Caused by: java.lang.IllegalStateException: unread block data
7 at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
8 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
9 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
10 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
11 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
12 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
13 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
14 at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
15 at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
16 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)
17 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
18 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
19 at java.lang.Thread.run(Thread.java:745)
20 |
1
2
| 1executor也要部署jar包，重新运行
2 |
1
2
3
4
| 1root@master:~# spark-sql --master spark://192.168.189.1:7077 --driver-class-path /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar --jars /usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar,/usr/local/apache-hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar,/usr/local/hbase-1.2.0/lib/hbase-client-1.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-common-1.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-protocol-1.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-server-1.2.0.jar,/usr/local/hbase-1.2.0/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.2.0/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.2.0/lib/hbase-hadoop2-compat-1.2.0.jar,/usr/local/hbase-1.2.0/lib/guava-12.0.1.jar,/usr/local/hbase-1.2.0/lib/protobuf-java-2.5.0.jar --executor-memory 512m --total-executor-cores 4
2
3
4 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
| 12018-06-14 14:29:05 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22018-06-14 14:29:06 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
32018-06-14 14:29:06 INFO HiveMetaStore:589 - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
42018-06-14 14:29:06 INFO ObjectStore:289 - ObjectStore, initialize called
52018-06-14 14:29:06 INFO Persistence:77 - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
62018-06-14 14:29:06 INFO Persistence:77 - Property datanucleus.cache.level2 unknown - will be ignored
72018-06-14 14:29:08 WARN HiveConf:2753 - HiveConf of name hive.server2.http.endpoint does not exist
82018-06-14 14:29:08 INFO ObjectStore:370 - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
92018-06-14 14:29:09 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
102018-06-14 14:29:09 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
112018-06-14 14:29:10 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
122018-06-14 14:29:10 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
132018-06-14 14:29:10 INFO Query:77 - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
142018-06-14 14:29:10 INFO MetaStoreDirectSql:139 - Using direct SQL, underlying DB is MYSQL
152018-06-14 14:29:10 INFO ObjectStore:272 - Initialized ObjectStore
162018-06-14 14:29:11 INFO HiveMetaStore:663 - Added admin role in metastore
172018-06-14 14:29:11 INFO HiveMetaStore:672 - Added public role in metastore
182018-06-14 14:29:11 INFO HiveMetaStore:712 - No user is added in admin role, since config is empty
192018-06-14 14:29:11 INFO HiveMetaStore:746 - 0: get_all_databases
202018-06-14 14:29:11 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_all_databases
212018-06-14 14:29:11 INFO HiveMetaStore:746 - 0: get_functions: db=default pat=*
222018-06-14 14:29:11 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
232018-06-14 14:29:11 INFO Datastore:77 - The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
242018-06-14 14:29:11 INFO HiveMetaStore:746 - 0: get_functions: db=hive pat=*
252018-06-14 14:29:11 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hive pat=*
262018-06-14 14:29:11 INFO HiveMetaStore:746 - 0: get_functions: db=hivestudy pat=*
272018-06-14 14:29:11 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hivestudy pat=*
282018-06-14 14:29:11 INFO HiveMetaStore:746 - 0: get_functions: db=hrs pat=*
292018-06-14 14:29:11 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_functions: db=hrs pat=*
302018-06-14 14:29:13 INFO SessionState:641 - Created local directory: /tmp/4fbed167-ab00-414b-bc9d-3c6b99b2f413_resources
312018-06-14 14:29:13 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/4fbed167-ab00-414b-bc9d-3c6b99b2f413
322018-06-14 14:29:13 INFO SessionState:641 - Created local directory: /tmp/root/4fbed167-ab00-414b-bc9d-3c6b99b2f413
332018-06-14 14:29:13 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/4fbed167-ab00-414b-bc9d-3c6b99b2f413/_tmp_space.db
342018-06-14 14:29:13 INFO SparkContext:54 - Running Spark version 2.3.0
352018-06-14 14:29:13 INFO SparkContext:54 - Submitted application: SparkSQL::master
362018-06-14 14:29:14 INFO SecurityManager:54 - Changing view acls to: root
372018-06-14 14:29:14 INFO SecurityManager:54 - Changing modify acls to: root
382018-06-14 14:29:14 INFO SecurityManager:54 - Changing view acls groups to:
392018-06-14 14:29:14 INFO SecurityManager:54 - Changing modify acls groups to:
402018-06-14 14:29:14 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
412018-06-14 14:29:15 INFO Utils:54 - Successfully started service 'sparkDriver' on port 49272.
422018-06-14 14:29:15 INFO SparkEnv:54 - Registering MapOutputTracker
432018-06-14 14:29:15 INFO SparkEnv:54 - Registering BlockManagerMaster
442018-06-14 14:29:15 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
452018-06-14 14:29:15 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
462018-06-14 14:29:15 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-73bbdb0f-9938-427d-8c50-16e5ebf79421
472018-06-14 14:29:15 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
482018-06-14 14:29:15 INFO SparkEnv:54 - Registering OutputCommitCoordinator
492018-06-14 14:29:15 INFO log:192 - Logging initialized @12155ms
502018-06-14 14:29:16 INFO Server:346 - jetty-9.3.z-SNAPSHOT
512018-06-14 14:29:16 INFO Server:414 - Started @12475ms
522018-06-14 14:29:16 INFO AbstractConnector:278 - Started ServerConnector@6a7031c8{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
532018-06-14 14:29:16 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
542018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2785db06{/jobs,null,AVAILABLE,@Spark}
552018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7fda2001{/jobs/json,null,AVAILABLE,@Spark}
562018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@162c1dfb{/jobs/job,null,AVAILABLE,@Spark}
572018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@799f916e{/jobs/job/json,null,AVAILABLE,@Spark}
582018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@576b7c74{/stages,null,AVAILABLE,@Spark}
592018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@21de60a7{/stages/json,null,AVAILABLE,@Spark}
602018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73894c5a{/stages/stage,null,AVAILABLE,@Spark}
612018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@28157173{/stages/stage/json,null,AVAILABLE,@Spark}
622018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@11c713b7{/stages/pool,null,AVAILABLE,@Spark}
632018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2251b3bc{/stages/pool/json,null,AVAILABLE,@Spark}
642018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3fbe503c{/storage,null,AVAILABLE,@Spark}
652018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3c232051{/storage/json,null,AVAILABLE,@Spark}
662018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15d0d6c9{/storage/rdd,null,AVAILABLE,@Spark}
672018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3ab35b9c{/storage/rdd/json,null,AVAILABLE,@Spark}
682018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7741d346{/environment,null,AVAILABLE,@Spark}
692018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52454457{/environment/json,null,AVAILABLE,@Spark}
702018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4130a648{/executors,null,AVAILABLE,@Spark}
712018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@61ff6a49{/executors/json,null,AVAILABLE,@Spark}
722018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@18dd5ed3{/executors/threadDump,null,AVAILABLE,@Spark}
732018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6042d663{/executors/threadDump/json,null,AVAILABLE,@Spark}
742018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@24043ec5{/static,null,AVAILABLE,@Spark}
752018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@da4cf09{/,null,AVAILABLE,@Spark}
762018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1980a3f{/api,null,AVAILABLE,@Spark}
772018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39941489{/jobs/job/kill,null,AVAILABLE,@Spark}
782018-06-14 14:29:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6f5d0190{/stages/stage/kill,null,AVAILABLE,@Spark}
792018-06-14 14:29:16 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
802018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar at spark://master:49272/jars/mysql-connector-java-5.1.13-bin.jar with timestamp 1528957756445
812018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/apache-hive-1.2.1/lib/hive-hbase-handler-1.2.1.jar at spark://master:49272/jars/hive-hbase-handler-1.2.1.jar with timestamp 1528957756447
822018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/hbase-client-1.2.0.jar at spark://master:49272/jars/hbase-client-1.2.0.jar with timestamp 1528957756448
832018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/hbase-common-1.2.0.jar at spark://master:49272/jars/hbase-common-1.2.0.jar with timestamp 1528957756449
842018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/hbase-protocol-1.2.0.jar at spark://master:49272/jars/hbase-protocol-1.2.0.jar with timestamp 1528957756451
852018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/hbase-server-1.2.0.jar at spark://master:49272/jars/hbase-server-1.2.0.jar with timestamp 1528957756452
862018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/htrace-core-3.1.0-incubating.jar at spark://master:49272/jars/htrace-core-3.1.0-incubating.jar with timestamp 1528957756453
872018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/metrics-core-2.2.0.jar at spark://master:49272/jars/metrics-core-2.2.0.jar with timestamp 1528957756455
882018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/hbase-hadoop2-compat-1.2.0.jar at spark://master:49272/jars/hbase-hadoop2-compat-1.2.0.jar with timestamp 1528957756456
892018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/guava-12.0.1.jar at spark://master:49272/jars/guava-12.0.1.jar with timestamp 1528957756458
902018-06-14 14:29:16 INFO SparkContext:54 - Added JAR file:///usr/local/hbase-1.2.0/lib/protobuf-java-2.5.0.jar at spark://master:49272/jars/protobuf-java-2.5.0.jar with timestamp 1528957756460
912018-06-14 14:29:16 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://192.168.189.1:7077...
922018-06-14 14:29:16 INFO TransportClientFactory:267 - Successfully created connection to /192.168.189.1:7077 after 55 ms (0 ms spent in bootstraps)
932018-06-14 14:29:17 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180614142917-0005
942018-06-14 14:29:17 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614142917-0005/0 on worker-20180614135333-worker2-34319 (worker2:34319) with 1 core(s)
952018-06-14 14:29:17 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614142917-0005/0 on hostPort worker2:34319 with 1 core(s), 512.0 MB RAM
962018-06-14 14:29:17 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614142917-0005/1 on worker-20180614135333-worker3-40134 (worker3:40134) with 1 core(s)
972018-06-14 14:29:17 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614142917-0005/1 on hostPort worker3:40134 with 1 core(s), 512.0 MB RAM
982018-06-14 14:29:17 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180614142917-0005/2 on worker-20180614135333-worker1-35690 (worker1:35690) with 1 core(s)
992018-06-14 14:29:17 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180614142917-0005/2 on hostPort worker1:35690 with 1 core(s), 512.0 MB RAM
1002018-06-14 14:29:17 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44472.
1012018-06-14 14:29:17 INFO NettyBlockTransferService:54 - Server created on master:44472
1022018-06-14 14:29:17 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
1032018-06-14 14:29:17 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 44472, None)
1042018-06-14 14:29:17 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614142917-0005/0 is now RUNNING
1052018-06-14 14:29:17 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614142917-0005/2 is now RUNNING
1062018-06-14 14:29:17 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180614142917-0005/1 is now RUNNING
1072018-06-14 14:29:17 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:44472 with 413.9 MB RAM, BlockManagerId(driver, master, 44472, None)
1082018-06-14 14:29:17 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 44472, None)
1092018-06-14 14:29:17 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 44472, None)
1102018-06-14 14:29:18 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a70d302{/metrics/json,null,AVAILABLE,@Spark}
1112018-06-14 14:29:18 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
1122018-06-14 14:29:18 INFO SharedState:54 - loading hive config file: file:/usr/local/spark-2.3.0-bin-hadoop2.6/conf/hive-site.xml
1132018-06-14 14:29:18 INFO SharedState:54 - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
1142018-06-14 14:29:18 INFO SharedState:54 - Warehouse path is '/user/hive/warehouse'.
1152018-06-14 14:29:18 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58278366{/SQL,null,AVAILABLE,@Spark}
1162018-06-14 14:29:18 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f5fcfe9{/SQL/json,null,AVAILABLE,@Spark}
1172018-06-14 14:29:18 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@548e43b1{/SQL/execution,null,AVAILABLE,@Spark}
1182018-06-14 14:29:18 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@67688110{/SQL/execution/json,null,AVAILABLE,@Spark}
1192018-06-14 14:29:18 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66e827a8{/static/sql,null,AVAILABLE,@Spark}
1202018-06-14 14:29:19 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
1212018-06-14 14:29:19 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
1222018-06-14 14:29:19 INFO HiveMetaStore:746 - 0: get_database: default
1232018-06-14 14:29:19 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
1242018-06-14 14:29:21 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
1252018-06-14 14:29:23 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.4:50383) with ID 1
1262018-06-14 14:29:23 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.2:47048) with ID 2
1272018-06-14 14:29:23 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.189.3:48320) with ID 0
1282018-06-14 14:29:23 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker3:37619 with 117.0 MB RAM, BlockManagerId(1, worker3, 37619, None)
1292018-06-14 14:29:23 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker1:37338 with 117.0 MB RAM, BlockManagerId(2, worker1, 37338, None)
1302018-06-14 14:29:23 INFO BlockManagerMasterEndpoint:54 - Registering block manager worker2:57591 with 117.0 MB RAM, BlockManagerId(0, worker2, 57591, None)
131spark-sql>
132 |
执行查询成功！
1
2
3
4
| 1spark-sql> use default;
2
3
4 |
1
2
3
4
5
6
7
8
9
10
| 12018-06-14 14:30:19 INFO HiveMetaStore:746 - 0: get_database: global_temp
22018-06-14 14:30:19 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: global_temp
32018-06-14 14:30:19 WARN ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException
42018-06-14 14:30:24 INFO HiveMetaStore:746 - 0: get_database: default
52018-06-14 14:30:24 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_database: default
6Time taken: 5.834 seconds
72018-06-14 14:30:25 INFO SparkSQLCLIDriver:951 - Time taken: 5.834 seconds
8
9
10 |
1
2
3
4
5
| 1spark-sql>
2 > select * from hbase_hive_wtb_ow_operation;
3
4
5 |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
| 12018-06-14 14:30:31 INFO HiveMetaStore:746 - 0: get_table : db=default tbl=hbase_hive_wtb_ow_operation
22018-06-14 14:30:31 INFO audit:371 - ugi=root ip=unknown-ip-addr cmd=get_table : db=default tbl=hbase_hive_wtb_ow_operation
32018-06-14 14:30:33 INFO ContextCleaner:54 - Cleaned accumulator 0
42018-06-14 14:30:35 INFO MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 237.4 KB, free 413.7 MB)
52018-06-14 14:30:35 INFO MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 22.7 KB, free 413.7 MB)
62018-06-14 14:30:35 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on master:44472 (size: 22.7 KB, free: 413.9 MB)
72018-06-14 14:30:35 INFO SparkContext:54 - Created broadcast 0 from
82018-06-14 14:30:36 INFO HBaseStorageHandler:385 - Configuring input job properties
92018-06-14 14:30:37 INFO RecoverableZooKeeper:120 - Process identifier=hconnection-0x6bd8c398 connecting to ZooKeeper ensemble=localhost:2181
102018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
112018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:host.name=master
122018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:java.version=1.8.0_60
132018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
142018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:java.home=/usr/local/jdk1.8.0_60/jre
152018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:java.class.path=/usr/local/apache-hive-1.2.1/lib/mysql-connector-java-5.1.13-bin.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/conf/:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-dataformat-yaml-2.6.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/snappy-0.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/libfb303-0.9.3.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/eigenbase-properties-1.1.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-pool-1.5.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hk2-locator-2.4.0-b34.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hive-beeline-1.2.1.spark2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spire_2.11-0.13.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/guice-servlet-3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/flatbuffers-1.2.0-3f79e055.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-network-common_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/leveldbjni-all-1.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/json4s-core_2.11-3.2.11.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/bonecp-0.8.0.RELEASE.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/java-xmlbuilder-1.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hive-exec-1.2.1.spark2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/zjsonpatch-0.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/metrics-jvm-3.1.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/scala-library-2.11.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-lang3-3.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-sketch_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/apacheds-i18n-2.0.0-M15.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/orc-core-1.4.1-nohive.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-core-asl-1.9.13.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jpam-1.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/okio-1.13.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/orc-mapreduce-1.4.1-nohive.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/arrow-memory-0.8.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/netty-all-4.1.17.Final.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-hive_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/javax.annotation-api-1.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/scala-reflect-2.11.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/parquet-format-2.3.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/avro-1.7.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/api-asn1-api-1.0.0-M20.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/py4j-0.10.6.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/univocity-parsers-2.5.9.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/joda-time-2.9.3.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-streaming_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/xmlenc-0.52.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/guice-3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-yarn-server-web-proxy-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-xc-1.9.13.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jaxb-api-2.2.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/xbean-asm5-shaded-4.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/stream-2.7.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/kubernetes-model-2.0.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/htrace-core-3.0.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/okhttp-3.8.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-dbcp-1.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jdo-api-3.0.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-mllib_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/parquet-common-1.8.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/chill_2.11-0.8.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/javolution-5.5.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-io-2.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-launcher_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/stringtemplate-3.2.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/parquet-encoding-1.8.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/minlog-1.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-net-2.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/httpcore-4.4.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/base64-2.3.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/kubernetes-client-3.0.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/curator-recipes-2.6.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-digester-1.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/aircompressor-0.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/core-1.1.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jodd-core-3.5.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-repl_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/JavaEWAH-0.3.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jul-to-slf4j-1.7.16.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/netty-3.9.9.Final.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-mapreduce-client-core-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-mapper-asl-1.9.13.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/xz-1.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/scalap-2.11.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-jaxrs-1.9.13.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/javassist-3.18.1-GA.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-mesos_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/kryo-shaded-3.0.3.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/httpclient-4.5.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/slf4j-api-1.7.16.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/generex-1.0.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/scala-parser-combinators_2.11-1.0.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/macro-compat_2.11-1.1.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/json4s-ast_2.11-3.2.11.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/zookeeper-3.4.6.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/metrics-json-3.1.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/gson-2.2.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-sql_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/arrow-format-0.8.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/avro-mapred-1.7.7-hadoop2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-compiler-3.0.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hk2-utils-2.4.0-b34.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-mapreduce-client-shuffle-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/curator-client-2.6.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-collections-3.2.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hppc-0.7.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/aopalliance-1.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/stax-api-1.0-2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/javax.servlet-api-3.1.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hive-metastore-1.2.1.spark2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/arrow-vector-0.8.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/osgi-resource-locator-1.0.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/datanucleus-core-3.2.10.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-yarn_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/parquet-column-1.8.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/xercesImpl-2.9.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-yarn-api-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/calcite-linq4j-1.2.0-incubating.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/lz4-java-1.4.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-graphx_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jersey-server-2.22.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-module-jaxb-annotations-2.6.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-mllib-local_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jets3t-0.9.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-module-scala_2.11-2.6.7.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-tags_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-httpclient-3.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hive-cli-1.2.1.spark2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jersey-container-servlet-2.22.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/opencsv-2.3.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/calcite-avatica-1.2.0-incubating.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jetty-util-6.1.26.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-catalyst_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-yarn-server-common-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-lang-2.6.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-codec-1.10.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-compress-1.4.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/antlr-2.7.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/metrics-core-3.1.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-math3-3.4.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-hdfs-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-auth-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/parquet-jackson-1.8.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/protobuf-java-2.5.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/stax-api-1.0.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/snakeyaml-1.15.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-databind-2.6.7.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/libthrift-0.9.3.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/breeze_2.11-0.13.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jetty-6.1.26.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jsr305-1.3.9.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/ivy-2.4.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/scala-compiler-2.11.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/javax.inject-2.4.0-b34.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-kubernetes_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/guava-14.0.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/oro-2.0.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-mapreduce-client-app-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/aopalliance-repackaged-2.4.0-b34.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-hive-thriftserver_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-mapreduce-client-common-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/antlr-runtime-3.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-unsafe_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-crypto-1.0.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/zstd-jni-1.3.2-2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-beanutils-1.7.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/javax.inject-1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-logging-1.1.3.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/apache-log4j-extras-1.2.17.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spire-macros_2.11-0.13.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jersey-media-jaxb-2.22.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/bcprov-jdk15on-1.58.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/janino-3.0.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/datanucleus-api-jdo-3.2.6.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jtransforms-2.4.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hk2-api-2.4.0-b34.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jersey-common-2.22.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-beanutils-core-1.8.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-annotations-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-core-2.6.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/calcite-core-1.2.0-incubating.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-module-paranamer-2.7.9.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/curator-framework-2.6.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-client-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/javax.ws.rs-api-2.0.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-configuration-1.6.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jackson-annotations-2.6.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-common-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/logging-interceptor-3.8.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/activation-1.1.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/mesos-1.4.0-shaded-protobuf.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/objenesis-2.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jersey-container-servlet-core-2.22.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/breeze-macros_2.11-0.13.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/antlr4-runtime-4.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/slf4j-log4j12-1.7.16.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/snappy-java-1.1.2.6.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/json4s-jackson_2.11-3.2.11.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/datanucleus-rdbms-3.2.9.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-yarn-client-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/validation-api-1.1.0.Final.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/api-util-1.0.0-M20.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/commons-cli-1.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/arpack_combined_all-0.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/ST4-4.0.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-mapreduce-client-jobclient-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jcl-over-slf4j-1.7.16.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/pyrolite-4.13.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-network-shuffle_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/super-csv-2.2.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/compress-lzf-1.0.3.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/parquet-hadoop-1.8.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/paranamer-2.8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/avro-ipc-1.7.7.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jersey-client-2.22.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jta-1.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/chill-java-0.8.4.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/RoaringBitmap-0.5.11.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/scala-xml_2.11-1.0.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jersey-guava-2.22.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hive-jdbc-1.2.1.spark2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/machinist_2.11-0.6.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-core_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/hadoop-yarn-common-2.6.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/parquet-hadoop-bundle-1.6.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/log4j-1.2.17.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/shapeless_2.11-2.3.2.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/jline-2.12.1.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/metrics-graphite-3.1.5.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/spark-kvstore_2.11-2.3.0.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/automaton-1.11-8.jar:/usr/local/spark-2.3.0-bin-hadoop2.6/jars/derby-10.12.1.1.jar:/usr/local/hadoop-2.6.0/etc/hadoop/
162018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
172018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
182018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:java.compiler=<NA>
192018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:os.name=Linux
202018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:os.arch=amd64
212018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:os.version=3.16.0-30-generic
222018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:user.name=root
232018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:user.home=/root
242018-06-14 14:30:37 INFO ZooKeeper:100 - Client environment:user.dir=/root
252018-06-14 14:30:37 INFO ZooKeeper:438 - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6bd8c3980x0, quorum=localhost:2181, baseZNode=/hbase
262018-06-14 14:30:37 INFO ClientCnxn:975 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
272018-06-14 14:30:37 INFO ClientCnxn:852 - Socket connection established to localhost/127.0.0.1:2181, initiating session
282018-06-14 14:30:37 INFO ClientCnxn:1235 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x163fcd5cd600008, negotiated timeout = 40000
292018-06-14 14:30:38 WARN TableInputFormatBase:618 - You are using an HTable instance that relies on an HBase-managed Connection. This is usually due to directly creating an HTable, which is deprecated. Instead, you should create a Connection object and then request a Table instance from it. If you don't need the Table instance for your own use, you should instead use the TableInputFormatBase.initalizeTable method directly.
302018-06-14 14:30:38 INFO TableInputFormatBase:623 - Creating an additional unmanaged connection because user provided one can't be used for administrative actions. We'll close it when we close out the table.
312018-06-14 14:30:38 INFO RecoverableZooKeeper:120 - Process identifier=hconnection-0x3cc2c4b8 connecting to ZooKeeper ensemble=localhost:2181
322018-06-14 14:30:38 INFO ZooKeeper:438 - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x3cc2c4b80x0, quorum=localhost:2181, baseZNode=/hbase
332018-06-14 14:30:38 INFO ClientCnxn:975 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
342018-06-14 14:30:38 INFO ClientCnxn:852 - Socket connection established to localhost/127.0.0.1:2181, initiating session
352018-06-14 14:30:38 INFO ClientCnxn:1235 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x163fcd5cd600009, negotiated timeout = 40000
362018-06-14 14:30:38 INFO RegionSizeCalculator:91 - Calculating region sizes for table "db_res:wtb_ow_operation".
372018-06-14 14:30:40 INFO SparkContext:54 - Starting job: processCmd at CliDriver.java:376
382018-06-14 14:30:40 INFO DAGScheduler:54 - Got job 0 (processCmd at CliDriver.java:376) with 1 output partitions
392018-06-14 14:30:40 INFO DAGScheduler:54 - Final stage: ResultStage 0 (processCmd at CliDriver.java:376)
402018-06-14 14:30:40 INFO DAGScheduler:54 - Parents of final stage: List()
412018-06-14 14:30:40 INFO DAGScheduler:54 - Missing parents: List()
422018-06-14 14:30:40 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[4] at processCmd at CliDriver.java:376), which has no missing parents
432018-06-14 14:30:41 INFO MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 15.6 KB, free 413.7 MB)
442018-06-14 14:30:41 INFO MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 8.4 KB, free 413.6 MB)
452018-06-14 14:30:41 INFO BlockManagerInfo:54 - Added broadcast_1_piece0 in memory on master:44472 (size: 8.4 KB, free: 413.9 MB)
462018-06-14 14:30:41 INFO SparkContext:54 - Created broadcast 1 from broadcast at DAGScheduler.scala:1039
472018-06-14 14:30:41 INFO DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at processCmd at CliDriver.java:376) (first 15 tasks are for partitions Vector(0))
482018-06-14 14:30:41 INFO TaskSchedulerImpl:54 - Adding task set 0.0 with 1 tasks
492018-06-14 14:30:41 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, worker1, executor 2, partition 0, NODE_LOCAL, 7965 bytes)
502018-06-14 14:30:46 INFO BlockManagerInfo:54 - Added broadcast_1_piece0 in memory on worker1:37338 (size: 8.4 KB, free: 117.0 MB)
512018-06-14 14:30:48 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on worker1:37338 (size: 22.7 KB, free: 116.9 MB)
522018-06-14 14:31:01 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 20218 ms on worker1 (executor 2) (1/1)
532018-06-14 14:31:01 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
542018-06-14 14:31:01 INFO DAGScheduler:54 - ResultStage 0 (processCmd at CliDriver.java:376) finished in 20.873 s
55
56
57 |
1
2
3
4
5
6
| 12018-06-14 14:31:01 INFO DAGScheduler:54 - Job 0 finished: processCmd at CliDriver.java:376, took 21.528504 s
210200000150470188 {"operate...time":"2017-12-20 13:22:17:17.0","operat..._time":"2017-12-30 15:22:17:17.0"}
3Time taken: 31.045 seconds, Fetched 1 row(s)
42018-06-14 14:31:02 INFO SparkSQLCLIDriver:951 - Time taken: 31.045 seconds, Fetched 1 row(s)
5spark-sql>
6 |
在spark sql中的查询：
在hbase中的查询：
在HIVE中的查询：
