hive mapreduce job 假死

hive> insert into ta.d11_match_detail_sea(roleId,game,channel,channelInfo,serverId,ip,utcTime,system,systemVersion,roleTime,accountId,androidIdUuid,immeiIdfa,gpsId,phoneModel,clientSystemVersion,zoneOffset,matchDataType,matchId,warmGameType,isWin,delay,endType,gamePlayType,averageFrameRate,maxDelay,minDelay,minPing,avgPing,maxPing,euro,star,coins,isAbroad,cap_full,grade,scoutCurrency,country,city,dt) select roleId,game,channel,channelInfo,serverId,ip,utcTime,system,systemVersion,roleTime,accountId,androidIdUuid,immeiIdfa,gpsId,phoneModel,clientSystemVersion,zoneOffset,matchDataType,matchId,warmGameType,isWin,delay,endType,gamePlayType,averageFrameRate,maxDelay,minDelay,minPing,avgPing,maxPing,euro,star,coins,isAbroad,cap_full,grade,scoutCurrency,country,city,date_format(utctime,'yyyyMMdd') as dt from ta.Double11MatchLogFileUploader_MATCH_DETAIL_20220607_db1c7157490c426ca717c1d79cdef57b_tmp;
Query ID = root_20220608152036_fade05e2-79f2-43e8-a0ea-ea9cd9bcba61
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1646986753665_27518, Tracking URL = http://bigdata-1:8088/proxy/application_1646986753665_27518/
Kill Command = /usr/local/hadoop-3.3.0/bin/mapred job  -kill job_1646986753665_27518

检查磁盘

hdfs fsck / | grep CORRUPT
root@bigdata-1:/usr/local/hadoop-3.3.0/logs# hdfs dfs -ls /data/hive/warehouse/ta.db/d11_match_detail_sea/dt=20220602
Found 5 items
-rw-r--r--   1 anonymous supergroup       4460 2022-06-06 00:03 /data/hive/warehouse/ta.db/d11_match_detail_sea/dt=20220602/000000_0
-rw-r--r--   1 anonymous supergroup       4428 2022-06-06 00:04 /data/hive/warehouse/ta.db/d11_match_detail_sea/dt=20220602/000000_0_copy_1
-rw-r--r--   1 anonymous supergroup       4490 2022-06-06 03:05 /data/hive/warehouse/ta.db/d11_match_detail_sea/dt=20220602/000000_0_copy_2
-rw-r--r--   1 anonymous supergroup       4648 2022-06-06 11:09 /data/hive/warehouse/ta.db/d11_match_detail_sea/dt=20220602/000000_0_copy_3
-rw-r--r--   1 anonymous supergroup       4552 2022-06-06 11:10 /data/hive/warehouse/ta.db/d11_match_detail_sea/dt=20220602/000000_0_copy_4

root@bigdata-1:/usr/local/hadoop-3.3.0/logs# tail -400f hadoop-hadoop-namenode-bigdata-1.log

2022-06-08 15:30:24,834 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/aopalliance-repackaged-2.5.0-b32.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,837 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291000_10550204, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/jline-2.12.jar
2022-06-08 15:30:24,840 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/jline-2.12.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,842 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291001_10550205, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/commons-crypto-1.0.0.jar
2022-06-08 15:30:24,845 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/commons-crypto-1.0.0.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291002_10550206, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/hive-classification-3.1.2.jar
2022-06-08 15:30:24,850 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/hive-classification-3.1.2.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,853 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291003_10550207, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/jetty-xml-9.3.20.v20170531.jar
2022-06-08 15:30:24,855 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/jetty-xml-9.3.20.v20170531.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,858 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291004_10550208, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/log4j-core-2.10.0.jar
2022-06-08 15:30:24,862 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/log4j-core-2.10.0.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,865 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291005_10550209, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/calcite-druid-1.16.0.jar
2022-06-08 15:30:24,868 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/libjars/calcite-druid-1.16.0.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,870 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291006_10550210, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.jar
2022-06-08 15:30:24,915 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.jar is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,923 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291007_10550211, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.split
2022-06-08 15:30:24,927 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.split is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,929 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291008_10550212, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.splitmetainfo
2022-06-08 15:30:24,932 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.splitmetainfo is closed by DFSClient_NONMAPREDUCE_1044994952_1
2022-06-08 15:30:24,940 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1084291009_10550213, replicas=172.23.15.173:9866 for /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.xml
2022-06-08 15:30:24,946 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1646986753665_27520/job.xml is closed by DFSClient_NONMAPREDUCE_1044994952_1

root@bigdata-1:/usr/local/hadoop-3.3.0/logs# tail -400f hadoop-hadoop-datanode-bigdata-1.log

2022-06-08 15:31:58,649 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22422, dest: /172.23.15.173:9866, bytes: 11590, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1, offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291224_10550428, duration(ns): 132203
2022-06-08 15:31:58,649 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291224_10550428, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:58,653 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2015265662-127.0.0.1-1611668073480:blk_1084291225_10550429 src: /172.23.15.173:22424 dest: /172.23.15.173:9866
2022-06-08 15:31:58,654 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22424, dest: /172.23.15.173:9866, bytes: 50771, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1, offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291225_10550429, duration(ns): 203974
2022-06-08 15:31:58,654 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291225_10550429, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:58,658 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2015265662-127.0.0.1-1611668073480:blk_1084291226_10550430 src: /172.23.15.173:22426 dest: /172.23.15.173:9866
2022-06-08 15:31:58,661 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22426, dest: /172.23.15.173:9866, bytes: 1597622, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1, offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291226_10550430, duration(ns): 1754887
2022-06-08 15:31:58,661 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291226_10550430, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:58,664 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2015265662-127.0.0.1-1611668073480:blk_1084291227_10550431 src: /172.23.15.173:22428 dest: /172.23.15.173:9866
2022-06-08 15:31:58,666 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22428, dest: /172.23.15.173:9866, bytes: 170621, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1, offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291227_10550431, duration(ns): 340798
2022-06-08 15:31:58,666 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291227_10550431, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:58,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2015265662-127.0.0.1-1611668073480:blk_1084291228_10550432 src: /172.23.15.173:22430 dest: /172.23.15.173:9866
2022-06-08 15:31:58,719 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22430, dest: /172.23.15.173:9866, bytes: 40623961, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1, offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291228_10550432, duration(ns): 48470966
2022-06-08 15:31:58,719 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291228_10550432, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:58,729 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2015265662-127.0.0.1-1611668073480:blk_1084291229_10550433 src: /172.23.15.173:22432 dest: /172.23.15.173:9866
2022-06-08 15:31:58,730 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22432, dest: /172.23.15.173:9866, bytes: 305, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1,offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291229_10550433, duration(ns): 131005
2022-06-08 15:31:58,730 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291229_10550433, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:58,734 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2015265662-127.0.0.1-1611668073480:blk_1084291230_10550434 src: /172.23.15.173:22434 dest: /172.23.15.173:9866
2022-06-08 15:31:58,735 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22434, dest: /172.23.15.173:9866, bytes: 24, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1, offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291230_10550434, duration(ns): 125912
2022-06-08 15:31:58,735 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291230_10550434, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:58,745 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2015265662-127.0.0.1-1611668073480:blk_1084291231_10550435 src: /172.23.15.173:22436 dest: /172.23.15.173:9866
2022-06-08 15:31:58,749 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.23.15.173:22436, dest: /172.23.15.173:9866, bytes: 418361, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1044994952_1, offset: 0, srvID: 05bdce56-367a-4b41-a11e-880a44e9a77f, blockid: BP-2015265662-127.0.0.1-1611668073480:blk_1084291231_10550435, duration(ns): 3210242
2022-06-08 15:31:58,749 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2015265662-127.0.0.1-1611668073480:blk_1084291231_10550435, type=LAST_IN_PIPELINE terminating
2022-06-08 15:31:59,237 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1084290788_10549992 replica FinalizedReplica, blk_1084290788_10549992, FINALIZED
  getNumBytes()     = 30637
  getBytesOnDisk()  = 30637
  getVisibleLength()= 30637
  getVolume()       = /data/hadoop/dfs/data
  getBlockURI()     = file:/data/hadoop/dfs/data/current/BP-2015265662-127.0.0.1-1611668073480/current/finalized/subdir0/subdir22/blk_1084290788 for deletion
2022-06-08 15:31:59,237 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-2015265662-127.0.0.1-1611668073480 blk_1084290788_10549992 URI file:/data/hadoop/dfs/data/current/BP-2015265662-127.0.0.1-1611668073480/current/finalized/subdir0/subdir22/blk_1084290788
2022-06-08 15:31:59,237 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1084290789_10549993 replica FinalizedReplica, blk_1084290789_10549993, FINALIZED
  getNumBytes()     = 40814
  getBytesOnDisk()  = 40814
  getVisibleLength()= 40814
  getVolume()       = /data/hadoop/dfs/data
  getBlockURI()     = file:/data/hadoop/dfs/data/current/BP-2015265662-127.0.0.1-1611668073480/current/finalized/subdir0/subdir22/blk_1084290789 for deletion
2022-06-08 15:31:59,237 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-2015265662-127.0.0.1-1611668073480 blk_1084290789_10549993 URI file:/data/hadoop/dfs/data/current/BP-2015265662-127.0.0.1-1611668073480/current/finalized/subdir0/subdir22/blk_1084290789

yarn application -appStates ALL -list | grep -v FINISHED | grep -v KILLED

yarn application -list |grep application_ | grep "ta.d11_" | awk '{print $1}' | xargs yarn application -kill

文档

Gateway

Spring Cloud Stream