加载中…hive查询异常 Java heap space
(2016-10-22 12:35:25)
标签：
hive异常javaheapspace |
最近在公司新搭建的测试环境用于测试作业，版本信息为apache
hadoop2.7.1、apache hive 1.2.0
java相关opts在下面；
dfs.block.size=128M
mapred.map.java.opts=-Xmx1536m
mapred.reduce.java.opts=-Xmx3072m
执行一个查询select * from test1.test_tbl1
where dt='20161018' and
service_order_id=63426953920898234233542561134；执行信息如下：
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0
since there's no reduce operator
Starting Job =
job_1476931422602_0263, Tracking URL =
http://*****:8088/proxy/application_1476931422602_0263/
Kill Command =
/opt/hadoop-2.7.1/bin/hadoop job -kill
job_1476931422602_0263
Hadoop job information for Stage-1:
number of mappers: 220; number of reducers: 0
2016-10-21 21:40:42,096 Stage-1 map
= 0%, reduce = 0%
2016-10-21 21:40:56,657 Stage-1 map
= 1%, reduce = 0%, Cumulative CPU 404.82
sec
2016-10-21 21:41:00,128 Stage-1 map
= 2%, reduce = 0%, Cumulative CPU 1073.7
sec
2016-10-21 21:41:01,256 Stage-1 map
= 3%, reduce = 0%, Cumulative CPU 1352.11
sec
~~~~~~~
2016-10-21 21:41:57,423 Stage-1 map
= 54%, reduce = 0%, Cumulative CPU 11017.88
sec
2016-10-21 21:41:58,539 Stage-1 map
= 55%, reduce = 0%, Cumulative CPU 11164.24
sec
2016-10-21 21:41:59,714 Stage-1 map
= 57%, reduce = 0%, Cumulative CPU 11558.41
sec
2016-10-21 21:42:01,116 Stage-1 map
= 58%, reduce = 0%, Cumulative CPU 11640.82
sec
2016-10-21 21:42:03,430 Stage-1 map
= 60%, reduce = 0%, Cumulative CPU 11911.81
sec
2016-10-21 21:42:04,579 Stage-1 map
= 61%, reduce = 0%, Cumulative CPU 12071.21
sec
2016-10-21 21:42:05,752 Stage-1 map
= 62%, reduce = 0%, Cumulative CPU 12100.39
sec
2016-10-21 21:42:06,903 Stage-1 map
= 63%, reduce = 0%, Cumulative CPU 12258.35
sec
2016-10-21 21:42:08,065 Stage-1 map
= 64%, reduce = 0%, Cumulative CPU 12460.41
sec
2016-10-21 21:42:09,183 Stage-1 map
= 65%, reduce = 0%, Cumulative CPU 12596.0
sec
2016-10-21 21:42:10,372 Stage-1 map
= 66%, reduce = 0%, Cumulative CPU 12880.38
sec
2016-10-21 21:42:11,779 Stage-1 map
= 100%, reduce = 0%, Cumulative CPU 10847.57
sec
Examining task ID:
task_1476931422602_0263_m_000074 (and more) from job
job_1476931422602_0263
Task with the most
failures(4):
-----
Task ID:
task_1476931422602_0263_m_000000
URL:
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1476931422602_0263&tipid=task_1476931422602_0263_m_000000
-----
Diagnostic Messages for this
Task:
Error: Java heap space
FAILED: Execution Error, return code
2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 220
Cumulative CPU: 10847.57 sec
HDFS Read: 35585836764 HDFS Write: 371941 FAIL
Total MapReduce CPU Time Spent: 0
days 3 hours 0 minutes 47 seconds 570 msec
作业异常结束，报java heap
space，增加mapred.map.java.opts值依然报错通过调试发现是这个参数mapred.child.java.opts使用的默认200M导致线程空间不足，将其改大一点就可以，但是官网提倡用mapred.map.java.opts和mapred.reduce.java.opts根据不同作业类型进行配置，为什么配置的未生效哪？还没来得及追踪源码，希望高手帮解答下。
喜欢
0
赠金笔