history service 没有启动导致看不到运行成功的mapreduce程序
| 分类: Hadoop |
2015-11-21 23:07:04,523 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1448025206651_0026_000002
2015-11-21 23:07:05,041 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-11-21 23:07:05,094 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2015-11-21 23:07:05,094 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 26 cluster_timestamp: 1448025206651 } attemptId: 2 } keyId: -161280683)
2015-11-21 23:07:05,929 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Attempt num: 2 is last retry: true because a commit was started.
2015-11-21 23:07:05,932 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$NoopEventHandler
2015-11-21 23:07:05,939 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2015-11-21 23:07:05,941 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2015-11-21 23:07:06,016 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://cluster1:8020]
2015-11-21 23:07:06,042 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://cluster1:8020]
2015-11-21 23:07:06,067 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://cluster1:8020]
2015-11-21 23:07:06,097 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2015-11-21 23:07:06,099 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Will not try to recover. recoveryEnabled: true recoverySupportedByCommitter: false numReduceTasks: 1 shuffleKeyValidForRecovery: true ApplicationAttemptID: 2
2015-11-21 23:07:06,123 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://cluster1:8020]
2015-11-21 23:07:06,130 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at hdfs://cluster1:8020/tmp/hadoop-yarn/staging/root/.staging/job_1448025206651_0026/job_1448025206651_0026_1.jhist
2015-11-21 23:07:06,582 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2015-11-21 23:07:06,619 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-11-21 23:07:06,675 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-11-21 23:07:06,675 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2015-11-21 23:07:06,693 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2015-11-21 23:07:06,694 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2015-11-21 23:07:06,694 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2015-11-21 23:07:06,733 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at hadoop0/172.18.32.177:8030
2015-11-21 23:07:06,810 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability:
2015-11-21 23:07:06,811 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: default
2015-11-21 23:07:06,863 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://cluster1:8020]
2015-11-21 23:07:06,873 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryCopyService: History file is at hdfs://cluster1:8020/tmp/hadoop-yarn/staging/root/.staging/job_1448025206651_0026/job_1448025206651_0026_1.jhist
2015-11-21 23:07:06,979 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1448025206651_0026, File: hdfs://cluster1:8020/tmp/hadoop-yarn/staging/root/.staging/job_1448025206651_0026/job_1448025206651_0026_2.jhist
2015-11-21 23:07:07,112 WARN [main] org.apache.hadoop.hdfs.DFSClient: DFSInputStream has been closed already
2015-11-21 23:07:07,113 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.io.IOException: Was asked to shut down.
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1521)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
2015-11-21 23:07:07,116 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
问题是 history service 没有启动 解决办法: 配置:选择(xxxxxxxxxx:
作为history sever) yarn.log-aggregation-enable true
mapreduce.jobhistory.address xxxxxxxxxx::10020
mapreduce.jobhistory.webapp.address xxxxxxxxxx:19888
sbin/mr-jobhistory-daemon.sh start
historyserver
mapreduce.jobhistory.webapp.address xxxxxxxxxx:19888
sbin/mr-jobhistory-daemon.sh
我的mapred-site.xml改为下面的配置则可以在http://hadoop0:8088/cluster/apps/FINISHED看到执行成功的mapreduce任务:
http://s3/mw690/001WbLonzy6XbpzlVuO02&690service 没有启动导致看不到运行成功的mapreduce程序" TITLE="history service 没有启动导致看不到运行成功的mapreduce程序" />
前一篇:关于MapReduce
后一篇:Oracle导入dmp文件

加载中…