我需要一些帮助来弄清楚我的工作失败的原因。我建了一个 节点集群只是为了尝试一下。我按照示例 here .
一切似乎都在正常工作。我格式化了 namenode，并且能够通过 web 界面连接到 jobtracker、datanode 和 namenode。我能够启动和停止所有 hadoop 服务。
但是，当我尝试运行 wordcount 示例时，我得到了这个:
Error initializing attempt_201105161023_0002_m_000011_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
我在本地文件系统上创建了目录。
$ sudo mkdir /app/hadoop/tmp
$ sudo chown hadoop:hadoop /app/hadoop/tmp
我还修改了core-site.xml:
<!-- In: conf/core-site.xml -->
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
当我格式化 namenode 时，它会在本地创建子目录 和 HDFS 成功。
当我查看失败的 wordcount 输出结果时，错误
消息提示 I/O 错误，在文件中
/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/
job_201105161023_0002/jobToken
我做了一些故障排除，可以浏览到这个 jobToken 文件
本地文件系统，没问题。内容类似于 HDTS MapReduce.job 201105161023_0002
所以是不是权限问题？我让 hadoop 进程用户能够写入所有子目录并且它能够创建文件。那么还有什么问题呢？
命令行的一些更详细的输出:
hadoop@jonathan-VirtualBox:/usr/local/hadoop/hadoop-0.20.203.0$ bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount app/download app/output4
11/05/16 13:38:56 INFO input.FileInputFormat: Total input paths to process : 3
11/05/16 13:39:05 INFO mapred.JobClient: Running job: job_201105161222_0003
11/05/16 13:39:06 INFO mapred.JobClient: map 0% reduce 0%
11/05/16 13:39:17 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stdout
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stderr
11/05/16 13:39:22 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stdout
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stderr
11/05/16 13:39:23 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stdout
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stderr
11/05/16 13:39:30 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stdout
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stderr
11/05/16 13:39:33 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stdout
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stderr
11/05/16 13:39:36 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
... 5 more
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stdout
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stderr
11/05/16 13:39:39 INFO mapred.JobClient: Job complete: job_201105161222_0003
11/05/16 13:39:39 INFO mapred.JobClient: Counters: 4
11/05/16 13:39:39 INFO mapred.JobClient: Job Counters
11/05/16 13:39:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=0
11/05/16 13:39:39 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
最佳答案
检查您的/etc/hosts 配置。 你必须提到主域名和从属域名。
例如: 127.0.0.1 localhost.localdomain localhost 107.108.206.64 主控ABC-ubuntu 107.108.208.24 奴隶 XYZ-ubuntu 107.108.87.81 slave1 QWE-ubuntu
因为在 reduce 处理过程中，它搜索 XYZ-ubuntu 主机名而不是 slave。
此外，您的主机配置应该全部相同。
关于Hadoop wordcount 无法运行 - 需要帮助解码 hadoop 错误消息，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/6027325/
总的来说，我对ruby还比较陌生，我正在为我正在创建的对象编写一些rspec测试用例。许多测试用例都非常基础，我只是想确保正确填充和返回值。我想知道是否有办法使用循环结构来执行此操作。不必为我要测试的每个方法都设置一个assertEquals。例如:describeitem,"TestingtheItem"doit"willhaveanullvaluetostart"doitem=Item.new#HereIcoulddotheitem.name.shouldbe_nil#thenIcoulddoitem.category.shouldbe_nilendend但我想要一些方法来使用
当我使用Bundler时，是否需要在我的Gemfile中将其列为依赖项？毕竟，我的代码中有些地方需要它。例如，当我进行Bundler设置时:require"bundler/setup" 最佳答案 没有。您可以尝试，但首先您必须用鞋带将自己抬离地面。 关于ruby-我需要将Bundler本身添加到Gemfile中吗？，我们在StackOverflow上找到一个类似的问题： https://stackoverflow.com/questions/4758609/
我在从html页面生成PDF时遇到问题。我正在使用PDFkit。在安装它的过程中，我注意到我需要wkhtmltopdf。所以我也安装了它。我做了PDFkit的文档所说的一切......现在我在尝试加载PDF时遇到了这个错误。这里是错误:commandfailed:"/usr/local/bin/wkhtmltopdf""--margin-right""0.75in""--page-size""Letter""--margin-top""0.75in""--margin-bottom""0.75in""--encoding""UTF-8""--margin-left""0.75in""-
大约一年前，我决定确保每个包含非唯一文本的Flash通知都将从模块中的方法中获取文本。我这样做的最初原因是为了避免一遍又一遍地输入相同的字符串。如果我想更改措辞，我可以在一个地方轻松完成，而且一遍又一遍地重复同一件事而出现拼写错误的可能性也会降低。我最终得到的是这样的:moduleMessagesdefformat_error_messages(errors)errors.map{|attribute,message|"Error:#{attribute.to_s.titleize}#{message}."}enddeferror_message_could_not_find(obje
在选择我想要运行操作的频率时，唯一的选项是“每天”、“每小时”和“每10分钟”。谢谢!我想为我的Rails3.1应用程序运行调度程序。 最佳答案 这不是一个优雅的解决方案，但您可以安排它每天运行，并在实际开始工作之前检查日期是否为当月的第一天。 关于ruby-如何每月在Heroku运行一次Scheduler插件？，我们在StackOverflow上找到一个类似的问题： https://stackoverflow.com/questions/8692687/
我对最新版本的Rails有疑问。我创建了一个新应用程序(railsnewMyProject)，但我没有脚本/生成，只有脚本/rails，当我输入ruby./script/railsgeneratepluginmy_plugin"Couldnotfindgeneratorplugin.".你知道如何生成插件模板吗？没有这个命令可以创建插件吗？PS:我正在使用Rails3.2.1和ruby1.8.7[universal-darwin11.0] 最佳答案 随着Rails3.2.0的发布，插件生成器已经被移除。查看变更日志here.现在
exe应该在我打开页面时运行。异步进程需要运行。有什么方法可以在ruby中使用两个参数异步运行exe吗？我已经尝试过ruby命令-system()、exec()但它正在等待过程完成。我需要用参数启动exe，无需等待进程完成是否有任何rubygems会支持我的问题？ 最佳答案 您可以使用Process.spawn和Process.wait2:pid=Process.spawn'your.exe','--option'#Later...pid,status=Process.wait2pid您的程序将作为解释器的子进程执行。除
我尝试运行2.x应用程序。我使用rvm并为此应用程序设置其他版本的ruby:$rvmuseree-1.8.7-head我尝试运行服务器，然后出现很多错误:$script/serverNOTE:Gem.source_indexisdeprecated,useSpecification.Itwillberemovedonorafter2011-11-01.Gem.source_indexcalledfrom/Users/serg/rails_projects_terminal/work_proj/spohelp/config/../vendor/rails/railties/lib/r
我正在尝试在我的centos服务器上安装therubyracer，但遇到了麻烦。$geminstalltherubyracerBuildingnativeextensions.Thiscouldtakeawhile...ERROR:Errorinstallingtherubyracer:ERROR:Failedtobuildgemnativeextension./usr/local/rvm/rubies/ruby-1.9.3-p125/bin/rubyextconf.rbcheckingformain()in-lpthread...yescheckingforv8.h...no***e
我花了三天的时间用头撞墙，试图弄清楚为什么简单的“rake”不能通过我的规范文件。如果您遇到这种情况:任何文件夹路径中都不要有空格!。严重地。事实上，从现在开始，您命名的任何内容都没有空格。这是我的控制台输出:(在/Users/*****/Desktop/LearningRuby/learn_ruby)$rake/Users/*******/Desktop/LearningRuby/learn_ruby/00_hello/hello_spec.rb:116:in`require':cannotloadsuchfile--hello(LoadError) 最佳