我有这个 hadoop 程序:
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.hadoop.mapreduce.Job;
public class Question1_1 {
public static class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
for (String word : value.toString().split("\\s+")) {
context.write(new Text(word), new IntWritable(1));
}
}
}
public static class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
Job job = Job.getInstance(conf, "Question1_1");
job.setJarByClass(Question1_1.class);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
// Types of Key/Value
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
// Input & Output Files
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
我针对一个 txt 文件运行这个程序。我在日志中得到了这个:
15:09:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15:09:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15:09:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15:09:12 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
15:09:12 INFO input.FileInputFormat: Total input paths to process : 1
15:09:12 INFO mapreduce.JobSubmitter: number of splits:1
15:09:12 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
15:09:12 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
15:09:12 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15:09:12 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
15:09:12 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
15:09:12 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
15:09:12 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
15:09:12 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15:09:12 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
15:09:12 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
15:09:12 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
15:09:12 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15:09:12 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
15:09:12 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
15:09:12 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
15:09:12 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
15:09:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local958404083_0001
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/staging/hamza958404083/.staging/job_local958404083_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/staging/hamza958404083/.staging/job_local958404083_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/local/localRunner/hamza/job_local958404083_0001/job_local958404083_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
15:09:12 WARN conf.Configuration: file:/tmp/hadoop-hamza/mapred/local/localRunner/hamza/job_local958404083_0001/job_local958404083_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
15:09:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15:09:12 INFO mapreduce.Job: Running job: job_local958404083_0001
15:09:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15:09:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
15:09:12 INFO mapred.LocalJobRunner: Waiting for map tasks
15:09:12 INFO mapred.LocalJobRunner: Starting task: attempt_local958404083_0001_m_000000_0
15:09:12 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
15:09:12 INFO mapred.MapTask: Processing split: file:/home/hamza/workspace/TPIntroHadoop/flickrSample.txt:0+53568
15:09:12 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15:09:12 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15:09:12 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15:09:12 INFO mapred.MapTask: soft limit at 83886080
15:09:12 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15:09:12 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15:09:12 INFO mapred.LocalJobRunner:
15:09:12 INFO mapred.MapTask: Starting flush of map output
15:09:12 INFO mapred.MapTask: Spilling map output
15:09:12 INFO mapred.MapTask: bufstart = 0; bufend = 62647; bufvoid = 104857600
15:09:12 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26205172(104820688); length = 9225/6553600
15:09:12 INFO mapred.MapTask: Finished spill 0
15:09:12 INFO mapred.Task: Task:attempt_local958404083_0001_m_000000_0 is done. And is in the process of committing
15:09:12 INFO mapred.LocalJobRunner: map
15:09:12 INFO mapred.Task: Task 'attempt_local958404083_0001_m_000000_0' done.
15:09:12 INFO mapred.LocalJobRunner: Finishing task: attempt_local958404083_0001_m_000000_0
15:09:12 INFO mapred.LocalJobRunner: Map task executor complete.
15:09:12 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
15:09:12 INFO mapred.Merger: Merging 1 sorted segments
15:09:12 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 41944 bytes
15:09:12 INFO mapred.LocalJobRunner:
15:09:12 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
15:09:12 INFO mapred.Task: Task:attempt_local958404083_0001_r_000000_0 is done. And is in the process of committing
15:09:12 INFO mapred.LocalJobRunner:
15:09:12 INFO mapred.Task: Task attempt_local958404083_0001_r_000000_0 is allowed to commit now
15:09:12 INFO output.FileOutputCommitter: Saved output of task 'attempt_local958404083_0001_r_000000_0' to file:/home/hamza/workspace/TPIntroHadoop/sresult/_temporary/0/task_local958404083_0001_r_000000
15:09:12 INFO mapred.LocalJobRunner: reduce > reduce
15:09:12 INFO mapred.Task: Task 'attempt_local958404083_0001_r_000000_0' done.
15:09:13 INFO mapreduce.Job: Job job_local958404083_0001 running in uber mode : false
15:09:13 INFO mapreduce.Job: map 100% reduce 100%
15:09:13 INFO mapreduce.Job: Job job_local958404083_0001 completed successfully
15:09:13 INFO mapreduce.Job: Counters: 27
File System Counters
FILE: Number of bytes read=149581
FILE: Number of bytes written=496089
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=100
Map output records=2307
Map output bytes=62647
Map output materialized bytes=42089
Input split bytes=122
Combine input records=2307
Combine output records=1218
Reduce input groups=1218
Reduce shuffle bytes=0
Reduce input records=1218
Reduce output records=1218
Spilled Records=2436
Shuffled Maps =0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=460324864
File Input Format Counters
Bytes Read=53568
File Output Format Counters
Bytes Written=37479
问题:Map输出记录和Reduce输入记录之间有什么关系？ Reduce 输入组代表什么？
最佳答案
在映射器中，您使用 context.write(key, value)
在 reducer 中，特定 key 的所有值来自映射器的被组合成 Iterable<?> values
获取要写入的内容 new Path(output) , 你需要使用 context再次从 reducer
关于java - 映射输出记录和减少输入记录之间的关系是什么，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/47511729/
类classAprivatedeffooputs:fooendpublicdefbarputs:barendprivatedefzimputs:zimendprotecteddefdibputs:dibendendA的实例a=A.new测试a.foorescueputs:faila.barrescueputs:faila.zimrescueputs:faila.dibrescueputs:faila.gazrescueputs:fail测试输出failbarfailfailfail.发送测试[:foo,:bar,:zim,:dib,:gaz].each{|m|a.send(m)resc
我有一个模型:classItem项目有一个属性“商店”基于存储的值，我希望Item对象对特定方法具有不同的行为。Rails中是否有针对此的通用设计模式？如果方法中没有大的if-else语句，这是如何干净利落地完成的？ 最佳答案 通常通过Single-TableInheritance. 关于ruby-on-rails-Rails-子类化模型的设计模式是什么？，我们在StackOverflow上找到一个类似的问题： https://stackoverflow.co
我正在使用的第三方API的文档状态:"[O]urAPIonlyacceptspaddedBase64encodedstrings."什么是“填充的Base64编码字符串”以及如何在Ruby中生成它们。下面的代码是我第一次尝试创建转换为Base64的JSON格式数据。xa=Base64.encode64(a.to_json) 最佳答案 他们说的padding其实就是Base64本身的一部分。它是末尾的“=”和“==”。Base64将3个字节的数据包编码为4个编码字符。所以如果你的输入数据有长度n和n%3=1=>"=="末尾用于填充n%
我主要使用Ruby来执行此操作，但到目前为止我的攻击计划如下:使用gemsrdf、rdf-rdfa和rdf-microdata或mida来解析给定任何URI的数据。我认为最好映射到像schema.org这样的统一模式，例如使用这个yaml文件，它试图描述数据词汇表和opengraph到schema.org之间的转换:#SchemaXtoschema.orgconversion#data-vocabularyDV:name:namestreet-address:streetAddressregion:addressRegionlocality:addressLocalityphoto:i
为什么4.1%2返回0.0999999999999996？但是4.2%2==0.2。 最佳答案 参见此处:WhatEveryProgrammerShouldKnowAboutFloating-PointArithmetic实数是无限的。计算机使用的位数有限(今天是32位、64位)。因此计算机进行的浮点运算不能代表所有的实数。0.1是这些数字之一。请注意，这不是与Ruby相关的问题，而是与所有编程语言相关的问题，因为它来自计算机表示实数的方式。 关于ruby-为什么4.1%2使用Ruby返
为了将Cucumber用于命令行脚本，我按照提供的说明安装了arubagem。它在我的Gemfile中，我可以验证是否安装了正确的版本并且我已经包含了require'aruba/cucumber'在'features/env.rb'中为了确保它能正常工作，我写了以下场景:@announceScenario:Testingcucumber/arubaGivenablankslateThentheoutputfrom"ls-la"shouldcontain"drw"假设事情应该失败。它确实失败了，但失败的原因是错误的:@announceScenario:Testingcucumber/ar
我构建了两个需要相互通信和发送文件的Rails应用程序。例如，一个Rails应用程序会发送请求以查看其他应用程序数据库中的表。然后另一个应用程序将呈现该表的json并将其发回。我还希望一个应用程序将存储在其公共(public)目录中的文本文件发送到另一个应用程序的公共(public)目录。我从来没有做过这样的事情，所以我什至不知道从哪里开始。任何帮助，将不胜感激。谢谢! 最佳答案 无论Rails是什么，几乎所有Web应用程序都有您的要求，大多数现代Web应用程序都需要相互通信。但是有一个小小的理解需要你坚持下去，网站不应直接访问彼此
我正在使用puppet为ruby程序提供一组常量。我需要提供一组主机名，我的程序将对其进行迭代。在我之前使用的bash脚本中，我只是将它作为一个puppet变量hosts=>"host1,host2"我将其提供给bash脚本作为HOSTS=显然这对ruby不太适用——我需要它的格式hosts=["host1","host2"]自从phosts和putsmy_array.inspect提供输出["host1","host2"]我希望使用其中之一。不幸的是，我终其一生都无法弄清楚如何让它发挥作用。我尝试了以下各项:我发现某处他们指出我需要在函数调用前放置“function_”……这
它不等于主线程的binding，这个toplevel作用域是什么？此作用域与主线程中的binding有何不同？>ruby-e'putsTOPLEVEL_BINDING===binding'false 最佳答案 事实是，TOPLEVEL_BINDING始终引用Binding的预定义全局实例，而Kernel#binding创建的新实例>Binding每次封装当前执行上下文。在顶层，它们都包含相同的绑定(bind)，但它们不是同一个对象，您无法使用==或===测试它们的绑定(bind)相等性。putsTOPLEVEL_BINDINGput
我可以得到Infinity和NaNn=9.0/0#=>Infinityn.class#=>Floatm=0/0.0#=>NaNm.class#=>Float但是当我想直接访问Infinity或NaN时:Infinity#=>uninitializedconstantInfinity(NameError)NaN#=>uninitializedconstantNaN(NameError)什么是Infinity和NaN？它们是对象、关键字还是其他东西？ 最佳答案 您看到打印为Infinity和NaN的只是Float类的两个特殊实例的字符串