我正在运行Pyspark作业:spark-submit--masteryarn-client--driver-memory150G--num-executors8--executor-cores4--executor-memory150Gbenchmark_script_1.pyhdfs:///tmp/data/sample150k128hdfs:///tmp/output/sample150k|tee~/output/sample150k.log工作本身非常标准。它只是抓取一些文件并对它们进行计数。:print(str(datetime.now())+"-Ingestingfiles

这是我在scala中使用toDebugString时得到的结果:scala>vala=sc.parallelize(Array(1,2,3)).distincta:org.apache.spark.rdd.RDD[Int]=MappedRDD[3]atdistinctat:12scala>a.toDebugStringres0:String=(4)MappedRDD[3]atdistinctat:12|ShuffledRDD[2]atdistinctat:12+-(4)MappedRDD[1]atdistinctat:12|ParallelCollectionRDD[0]atparal