我正在寻找基于 Hadoop Multinodes 的 Spark 使用，我对我的集群模式 pythonic 脚本有疑问。
我进入了我的 Hadoop 集群:
所以我想在 Python 中执行我的脚本以使用这个集群。我知道 Spark 可以用作独立模式，但我想使用我的节点。
这是一个非常简单的脚本，可以用来计算文本中的字数。
import sys
from pyspark import SparkContext
sc = SparkContext()
lines = sc.textFile(sys.argv[1])
words = lines.flatMap(lambda line: line.split(' '))
words_with_1 = words.map(lambda word: (word, 1))
word_counts = words_with_1.reduceByKey(lambda count1, count2: count1 + count2)
result = word_counts.collect()
for (word, count) in result:
print word.encode("utf8"), count
为了使用 Spark，我这样做:
time ./bin/spark-submit --master spark://master:7077 /home/hduser/count.py /data.txt
但是，这个命令可以在独立模式下执行 Spark 对吗？ 我如何使用我的 Hadoop 集群(例如 yarn)执行 Spark 并在我的集群上进行并行和分布式计算？
我试过了:
time ./bin/spark-submit --master yarn /home/hduser/count.py /data.txt
我遇到了问题:
2018-03-15 10:13:14 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-03-15 10:13:15 INFO SparkContext:54 - Running Spark version 2.3.0
2018-03-15 10:13:15 INFO SparkContext:54 - Submitted application: count.py
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing view acls to: hduser
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing modify acls to: hduser
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing view acls groups to:
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing modify acls groups to:
2018-03-15 10:13:15 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser)$
2018-03-15 10:13:16 INFO Utils:54 - Successfully started service 'sparkDriver' on port 40388.
2018-03-15 10:13:16 INFO SparkEnv:54 - Registering MapOutputTracker
2018-03-15 10:13:16 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-03-15 10:13:16 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-03-15 10:13:16 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-03-15 10:13:16 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-b131528e-849e-4ba7-94fe-c552572f12fc
2018-03-15 10:13:16 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-03-15 10:13:16 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-03-15 10:13:17 INFO log:192 - Logging initialized @5400ms
2018-03-15 10:13:17 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2018-03-15 10:13:17 INFO Server:414 - Started @5667ms
2018-03-15 10:13:17 INFO AbstractConnector:278 - Started ServerConnector@4f835332{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-03-15 10:13:17 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2f867b0c{/jobs,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a0105b7{/jobs/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3fd04590{/jobs/job,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2637750b{/jobs/job/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@439f0c7{/stages,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3978d915{/stages/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596dc76d{/stages/stage,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7054d173{/stages/stage/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47b526bb{/stages/pool,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7896fc75{/stages/pool/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2fd54632{/storage,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@79dcd5f2{/storage/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1732b48c{/storage/rdd,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5888874b{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5de9bebe{/environment,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@428593b4{/environment/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4011c9bc{/executors,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5cbfbc2a{/executors/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4c33f54d{/executors/threadDump,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22c5d74c{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6cd7b681{/static,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ee342f2{/,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4d68a347{/api,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e878af1{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@590aa379{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-03-15 10:13:19 INFO RMProxy:98 - Connecting to ResourceManager at master/172.30.10.64:8050
2018-03-15 10:13:20 INFO Client:54 - Requesting a new application from cluster with 3 NodeManagers
2018-03-15 10:13:20 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-03-15 10:13:20 INFO Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2018-03-15 10:13:20 INFO Client:54 - Setting up container launch context for our AM
2018-03-15 10:13:20 INFO Client:54 - Setting up the launch environment for our AM container
2018-03-15 10:13:20 INFO Client:54 - Preparing resources for our AM container
2018-03-15 10:13:24 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-03-15 10:13:29 INFO Client:54 - Uploading resource file:/tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606/__spark_libs__580552500091841387.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/__s$
2018-03-15 10:13:33 INFO Client:54 - Uploading resource file:/usr/local/spark/python/lib/pyspark.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/pyspark.zip
2018-03-15 10:13:33 INFO Client:54 - Uploading resource file:/usr/local/spark/python/lib/py4j-0.10.6-src.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/py4j-0.10.6-src.zip
2018-03-15 10:13:34 INFO Client:54 - Uploading resource file:/tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606/__spark_conf__7840630163677580304.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/__$
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing view acls to: hduser
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing modify acls to: hduser
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing view acls groups to:
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing modify acls groups to:
2018-03-15 10:13:34 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser)$
2018-03-15 10:13:34 INFO Client:54 - Submitting application application_1521023754917_0007 to ResourceManager
2018-03-15 10:13:34 INFO YarnClientImpl:251 - Submitted application application_1521023754917_0007
2018-03-15 10:13:34 INFO SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1521023754917_0007 and attemptId None
2018-03-15 10:13:35 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:35 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1521105214408
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1521023754917_0007/
user: hduser
2018-03-15 10:13:36 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:37 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:38 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:39 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:40 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:41 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:42 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:43 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:44 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:45 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:46 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:47 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:48 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:49 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:50 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:51 INFO Client:54 - Application report for application_1521023754917_0007 (state: FAILED)
2018-03-15 10:13:51 INFO Client:54 -
client token: N/A
diagnostics: Application application_1521023754917_0007 failed 2 times due to AM Container for appattempt_1521023754917_0007_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1521023754917_0007Then, click on links to logs of each attempt.
Diagnostics: Container [pid=9363,containerID=container_1521023754917_0007_02_000001] is running beyond virtual memory limits. Current usage: 147.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing cont$
Dump of the process-tree for container_1521023754917_0007_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 9369 9363 9363 9363 (java) 454 16 2250776576 37073 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/co$
|- 9363 9361 9363 9363 (bash) 0 0 12869632 742 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0$
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1521105214408
final status: FAILED
tracking URL: http://master:8088/cluster/app/application_1521023754917_0007
user: hduser
2018-03-15 10:13:51 INFO Client:54 - Deleted staging directory hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007
2018-03-15 10:13:51 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:13:51 INFO AbstractConnector:318 - Stopped Spark@4f835332{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-03-15 10:13:51 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-03-15 10:13:51 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:66 - Attempted to request executors before the AM has registered!
2018-03-15 10:13:51 INFO YarnClientSchedulerBackend:54 - Shutting down all executors
2018-03-15 10:13:51 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down
2018-03-15 10:13:51 INFO SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
2018-03-15 10:13:51 INFO YarnClientSchedulerBackend:54 - Stopped
2018-03-15 10:13:51 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-03-15 10:13:51 INFO MemoryStore:54 - MemoryStore cleared
2018-03-15 10:13:51 INFO BlockManager:54 - BlockManager stopped
2018-03-15 10:13:51 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-03-15 10:13:51 WARN MetricsSystem:66 - Stopping a MetricsSystem that is not running
2018-03-15 10:13:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-03-15 10:13:52 INFO SparkContext:54 - Successfully stopped SparkContext
Traceback (most recent call last):
File "/home/hduser/count.py", line 4, in <module>
sc = SparkContext()
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 180, in _do_init
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 270, in _initialize_context
File "/usr/local/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1428, in __call__
File "/usr/local/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:13:52 INFO ShutdownHookManager:54 - Shutdown hook called
2018-03-15 10:13:52 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606
2018-03-15 10:13:52 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-f5d31d54-e456-4fcb-bf48-9f950233ad4b
当我想将我的集群与 Spark 一起使用时，我总是遇到FAILED
终于试过了:
time ./bin/spark-submit --master yarn --deploy-mode cluster /home/hduser/count.py /data.txt
但我又遇到了一次问题。
我有什么不明白？我对大数据很陌生，所以有可能:/ 编辑:
这是我获得的:yarn application -status application_1521023754917_0007
18/03/15 10:52:07 INFO client.RMProxy: Connecting to ResourceManager at master/172.30.10.64:8050
18/03/15 10:52:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Application Report :
Application-Id : application_1521023754917_0007
Application-Name : count.py
Application-Type : SPARK
User : hduser
Queue : default
Start-Time : 1521105214408
Finish-Time : 1521105231067
Progress : 0%
State : FAILED
Final-State : FAILED
Tracking-URL : http://master:8088/cluster/app/application_1521023754917_0007
RPC Port : -1
AM Host : N/A
Aggregate Resource Allocation : 16329 MB-seconds, 15 vcore-seconds
Diagnostics : Application application_1521023754917_0007 failed 2 times due to AM Container for appattempt_1521023754917_0007_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1521023754917_0007Then, click on links to logs of each attempt.
Diagnostics: Container [pid=9363,containerID=container_1521023754917_0007_02_000001] is running beyond virtual memory limits. Current usage: 147.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1521023754917_0007_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 9369 9363 9363 9363 (java) 454 16 2250776576 37073 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/tmp -Dspark.yarn.app.container.log.dir=/usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg master:40388 --properties-file /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/__spark_conf__/__spark_conf__.properties
|- 9363 9361 9363 9363 (bash) 0 0 12869632 742 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/tmp -Dspark.yarn.app.container.log.dir=/usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 'master:40388' --properties-file /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/__spark_conf__/__spark_conf__.properties 1> /usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001/stdout 2> /usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
最佳答案
对我来说，这个 spark 提交在所有 spark 节点上运行 python:
spark-submit --master yarn
--deploy-mode cluster
--num-executors 1
--driver-memory 2g
--executor-memory 1g
--executor-cores 1
hdfs://<host>:<port>/home/hduser/count.py /data.txt
Spark 环境需要扩展: 导出 PYSPARK_PYTHON=/opt/bin/python
此外，py 文件需要位于 hdfs 上，以便集群中的所有 spark 节点都可以读取它。 spark 用户需要可以访问 py 文件。
关于hadoop - Spark : Execute python script with Spark based on Hadoop Multinode，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/49295517/
1.1.1 YARN的介绍 为克服Hadoop1.0中HDFS和MapReduce存在的各种问题⽽提出的，针对Hadoop1.0中的MapReduce在扩展性和多框架⽀持⽅⾯的不⾜，提出了全新的资源管理框架YARN. ApacheYARN（YetanotherResourceNegotiator的缩写）是Hadoop集群的资源管理系统，负责为计算程序提供服务器计算资源，相当于⼀个分布式的操作系统平台，⽽MapReduce等计算程序则相当于运⾏于操作系统之上的应⽤程序。 YARN被引⼊Hadoop2,最初是为了改善MapReduce的实现，但是因为具有⾜够的通⽤性，同样可以⽀持其他的分布式计算模
目录：一、简介二、HQL的执行流程三、索引四、索引案例五、Hive常用DDL操作六、Hive常用DML操作七、查询结果插入到表八、更新和删除操作九、查询结果写出到文件系统十、HiveCLI和Beeline命令行的基本使用十一、Hive配置一、简介Hive是一个构建在Hadoop之上的数据仓库，它可以将结构化的数据文件映射成表，并提供类SQL查询功能，用于查询的SQL语句会被转化为MapReduce作业，然后提交到Hadoop上运行。特点：简单、容易上手(提供了类似sql的查询语言hql)，使得精通sql但是不了解Java编程的人也能很好地进行大数据分析；灵活性高，可以自定义用户函数(UDF)和
一、获取当前时间1、current_date当前日期（年月日）Examples：SELECTcurrent_date;2、current_timestamp/now()当前日期（时间戳）Examples：SELECTcurrent_timestamp;二、从日期字段中提取时间1、year,month,day/dayofmonth,hour,minute,secondExamples：SELECTyear(now())；其他的日期函数以此类推month:1day:12（当月的第几天）dayofmonth：12hour,minute,second：分别对应时分秒2、dayofweek、dayofm
云计算实验中要求我们在Linux系统安装Hadoop，故来做一个简单的记录。· 注：我的操作系统环境是Ubuntu-20.04.3，安装的JDK版本为jdk1.8.0_301，安装的Hadoop版本为hadoop2.7.1。（不确定其他版本是否会出现版本兼容问题）Hadoop安装步骤如下： 一、更新apt和安装vim编辑器 二、配置本机无密码登录SSH 三、安装JAVA环境 四、下载安装Hadoop 五、伪分布式搭建一、更新apt和安装vim编辑器1、更新aptsudoapt-getupdate2、安装vim
一、设置免密登录1、系统偏好设置-----共享----勾选远程登录，所有用户2、打开终端，输入命令ssh-keygen-trsa，一直回车即可2.查看生成的公钥和私钥 cd~/.ssh ls会看到~/.ssh目录下有两个文件：①私钥：id_rsa②公钥：id_rsa.pub3.将公钥内容写入到~/.ssh/authorized_keys中 cat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys4.测试在terminal终端输入 sshlocalhost如果出现以下询问输入yes，不需要输入密码就能登录，说明配置成功Areyousureyouw
spark官方提供了两种方法实现从RDD转换到DataFrame。第一种方法是利用反射机制来推断包含特定类型对象的Schema，这种方式适用于对已知的数据结构的RDD转换； 第二种方法通过编程接口构造一个Schema，并将其应用在已知的RDD数据中。一、反射机制推断Schema实现反射机制Schema需要定义一个caseclass样例类，定义字段和属性，样例类的参数名称会被反射机制利用作为列名objectRddToDataFrameByReflect{//定义一个student样例类caseclassStudent(name:String,age:Int)defmain(args:Array[
我使用Kafka流媒体从KAFKA主题中消费。（KafkaDirect流）此主题中的数据每5分钟从另一个来源到达。现在，我需要处理每5分钟后到达的数据，并将其转换为SparkDataFrame。现在，流是数据的连续流。我的问题是，如何确定我已经完成了在Kafka主题中加载的第一组数据的阅读？（以便我可以将其转换为数据框架并开始我的工作）我知道我可以提及某个数字的批处理间隔（在JavastreamingContext中），但是即使那样，我也永远无法确定源将数据将数据推到主题的时间。欢迎任何建议。看答案如果我正确理解您的问题，您希望不创建批处理，直到阅读5分钟的所有数据。开箱即用的Spark不会提
问题：帖子的请求参数作为请求主体，而不是请求参数。我正在使用下面的此语法来调用SparkJavaWeb服务。http：//localhost：8080/cumbcustomer？custId＃4＆amp;name=fredj"SparkJava告诉我：请求IP0：0：0：0：0：0：0：0：1请求动词post请求接收到：CUSTID＃4＆amp;name=fredj（-＆gt;request.body.body（））url接收：http：//localhost：8080/cumbscustomer有什么想法为什么这些变量作为请求主体而不是请求参数的一部分出现？提前致谢，看答案利用request
博学之，审问之，慎思之，明辨之，笃行之🏂hiveonspark搭建好后，任务提交会有问题，因为通过hive会话提交的任务一直存在且不会结束（除非关掉这个hive会话），根本原因是这些任务提交到了Yarn的同一个队列中，前面的任务没有执行完毕后面的任务不会执行，所以解决办法是增加一个Yarn队列，指定任务提交的队列，这样就不会出现任务的阻塞。目录一、情景复现二、原因三、Yarn队列配置—增加队列1.情景复现：搭建好hiveonspark后，在命令行直接进入hive会话，提交任务后，在ResourceManager上jps查看进程可以看到有个进程ApplicationMaster一直存在，打开Re
目录SparkStreaming的核心是DStream一、DStream简介二．DStream编程模型三．DStream转换操作SparkStreaming的核心是DStream一、DStream简介1.Spark Streaming提供了一个高级抽象的流，即DStream(离散流)。2.DStream的内部结构是由一系列连续的RDD组成，每个RDD都是一小段由时间分隔开来的数据集。二．DStream编程模型三．DStream转换操作transform（）1．在3个节点启动zookeeper集群服务$zkServer.shstart2．启动kafka（3个节点都要）$/opt/module/k