spark-submit --class ${main_class} \
--master yarn \
--deploy-mode client \
--driver-memory 8g \
--executor-memory 6g \
--num-executors 1 \
--executor-cores 34 \
${HOME_PATH}/obs_finance-1.0-SNAPSHOT-jar-with-dependencies.jar ${year_month}
注意这个34超过了我们yarn设置的最大32
不出意外就报错了
Caused by: java.io.IOException: Failed to send RPC RPC 6675935870406582041 to /9.135.247.xxx:36633: io.netty.channel.StacklessClosedChannelException
23/03/07 10:22:22 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(1,0,Map(),Set()) to AM was unsuccessful
23/03/07 10:22:22 ERROR util.Utils: Uncaught exception in thread main
org.apache.spark.SparkException: Exception thrown in awaitResult:
Caused by: java.io.IOException: Connection reset by peer
我们分析报错
rpc / Connection reset by peer/网络原因？其实不是。
Sending RequestExecutors(1,0,Map(),Set()) to AM was unsuccessful 这句话才是最主要的。
就是向AM申请executors资源的时候失败了，为啥失败，网络原因？还是资源超过上限，按照我们瞎搞的肯定就是资源不够了。
所以合理分配资源。
------------------------------------------------------------------------------------------------------------------------
