spark No more replicas available for rdd异常

运行一个客户端模式spark程序出现No more replicas available for rdd异常，如下：

18-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 507), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 594), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 406), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 525), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 43), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Executor lost: 34 (epoch 34)
2018-04-16 18:01:53,750 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Trying to remove executor 34 from BlockManagerMaster.
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_180 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_183 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_184 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_181 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_182 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_188 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_185 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_186 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_178 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_299 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_179 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_194 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_195 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_192 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_193 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_198 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_199 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_196 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_189 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_53 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_54 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_55 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_56 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_57 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_50 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_51 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_52 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_43 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_44 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_45 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_46 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_47 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_49 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_40 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_75 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_76 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_79 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_71 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_72 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_74 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_64 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_65 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_66 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_67 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_69 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_60 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_63 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_2 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_17 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_18 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_19 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_5 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_6 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_7 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_8 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_9 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_10 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_11 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_12 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_14 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_16 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_602 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_603 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_600 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_601 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_606 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_607 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_604 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_0 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_608 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_1 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_609 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_39 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_31 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_32 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_33 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_34 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_35 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_36 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_37 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_38 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_29 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_20 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_21 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_22 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_23 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_24 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_25 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_510 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_511 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_514 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_515 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_513 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_518 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_519 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_516 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_517 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_521 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_400 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_401 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_404 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_525 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_526 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_405 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_523 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_524 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_403 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_408 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_529 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_409 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_406 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_527 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_407 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_610 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_614 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_612 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_615 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_616 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_500 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_503 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_504 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_502 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_507 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_508 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_509 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_97 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_98 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_99 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_561 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_562 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_441 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_560 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_312 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_554 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_434 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_431 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_311 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_432 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_316 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_93 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_437 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_558 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_94 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_559 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_317 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_556 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_95 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_96 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_557 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_315 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_436 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_439 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_318 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_319 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_86 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_87 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_89 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_451 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_572 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_330 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_331 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_573 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_210 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_452 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_571 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_450 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_323 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_202 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_445 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_324 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_566 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_203 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_200 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_321 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_564 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_322 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_81 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_82 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_327 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_569 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_206 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_449 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_328 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_83 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_84 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_567 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_205 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_326 !
2018-04-16 18:01:53,770 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_447 !
....
available for rdd_329_146 !
2018-04-16 18:01:53,794 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Removing block manager BlockManagerId(34, 10.42.123.220, 41352, None)
2018-04-16 18:01:53,795 INFO [dag-scheduler-event-loop] o.a.s.s.BlockManagerMaster [Logging.scala:54] Removed 34 successfully in removeExecutor
2018-04-16 18:01:53,795 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Shuffle files lost for executor: 34 (epoch 34)
2018-04-16 18:01:53,795 INFO [dag-scheduler-event-loop] o.a.s.s.ShuffleMapStage [Logging.scala:54] ShuffleMapStage 11 is now unavailable on executor 34 (0/1320, false)
2018-04-16 18:01:53,955 INFO [dispatcher-event-loop-11] o.a.s.d.c.StandaloneAppClient$ClientEndpoint [Logging.scala:54] Executor updated: app-20180416092531-0001/34 is now EXITED (Command exited with code 137)
2018-04-16 18:01:53,956 INFO [dispatcher-event-loop-11] o.a.s.s.c.StandaloneSchedulerBackend [Logging.scala:54] Executor app-20180416092531-0001/34 removed: Command exited with code 137
2018-04-16 18:01:53,956 INFO [dispatcher-event-loop-11] o.a.s.s.BlockManagerMaster [Logging.scala:54] Removal of executor 34 requested
2018-04-16 18:01:53,957 INFO [dispatcher-event-loop-9] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Trying to remove executor 34 from BlockManagerMaster.
2018-04-16 18:01:53,957 INFO [dispatcher-event-loop-11] o.a.s.s.c.CoarseGrainedSchedulerBackend$DriverEndpoint [Logging.scala:54] Asked to remove non-existent executor 34
2018-04-16 18:01:53,958 INFO [dispatcher-event-loop-0] o.a.s.d.c.StandaloneAppClient$ClientEndpoint [Logging.scala:54] Executor added: app-20180416092531-0001/35 on worker-20180416090258-10.42.123.220-35640 (10.42.123.220:35640) with 8 cores
2018-04-16 18:01:53,958 INFO [dispatcher-event-loop-0] o.a.s.s.c.StandaloneSchedulerBackend [Logging.scala:54] Granted executor ID app-20180416092531-0001/35 on hostPort 10.42.123.220:35640 with 8 cores, 4.0 GB RAM
2018-04-16 18:01:54,143 INFO [dispatcher-event-loop-1] o.a.s.d.c.StandaloneAppClient$ClientEndpoint [Logging.scala:54] Executor updated: app-20180416092531-0001/35 is now RUNNING
2018-04-16 18:02:41,526 INFO [dispatcher-event-loop-0] o.a.s.s.c.CoarseGrainedSchedulerBackend$DriverEndpoint [Logging.scala:54] Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.42.123.220:46230) with ID 35
2018-04-16 18:02:41,529 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 190.19 in stage 11.0 (TID 14490, 10.42.123.220, executor 35, partition 190, PROCESS_LOCAL, 213702 bytes)
2018-04-16 18:02:41,531 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 102.16 in stage 11.0 (TID 14491, 10.42.123.220, executor 35, partition 102, PROCESS_LOCAL, 209783 bytes)
2018-04-16 18:02:41,534 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 577.10 in stage 11.0 (TID 14492, 10.42.123.220, executor 35, partition 577, PROCESS_LOCAL, 259277 bytes)
2018-04-16 18:02:41,536 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 302.18 in stage 11.0 (TID 14493, 10.42.123.220, executor 35, partition 302, PROCESS_LOCAL, 260706 bytes)
2018-04-16 18:02:41,538 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 592.5 in stage 11.0 (TID 14494, 10.42.123.220, executor 35, partition 592, PROCESS_LOCAL, 256686 bytes)
2018-04-16 18:02:41,541 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 310.19 in stage 11.0 (TID 14495, 10.42.123.220, executor 35, partition 310, PROCESS_LOCAL, 261343 bytes)
2018-04-16 18:02:41,543 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 343.13 in stage 11.0 (TID 14496, 10.42.123.220, executor 35, partition 343, PROCESS_LOCAL, 257803 bytes)
2018-04-16 18:02:41,545 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 191.13 in stage 11.0 (TID 14497, 10.42.123.220, executor 35, partition 191, PROCESS_LOCAL, 211660 bytes)
2018-04-16 18:02:41,862 INFO [dispatcher-event-loop-7] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Registering block manager 10.42.123.220:35262 with 2004.6 MB RAM, BlockManagerId(35, 10.42.123.220, 35262, None)
2018-04-16 18:02:45,758 INFO [dispatcher-event-loop-3] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added broadcast_11_piece0 in memory on 10.42.123.220:35262 (size: 22.9 KB, free: 2004.6 MB)
2018-04-16 18:02:50,493 INFO [dispatcher-event-loop-10] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_310 in memory on 10.42.123.220:35262 (size: 128.1 KB, free: 2004.5 MB)
2018-04-16 18:02:50,506 INFO [dispatcher-event-loop-8] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_102 in memory on 10.42.123.220:35262 (size: 91.9 KB, free: 2004.4 MB)
2018-04-16 18:02:50,509 INFO [dispatcher-event-loop-8] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_191 in memory on 10.42.123.220:35262 (size: 95.6 KB, free: 2004.3 MB)
2018-04-16 18:02:50,513 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_592 in memory on 10.42.123.220:35262 (size: 121.8 KB, free: 2004.2 MB)
2018-04-16 18:02:50,517 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_343 in memory on 10.42.123.220:35262 (size: 124.4 KB, free: 2004.0 MB)
2018-04-16 18:02:50,517 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_190 in memory on 10.42.123.220:35262 (size: 96.0 KB, free: 2003.9 MB)
2018-04-16 18:02:50,521 INFO [dispatcher-event-loop-0] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_302 in memory on 10.42.123.220:35262 (size: 127.6 KB, free: 2003.8 MB)
2018-04-16 18:02:50,523 INFO [dispatcher-event-loop-5] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_577 in memory on 10.42.123.220:35262 (size: 123.6 KB, free: 2003.7 MB)
2018-04-16 18:02:55,474 INFO [dispatcher-event-loop-9] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 43.29 in stage 11.0 (TID 14498, 10.42.123.220, executor 35, partition 43, PROCESS_LOCAL, 213324 bytes)
2018-04-16 18:02:55,475 INFO [task-result-getter-3] o.a.s.s.TaskSetManager [Logging.scala:54] Finished task 343.13 in stage 11.0 (TID 14496) in 13934 ms on 10.42.123.220 (executor 35) (1/1320)
2018-04-16 18:02:55,480 INFO [dispatcher-event-loop-9] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 525.27 in stage 11.0 (TID 14499, 10.42.123.220, executor 35, partition 525, PROCESS_LOCAL, 258797 bytes)
2018-04-16 18:02:55,480 INFO [task-result-getter-1] o.a.s.s.TaskSetManager [Logging.scala:54] Finished task 577.10 in stage 11.0 (TID 14492) in 13949 ms on 10.42.123.220 (executor 35) (2/1320)
2018-04-16 18:02:55,485 INFO [dispatcher-event-loop-9] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 406.23 in stage 11.0 (TID 14500, 10.42.123.220, executor 35, partition 406, PROCESS_LOCAL, 259253 bytes)
2018-04-16 18:02:55,486 INFO [task-result-getter-0] o.a.s.s.TaskSetManager [Loggin

主要是driver日志中出现了
No more replicas available for rdd 、Resubmitted ShuffleMapTask（任务反复提交）、
Shuffle files lost for executor
等异常信息而且job一直不出结果。

查看spark UI
Running Applications > 某个应用 > Applcation Detail UI > stages > tasks 某个status为失败的任务

ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

原因：
这里遇到的问题主要是因为数据源数据量过大，而机器的内存无法满足需求，导致长时间执行超时断开的情况，数据无法有效进行交互计算，因此有必要增加内存

具体解决办法：spark应用宿主机上部署了多个虚拟机，关停其中空闲的虚拟机，给运行spark的虚拟机腾出内存空间