Задание выполнено успешно, но SparkSubmit не выполнен
У меня проблема с работой sparkSubmit.
Я пытаюсь запустить задание на hadoop через sparkSubmit: (Я прокомментировал весь код, чтобы быть уверенным, что он не пришел с работы. Я, конечно, также пробовал весь код, и у меня та же проблема)
public static void main( String[] args )
{
LOGGER.info("START EXTRACT");
LOGGER.info("args: {}", args);
//TODO delete after tests
LOGGER.info("END EXTRACT");
System.exit(0);
/* real code in commentary */
}
когда я запускаю работу, у меня есть это:
#10 Mar 2017 11:38:42,128 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:42 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:42 INFO Client:
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: client token: N/A
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: diagnostics: N/A
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster host: N/A
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster RPC port: -1
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: queue: default
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: start time: 1489142321100
#10 Mar 2017 11:38:42,132 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: final status: UNDEFINED
#10 Mar 2017 11:38:42,132 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: tracking URL: http:***************/proxy/application_1460635834146_0167/
#10 Mar 2017 11:38:42,132 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: user: Joe
#10 Mar 2017 11:38:43,135 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:43 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:44,138 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:44 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:45,140 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:45 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:46,144 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:46 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:47,146 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:47 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:48,148 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:48 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:49,150 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:49 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:50,153 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:50 INFO Client: Application report for application_1460635834146_0167 (state: FAILED)
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:50 INFO Client:
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: client token: N/A
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: diagnostics: Application application_1460635834146_0167 failed 2 times due to AM Container for appattempt_1460635834146_0167_000002 exited with exitCode: 0
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: For more detailed output, check application tracking page:http://**************/cluster/app/application_1460635834146_0167Then, click on links to logs of each attempt.
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: Diagnostics: Failing this attempt. Failing the application.
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster host: N/A
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster RPC port: -1
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: queue: default
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: start time: 1489142321100
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: final status: FAILED
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: tracking URL: http://*******************/cluster/app/application_1460635834146_0167
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: user: Joe
#10 Mar 2017 11:38:50,158 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: Error: application failed with exception
#10 Mar 2017 11:38:50,159 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: org.apache.spark.SparkException: Application finished with failed status
#10 Mar 2017 11:38:50,159 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
#10 Mar 2017 11:38:50,159 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.yarn.Client.main(Client.scala)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at java.lang.reflect.Method.invoke(Method.java:497)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:577)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:174)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:197)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
#10 Mar 2017 11:38:50,174 [DEBUG] (SparkServicesImpl.java:submitSparkJobs:124): End submit SparkJob. Exit value: 1
#10 Mar 2017 11:38:50,174 [DEBUG] (Main.java:main:42): END sparkTest
И когда я проверяю логи в hadoop, у меня есть это:
17/03/10 11:07:56 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/03/10 11:07:57 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1460635834146_0165_000001
17/03/10 11:07:57 INFO spark.SecurityManager: Changing view acls to: yarn,Joe
17/03/10 11:07:57 INFO spark.SecurityManager: Changing modify acls to: yarn,Joe
17/03/10 11:07:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, Joe); users with modify permissions: Set(yarn, Joe)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: START EXTRACT
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: args: telrptextract-application-context.xml
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: END EXTRACT
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1460635834146_0165
End of LogType:stderr
Я не понимаю, почему у меня есть выходной код 0 для моей работы, но код ошибки для sparkSubmit.
Я попробовал с полной работой, но безуспешно, затем с наименьшим возможным кодом в работе (как это представлено в части кода выше) Если у кого-то есть идея:) Спасибо!