Spark Submit дает ошибку в Pentaho Spoon

Question

Я новичок в Pentaho. Я использую дистрибутив MAPR. Когда я отправляю задание spark, я получаю следующую ошибку. Пожалуйста, помогите мне с этим. Я выполнил необходимую настройку для интеграции spark и pentaho. Пожалуйста, найдите Прикрепленные скриншоты работы Pentaho Spark Submit.

2017/09/12 12:41:44 - Spoon - Starting job...
2017/09/12 12:41:44 - spark_submit - Start of job execution
2017/09/12 12:41:44 - spark_submit - Starting entry [Spark Submit]
2017/09/12 12:41:44 - Spark Submit - Submitting Spark Script
2017/09/12 12:41:45 - Spark Submit - Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
2017/09/12 12:41:45 - Spark Submit - SLF4J: Class path contains multiple SLF4J bindings.
2017/09/12 12:41:45 - Spark Submit - SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017/09/12 12:41:45 - Spark Submit - SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017/09/12 12:41:45 - Spark Submit - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2017/09/12 12:41:45 - Spark Submit - SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017/09/12 12:41:45 - Spark Submit - 17/09/12 12:41:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017/09/12 12:41:46 - Spark Submit - 17/09/12 12:41:46 ERROR MapRZKRMFinderUtils: Zookeeper address not configured in Yarn configuration. Please check yarn-site.xml.
2017/09/12 12:41:46 - Spark Submit - 17/09/12 12:41:46 ERROR MapRZKRMFinderUtils: Unable to determine ResourceManager service address from Zookeeper.
2017/09/12 12:41:46 - Spark Submit - 17/09/12 12:41:46 ERROR MapRZKBasedRMFailoverProxyProvider: Unable to create proxy to the ResourceManager null
2017/09/12 12:41:46 - Spark Submit - Exception in thread "main" java.lang.RuntimeException: Unable to create proxy to the ResourceManager null
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.getProxy(MapRZKBasedRMFailoverProxyProvider.java:135)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.<init>(RetryInvocationHandler.java:195)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:304)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:298)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:58)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:95)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:73)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:193)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:152)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1154)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
2017/09/12 12:41:46 - Spark Submit -    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017/09/12 12:41:46 - Spark Submit -    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017/09/12 12:41:46 - Spark Submit -    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017/09/12 12:41:46 - Spark Submit -    at java.lang.reflect.Method.invoke(Method.java:498)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:733)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:177)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:202)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:116)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2017/09/12 12:41:46 - Spark Submit - Caused by: java.lang.RuntimeException: Zookeeper address not found from MapR Filesystem and is also not configured in Yarn configuration.
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.yarn.client.MapRZKRMFinderUtils.mapRZkBasedRMFinder(MapRZKRMFinderUtils.java:99)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.updateCurrentRMAddress(MapRZKBasedRMFailoverProxyProvider.java:64)
2017/09/12 12:41:46 - Spark Submit -    at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.getProxy(MapRZKBasedRMFailoverProxyProvider.java:131)
2017/09/12 12:41:46 - Spark Submit -    ... 21 more
2017/09/12 12:41:46 - spark_submit - Finished job entry [Spark Submit] (result=[false])
2017/09/12 12:41:46 - spark_submit - Starting entry [Spark Submit]
2017/09/12 12:41:46 - Spark Submit - Submitting Spark Script

Answer 1

По моему мнению, PDI говорит вам о проблеме в строке 6: Class path contains multiple SLF4J bindings и ссылку на подробные пояснения в 4 строки ниже: http://www.slf4j.org/codes.html.

Неоднозначным классом является StaticLoggerBinder, который может быть в opt/mapr/lib/slf4j-log4j12-1.7.12.jar или в opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar,

Удалите один из них и перезапустите.