rmr2 MapReduce всегда генерирует $val NULL

Я использую R версии 3.4.2 и Hadoop 2.6.0 . При выполнении программы mapreduce с использованием rmr2 mapreduce выполняется успешно, но когда я получаю значение из dfs, я получаю значение как NULL . Пожалуйста, найдите образец сценария ниже.

> library(rmr2)
> library(rhdfs)
Loading required package: rJava

HADOOP_CMD=/usr/local/hadoop/bin/hadoop

Be sure to run hdfs.init()
> hdfs.init()

> calc = mapreduce(input = ints,map = function(k, v) cbind(v, 2*v))


 17/11/06 01:31:41 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
packageJobJar: [/tmp/hadoop-unjar4668480277385240230/] [] /tmp/streamjob2472233183397238055.jar tmpDir=null



 17/11/06 01:31:42 INFO client.RMProxy: Connecting to ResourceManager at  /0.0.0.0:8032
   17/11/06 01:31:42 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
   17/11/06 01:31:44 INFO mapred.FileInputFormat: Total input paths to process : 1
   17/11/06 01:31:44 INFO mapreduce.JobSubmitter: number of splits:2
   17/11/06 01:31:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1509909875158_0002
   17/11/06 01:31:45 INFO impl.YarnClientImpl: Submitted application application_1509909875158_0002
   17/11/06 01:31:45 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1509909875158_0002/
   17/11/06 01:31:45 INFO mapreduce.Job: Running job: job_1509909875158_0002
   17/11/06 01:31:54 INFO mapreduce.Job: Job job_1509909875158_0002 running in uber mode : false
   17/11/06 01:31:54 INFO mapreduce.Job:  map 0% reduce 0%
   17/11/06 01:32:08 INFO mapreduce.Job:  map 50% reduce 0%
   17/11/06 01:32:09 INFO mapreduce.Job:  map 100% reduce 0%
   17/11/06 01:32:09 INFO mapreduce.Job: Job job_1509909875158_0002 completed successfully
   17/11/06 01:32:10 INFO mapreduce.Job: Counters: 30
   File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=220244
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=3588
        HDFS: Number of bytes written=244
        HDFS: Number of read operations=14
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=4
    Job Counters 
        Launched map tasks=2
        Data-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=24409
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=24409
        Total vcore-seconds taken by all map tasks=24409
        Total megabyte-seconds taken by all map tasks=24994816
    Map-Reduce Framework
        Map input records=3
        Map output records=0
        Input split bytes=188
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=288
        CPU time spent (ms)=1590
        Physical memory (bytes) snapshot=235175936
        Virtual memory (bytes) snapshot=4182507520
        Total committed heap usage (bytes)=158334976
    File Input Format Counters 
        Bytes Read=3400
    File Output Format Counters 
        Bytes Written=244
17/11/06 01:32:10 INFO streaming.StreamJob: Output directory: /tmp/file20421db4106c

> from.dfs(calc)
$key
NULL

$val
NULL

Пожалуйста, помогите с этой проблемой. В отдельной ветке я читал о похожей проблеме, когда пользователь пытался установить rmr2 и другие связанные пакеты в режиме sudo R cmd и смог решить эту проблему. Я пробовал то же самое, но безрезультатно.

0 ответов

Другие вопросы по тегам