Spark & Hadoop в Aliyun ECS
Это время для настройки кластера в aliyun ECS. Есть четыре узла, один из которых является главным и тремя подчиненными. В любом случае, это узел 2G Memory и один ядерный процессор. После установки hadoop и spark, когда я запускаю пряжу Примеры кластеров в искре, это не имеет значения.
# ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
17/09/07 11:02:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/09/07 11:02:43 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.134:8032
17/09/07 11:02:43 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
17/09/07 11:02:43 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/09/07 11:02:43 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/09/07 11:02:43 INFO yarn.Client: Setting up container launch context for our AM
17/09/07 11:02:43 INFO yarn.Client: Setting up the launch environment for our AM container
17/09/07 11:02:43 INFO yarn.Client: Preparing resources for our AM container
17/09/07 11:02:44 INFO yarn.Client: Uploading resource file:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/spark-assembly-1.6.3-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1502333784376_0007/spark-assembly-1.6.3-hadoop2.6.0.jar
Терминал не ответил. Затем я смотрю журнал искры.
# tail spark-root-org.apache.spark.deploy.master.Master-1-master.out
Spark Command: /usr/local/jdk1.8/bin/java -cp /usr/local/src/spark-1.6.3-bin-hadoop2.6/conf/:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/spark-assembly-1.6.3-hadoop2.6.0.jar:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/usr/local/src/hadoop-2.6.5/etc/hadoop/ -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip master --port 7077 --webui-port 8080
========================================
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d5550000, 715849728, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 715849728 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/src/hs_err_pid4954.log
Это информация для hs_err_pid.log, приведенная ниже
[root@master logs]# cat /usr/local/src/hs_err_pid4954.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 715849728 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2640), pid=4954, tid=0x00007f9935f9f700
#
# JRE version: (8.0_144-b01) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007f9930008800): JavaThread "Unknown thread" [_thread_in_vm, id=4974, stack(0x00007f9935e9f000,0x00007f9935fa0000)]
Stack: [0x00007f9935e9f000,0x00007f9935fa0000], sp=0x00007f9935f9e310, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xacb18a] VMError::report_and_die()+0x2ba
V [libjvm.so+0x4ff4db] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V [libjvm.so+0x927d23] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103
V [libjvm.so+0x928279] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V [libjvm.so+0x92249a] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V [libjvm.so+0xac6ee9] VirtualSpace::expand_by(unsigned long, bool)+0x199
V [libjvm.so+0xac7a3e] VirtualSpace::initialize(ReservedSpace, unsigned long)+0xee
V [libjvm.so+0x5fdaa1] CardGeneration::CardGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0xf1
V [libjvm.so+0xa6da8f] TenuredGeneration::TenuredGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0x5f
V [libjvm.so+0x5feda9] GenerationSpec::init(ReservedSpace, int, GenRemSet*)+0x339
V [libjvm.so+0x5ed98e] GenCollectedHeap::initialize()+0x1de
V [libjvm.so+0xa93203] Universe::initialize_heap()+0xf3
V [libjvm.so+0xa9376e] universe_init()+0x3e
V [libjvm.so+0x640565] init_globals()+0x65
V [libjvm.so+0xa76e5e] Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
V [libjvm.so+0x6d4e64] JNI_CreateJavaVM+0x74
C [libjli.so+0x745e] JavaMain+0x9e
C [libpthread.so.0+0x7dc5] start_thread+0xc5
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
Other Threads:
=>0x00007f9930008800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=4974, stack(0x00007f9935e9f000,0x00007f9935fa0000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events
Dynamic libraries:
00400000-00401000 r-xp 00000000 fd:01 1720661 /usr/local/jdk1.8/bin/java
00600000-00601000 rw-p 00000000 fd:01 1720661 /usr/local/jdk1.8/bin/java
01537000-01558000 rw-p 00000000 00:00 0 [heap]
c0000000-d5550000 rw-p 00000000 00:00 0
7f991fd47000-7f9920079000 rw-p 00000000 00:00 0
7f9920079000-7f99201ce000 ---p 00000000 00:00 0
7f99201ce000-7f99201d9000 rw-p 00000000 00:00 0
7f99201d9000-7f992058f000 ---p 00000000 00:00 0
7f992058f000-7f99207ff000 rwxp 00000000 00:00 0
7f99207ff000-7f992f58f000 ---p 00000000 00:00 0
7f992f58f000-7f992f5a9000 r-xp 00000000 fd:01 1720766 /usr/local/jdk1.8/jre/lib/amd64/libzip.so
7f992f5a9000-7f992f7a9000 ---p 0001a000 fd:01 1720766 /usr/local/jdk1.8/jre/lib/amd64/libzip.so
7f992f7a9000-7f992f7aa000 rw-p 0001a000 fd:01 1720766 /usr/local/jdk1.8/jre/lib/amd64/libzip.so
7f992f7aa000-7f992f7b6000 r-xp 00000000 fd:01 1507926 /usr/lib64/libnss_files-2.17.so
7f992f7b6000-7f992f9b5000 ---p 0000c000 fd:01 1507926 /usr/lib64/libnss_files-2.17.so
7f992f9b5000-7f992f9b6000 r--p 0000b000 fd:01 1507926 /usr/lib64/libnss_files-2.17.so
7f992f9b6000-7f992f9b7000 rw-p 0000c000 fd:01 1507926 /usr/lib64/libnss_files-2.17.so
7f992f9b7000-7f992f9bd000 rw-p 00000000 00:00 0
7f992f9bd000-7f992f9e8000 r-xp 00000000 fd:01 1720793 /usr/local/jdk1.8/jre/lib/amd64/libjava.so
7f992f9e8000-7f992fbe7000 ---p 0002b000 fd:01 1720793 /usr/local/jdk1.8/jre/lib/amd64/libjava.so
7f992fbe7000-7f992fbe9000 rw-p 0002a000 fd:01 1720793 /usr/local/jdk1.8/jre/lib/amd64/libjava.so
7f992fbe9000-7f992fbf6000 r-xp 00000000 fd:01 1720744 /usr/local/jdk1.8/jre/lib/amd64/libverify.so
7f992fbf6000-7f992fdf6000 ---p 0000d000 fd:01 1720744 /usr/local/jdk1.8/jre/lib/amd64/libverify.so
7f992fdf6000-7f992fdf8000 rw-p 0000d000 fd:01 1720744 /usr/local/jdk1.8/jre/lib/amd64/libverify.so
7f992fdf8000-7f992fdff000 r-xp 00000000 fd:01 1507938 /usr/lib64/librt-2.17.so
7f992fdff000-7f992fffe000 ---p 00007000 fd:01 1507938 /usr/lib64/librt-2.17.so
7f992fffe000-7f992ffff000 r--p 00006000 fd:01 1507938 /usr/lib64/librt-2.17.so
7f992ffff000-7f9930000000 rw-p 00007000 fd:01 1507938 /usr/lib64/librt-2.17.so
7f9930000000-7f9930034000 rw-p 00000000 00:00 0
7f9930034000-7f9934000000 ---p 00000000 00:00 0
7f993409f000-7f993419f000 r-xp 00000000 fd:01 1507916 /usr/lib64/libm-2.17.so
7f993419f000-7f993439f000 ---p 00100000 fd:01 1507916 /usr/lib64/libm-2.17.so
7f993439f000-7f99343a0000 r--p 00100000 fd:01 1507916 /usr/lib64/libm-2.17.so
7f99343a0000-7f99343a1000 rw-p 00101000 fd:01 1507916 /usr/lib64/libm-2.17.so
7f99343a1000-7f9935074000 r-xp 00000000 fd:01 1720797 /usr/local/jdk1.8/jre/lib/amd64/server/libjvm.so
7f9935074000-7f9935273000 ---p 00cd3000 fd:01 1720797 /usr/local/jdk1.8/jre/lib/amd64/server/libjvm.so
7f9935273000-7f993534d000 rw-p 00cd2000 fd:01 1720797 /usr/local/jdk1.8/jre/lib/amd64/server/libjvm.so
7f993534d000-7f9935398000 rw-p 00000000 00:00 0
7f9935398000-7f993554e000 r-xp 00000000 fd:01 1507908 /usr/lib64/libc-2.17.so
7f993554e000-7f993574e000 ---p 001b6000 fd:01 1507908 /usr/lib64/libc-2.17.so
7f993574e000-7f9935752000 r--p 001b6000 fd:01 1507908 /usr/lib64/libc-2.17.so
7f9935752000-7f9935754000 rw-p 001ba000 fd:01 1507908 /usr/lib64/libc-2.17.so
7f9935754000-7f9935759000 rw-p 00000000 00:00 0
7f9935759000-7f993575b000 r-xp 00000000 fd:01 1507914 /usr/lib64/libdl-2.17.so
7f993575b000-7f993595b000 ---p 00002000 fd:01 1507914 /usr/lib64/libdl-2.17.so
7f993595b000-7f993595c000 r--p 00002000 fd:01 1507914 /usr/lib64/libdl-2.17.so
7f993595c000-7f993595d000 rw-p 00003000 fd:01 1507914 /usr/lib64/libdl-2.17.so
7f993595d000-7f9935972000 r-xp 00000000 fd:01 1714306 /usr/local/jdk1.8/lib/amd64/jli/libjli.so
7f9935972000-7f9935b72000 ---p 00015000 fd:01 1714306 /usr/local/jdk1.8/lib/amd64/jli/libjli.so
7f9935b72000-7f9935b73000 rw-p 00015000 fd:01 1714306 /usr/local/jdk1.8/lib/amd64/jli/libjli.so
7f9935b73000-7f9935b8a000 r-xp 00000000 fd:01 1507934 /usr/lib64/libpthread-2.17.so
7f9935b8a000-7f9935d89000 ---p 00017000 fd:01 1507934 /usr/lib64/libpthread-2.17.so
7f9935d89000-7f9935d8a000 r--p 00016000 fd:01 1507934 /usr/lib64/libpthread-2.17.so
7f9935d8a000-7f9935d8b000 rw-p 00017000 fd:01 1507934 /usr/lib64/libpthread-2.17.so
7f9935d8b000-7f9935d8f000 rw-p 00000000 00:00 0
7f9935d8f000-7f9935daf000 r-xp 00000000 fd:01 1507901 /usr/lib64/ld-2.17.so
7f9935e97000-7f9935e9f000 rw-s 00000000 fd:01 819213 /tmp/hsperfdata_root/4954
7f9935e9f000-7f9935ea2000 ---p 00000000 00:00 0
7f9935ea2000-7f9935fa4000 rw-p 00000000 00:00 0 [stack:4974]
7f9935fa8000-7f9935fac000 rw-p 00000000 00:00 0
7f9935fac000-7f9935fad000 r--p 00000000 00:00 0
7f9935fad000-7f9935fae000 rw-p 00000000 00:00 0
7f9935fae000-7f9935faf000 r--p 0001f000 fd:01 1507901 /usr/lib64/ld-2.17.so
7f9935faf000-7f9935fb0000 rw-p 00020000 fd:01 1507901 /usr/lib64/ld-2.17.so
7f9935fb0000-7f9935fb1000 rw-p 00000000 00:00 0
7ffce7d06000-7ffce7d27000 rw-p 00000000 00:00 0 [stack]
7ffce7dc0000-7ffce7dc2000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
VM Arguments:
jvm_args: -Xms1g -Xmx1g
java_command: org.apache.spark.deploy.master.Master --ip master --port 7077 --webui-port 8080
java_class_path (initial): /usr/local/src/spark-1.6.3-bin-hadoop2.6/conf/:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/spark-assembly-1.6.3-hadoop2.6.0.jar:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/usr/local/src/hadoop-2.6.5/etc/hadoop/
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/local/jdk1.8
PATH=/usr/local/jdk1.8/bin/:/usr/local/jdk1.8/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/protobuf/bin:/root/bin
LD_LIBRARY_PATH=:/usr/local/protobuf/lib
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0xacba10], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xacba10], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x924ba0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x924ba0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x924ba0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x924ba0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x9263e0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
--------------- S Y S T E M ---------------
OS:CentOS Linux release 7.2.1511 (Core)
uname:Linux 3.10.0-514.6.2.el7.x86_64 #1 SMP Thu Feb 23 03:04:39 UTC 2017 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 7281, NOFILE 65535, AS infinity
load average:0.08 0.08 0.07
/proc/meminfo:
MemTotal: 1883740 kB
MemFree: 115116 kB
MemAvailable: 530124 kB
Buffers: 156464 kB
Cached: 363880 kB
SwapCached: 0 kB
Active: 1417712 kB
Inactive: 239988 kB
Active(anon): 1137696 kB
Inactive(anon): 180 kB
Active(file): 280016 kB
Inactive(file): 239808 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 228 kB
Writeback: 0 kB
AnonPages: 1137404 kB
Mapped: 51064 kB
Shmem: 460 kB
Slab: 68884 kB
SReclaimable: 52716 kB
SUnreclaim: 16168 kB
KernelStack: 8192 kB
PageTables: 12348 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 941868 kB
Committed_AS: 3549984 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 10468 kB
VmallocChunk: 34359719676 kB
HardwareCorrupted: 0 kB
AnonHugePages: 802816 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 57216 kB
DirectMap2M: 2039808 kB
DirectMap1G: 0 kB
CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 79 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, tsc, bmi1, bmi2, adx
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz
stepping : 1
microcode : 0x1
cpu MHz : 2494.222
cache size : 40960 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt
bogomips : 4988.44
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 1883740k(115116k free), swap 0k(0k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.144-b01) for linux-amd64 JRE (1.8.0_144-b01), built on Jul 21 2017 21:57:33 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Mon Aug 21 20:07:34 2017
elapsed time: 0 seconds (0d 0h 0m 0s)
Кое-что, что я не знаю, - то, что кластер может работать на локальном компьютере, но это не имеет значения в кластере aliyun, который является 2G Memory. информация для узла приведена ниже
(1)master:
top - 11:15:43 up 28 days, 23:06, 2 users, load average: 0.00, 0.06, 0.06
Tasks: 84 total, 2 running, 81 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1883740 total, 92740 free, 1331376 used, 459624 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 370840 avail Mem
(2)slave1
top - 11:16:19 up 28 days, 23:12, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 68 total, 1 running, 67 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.7 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1883740 total, 68640 free, 838368 used, 976732 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 864688 avail Mem
master
# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz
stepping : 1
microcode : 0x1
cpu MHz : 2494.222
cache size : 40960 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt
bogomips : 4988.44
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management: