the difference between Hadoop-1 and Hadoop-3.

Briefly discuss the difference between Hadoop-1 and Hadoop-3.

하둡 버전 1과 하둡 3의 차이점 비교 _ 이레이져 코딩 / YARN / Namenodes 

(1) Erasure Coding

In the previous version, one data was copied and stored in multiple copies to prepare for possible loss. It was usually made by duplicating one file into three, so storage space was required three times the original size.

 However, in Hadoop 3, data is encoded using algorithms such as reed-solomon, and when data is lost, it is restored through a decoding process. (For each original data cells, a certain number of parity cells are calculated and stored , which is called encoding. The error can be recovered through decoding calculation based on surviving data and parity cells, which is called decoding.)

Therefore, HDFS uses Erasure Coding in place of replication to provide the same level of fault tolerance with storage overhead to be not more than 50%.

 

(2) YARN Timeline Service v.2

 The previous version was limited to a single instance of writer/reader and storage, and does not scale well beyond small clusters. The YARN Timeline Service v.2 uses a more scalable distributed writer architecture and a scalable backend storage.

(3) Supporting for more than 2 NameNodes

The NameNode was a single point of failure (SPOF) in Hadoop 1.0. Each cluster has a single NameNode and If that NameNode gets fail the whole Hadoop cluster will be not available.

Hadoop 2.0 overcomes this SPOF by providing support for 2 NameNodes( 1 Active NameNode, 1 Standby NameNode. ).

Hadoop 3.0 supports more than 2 NameNodes.(N active NameNodes, N Standby NameNodes.) So user can run many standby namenodes. As a result hadoop cluster is able to tolerate the failure of two nodes rather than one.

Understanding Big Data Analytics _ Assignment 1

참고하면 좋은 블로그+

https://joonyon.tistory.com/52

Hadoop 3.0 무엇이 달라졌나???