Hadoop is one of the best open source for store and processing big data. It has a lot of supports from community and many big companies have used it for their products. In my company, Hadoop ecosystem have used to store message chat and information log, it is very effective but it required many resources server as ram, cpu and disk. If your product is small system you should consider using it.
Ok let start find answer for question “How to build hadoop from source ?”
Step 1 : The fist you should disable Firewall local
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
Step 2 : Download JDK and setup environment
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
tar -xzf jdk-8u45-linux-x64.tar.gz -C /opt/
Step 3 : Create user and group “hadoop” as user run service
groupadd hadgroup
useradd haduser -G hadgroup
passwd haduser
Step 4 : Create ssh-key for authentication between servers
ssh-keygen
Step 5 : Install tool development and library
yum groupinstall "Development Tools" "Development Libraries"
yum install openssl-devel cmake
Step 6 : Install maven to build Hadoop (source)
tar -zxf apache-maven-3.3.9-bin.tar.gz -C /opt/
Step 7 : Setup maven environment
export JAVA_HOME=/opt/jdk1.8.0_45
export M3_HOME=/opt/apache-maven-3.3.9
export PATH=/opt/apache-maven-3.3.9/bin:$PATH
Step 8 : Build Protobuf (source)
tar -xzf protobuf-2.5.0.tar.gz -C /root
./configure
make
make install
sudo ldconfig
Step 9 : Download source and build Hadoop (source)
tar -xvf hadoop-2.7.1-src.tar.gz
cd hadoop-2.7.1-src
mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true -Dmaven.javadoc.failOnError=false
Step 10 : Move build to new folder
mv hadoop-2.7.0-src/hadoop-dist/target/hadoop-2.7.0 /opt/
Done, and now you have Hadoop was built at path /opt/hadoop-2.7.0
In the next post, i will write how to setup hadoop as cluster. Thank you!
Learn to become a developer 