Matrix Multiplication Using Hadoop Map-Reduce

10 pages
0 views

Please download to get full document.

View again

of 10
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Share
Description
Description:
Tags
Transcript
    Matrix Multiplication using Hadoop Map-Reduce Step 1: Install Hadoop in Stand-Alone Mode Step 2: Matrix MultiplicationUsing MapReduce Programming 1.1 Installing Java Check Existing Java version by running command  java -version 1.2 Create hadoop home directory We will use hadoop 3.1.2.tar.gz here. Extract hadoop file using following command tar -xzvf hadoop-2.7.3.tar.gz Move hadoop to /usr/local  sudo mv hadoop-3.1.2 /usr/local/hadoop  1.3 Configuring Hadoop's Java_home Hadoop requires that you set the path to Java, either as an environment variable or in the Hadoop configuration file. The path to Java, /usr/bin/java is a symlink to /etc/alternatives/java, which is in turn a symlink to default Java binary. We will use readlink with the -f flag to follow every symlink in every part of the path, recursively. Then, we'll use sed to trim bin/java from the output to give us the correct value for JAVA_HOME To find the default Java path readlink -f /usr/bin/java | sed s:bin/java:: Output :  /usr/lib/jvm/java-11-openjdk-amd64/ Use Readlink to Set the Value Dynamically Sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh Add this line for export JAVA_HOME=$(readlink -f /usr/bin/java | sed s:bin/java:: )  1.4 Running Hadoop  Now we should be able to run Hadoop:  /usr/local/hadoop/bin/hadoop Output : The help means we've successfully configured Hadoop to run in stand-alone mode. We'll ensure that it is functioning properly by running the example MapReduce  program it ships with. To do so, create a directory called input in our home directory and copy Hadoop's configuration files into it to use those files as our data. mkdir ~/input cp /usr/local/hadoop/etc/hadoop/*.xml ~/input   Next, we can use the following command to run the MapReduce hadoop-mapreduce-examples  program, a Java archive with several options. We'll invoke its grep program, one of many examples included in hadoop-mapreduce-examples, followed by the input directory, input and the output directory grep_example. The MapReduce grep program will count the matches of a literal word or regular expression. Finally, we'll supply a regular expression to find occurrences of the word principal within or at the end of a declarative sentence. The expression is case-sensitive, so we wouldn't find the word if it were capitalized at the  beginning of a sentence:  /usr/local/hadoop/bin/hadoop jar  /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar grep ~/input ~/grep_example 'principal[.]*' When the task completes, it provides a summary of what has been processed and errors it has encountered, but this doesn't contain the actual results Results are stored in the output directory and can be checked by running cat on the output directory: cat ~/grep_example/*
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x