Creating Eclipse Plugin for Hadoop-2.2.0

Today I am going to Create Hadoop Plugin for Eclipse.

1.Download and Install Eclipse IDE
2.Download source code for Eclipse Plugin for Hadoop
3.Create Eclipse Plugin for Hadoop.
4.Compile and create jar
5.Install the plugin to eclipse.

1.First Download and install Eclipse.

#wget http://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/kepler/SR1/eclipse-standard-kepler-SR1-linux-gtk-x86_64.tar.gz

2.Then install git on your CentOS Virtual Machine.Run the following command as root.

#sudo yum install git

If not installed ant then download using the following

#sudo cd /usr/local

#wget http://apache.osuosl.org/ant/binaries/apache-ant-1.9.3-bin.tar.gz

#tar -zxvf apache-ant-1.9.3-bin.tar.gz

Set the Path in /etc/profile file and then logout and relogin as normal hduser.

As a normal hduser run the following command.

3.Download source code for Hadoop Plugin for Eclipse from git

$git clone https://github.com/winghc/hadoop2x-eclipse-plugin.git

4.Compile and create jar

$cd /home/hduser/hadoop2x-eclipse-plugin/src/contrib/eclipse-plugin

$ant jar  -Dversion=2.2.0 -Declipse.home=/usr/local/eclipse -Dhadoop.home=/usr/local/hadoop-2.2.0

The build output can be found at

/home/hduser/hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.2.0.jar

build-plugin

Advertisements

14 thoughts on “Creating Eclipse Plugin for Hadoop-2.2.0

  1. Anumoy Sutradhar says:

    Hi, the blog has help me so much that I can not explain. But how to configure the hadoop location in the plugin as Hadoop 2.2.0 does’t have a job-tracker of its own (yarn resource managers are there).
    So which port I have to give??? (Big problem) Any help is appreciable. Thanks in advance.

    Like

    • Anumoy Sutradhar says:

      After a long search I have manage to do it. I want to share this with you (because it happened only with the help of your blog). In the mapred-site.xml add the below properties with the other mentioned in all installation steps:-

      mapreduce.jobtracker.address
      localhost:54311

      mapreduce.jobtracker.http.address
      0.0.0.0:50030

      mapreduce.jobhistory.address
      0.0.0.0:10020

      mapreduce.jobhistory.webapp.address
      0.0.0.0:19888

      and after that configure your hadoop location as:-
      Map/Reduce (V2) Master
      Host: localhost
      port: 54311

      DFS Master
      check the checkbox use M/R master host
      port: 9000

      Thank to all.
      I am very happy!!!

      Like

      • ankur says:

        Anumoy Sutradhar : I am not able to connet to Hadoop via plugin using your method

        my mapred-site.xml is :-

        mapreduce.framework.name
        yarn

        mapreduce.jobtracker.address
        localhost:54311

        mapreduce.jobtracker.http.address
        0.0.0.0:50030

        mapreduce.jobhistory.address
        0.0.0.0:10020

        mapreduce.jobhistory.webapp.address
        0.0.0.0:19888

        and my configurationg is
        Location name localhost
        Map/Reduce (V2) Master
        host localhost
        port 54311
        DFS Master
        use M/R Master host checked
        port 9000

        still not able to connect

        error failed on local exception java.net.Socket ..

        Like

  2. Anumoy Sutradhar says:

    Hi Ankur,
    Check you Hadoop version, mode – single, pseudo, or multi. Java version. Plug jar built successfully or not. Because my one is working nice still now with out any disturbance. As per your error mentioned I think there is some small problem with you java.
    And please mention your core-site.xml configuration.

    Like

  3. Sibel Karabulut says:

    Hi;

    My build seem to succesfull, but there are some errors like below:

    So tell me that how can I solve this problem ..

    hduser@sibel-Qosmio-X875:~/hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin$ ant jar -Dversion=2.2.0 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/local/hadoop
    Buildfile: /home/hduser/hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin/build.xml

    check-contrib:

    init:
    [echo] contrib: eclipse-plugin

    init-contrib:

    ivy-download:
    [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
    [get] To: /home/hduser/hadoop2x-eclipse-plugin-master/ivy/ivy-2.1.0.jar
    [get] Not modified – so not downloaded

    ivy-probe-antlib:

    ivy-init-antlib:

    ivy-init:
    [ivy:configure] :: Ivy 2.1.0 – 20090925235825 :: http://ant.apache.org/ivy/ ::
    [ivy:configure] :: loading settings :: file = /home/hduser/hadoop2x-eclipse-plugin-master/ivy/ivysettings.xml

    ivy-resolve-common:

    ivy-retrieve-common:
    [ivy:cachepath] DEPRECATED: ‘ivy.conf.file’ is deprecated, use ‘ivy.settings.file’ instead
    [ivy:cachepath] :: loading settings :: file = /home/hduser/hadoop2x-eclipse-plugin-master/ivy/ivysettings.xml

    compile:
    [echo] contrib: eclipse-plugin
    [javac] /home/hduser/hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin/build.xml:76: warning: ‘includeantruntime’ was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds

    jar:

    BUILD SUCCESSFUL
    Total time: 3 seconds

    Like

  4. L.GANESH says:

    Hi Gentle Man, (I could not find your name, even in About me Link),
    Excellent article, I wasted my time to find solution with Hortonworks sandbox and Eclipse connectivity.
    In simple click, i created jar file for eclipse and works like YOU.
    THANKS LOT,
    Please keep share your experience.
    L.GANESH

    Like

  5. peter says:

    i am trying to compile this and i get the message: “Target “jar” does not exist in the project “hadoopcontrib””. Do you have any idea what could give this error? any help is appreciated!

    Like

  6. peter says:

    OK, so i realized I was executing from the ‘contrib’ directory NOT the ‘eclipse-plugin’ directory. i changed and got it to compile. i threw it into Eclipse and i’m now talking to the my hortonworks VM’s 2.2.0 hadoop install.

    THANK YOU THANK YOU!!! i’ve been trying to do this myself, but VERY unsuccessfully. thanks!!!!!!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s