From Fedora Project Wiki
Line 67: Line 67:
  ! State !! Notes
  ! State !! Notes
  |-
  |-
  | Available || free for someone to take
 
  | '''<span style="color:darkviolet">Available</span>''' || free for someone to take
  |-
  |-
  | Active    || actively being packaged
  | '''<span style="color:blue">Active</span>'''   || actively being packaged
  |-
  |-
  | Blocked  || pending packages for dependencies
  | '''<span style="color:red">Blocked</span>'''   || pending packages for dependencies
  |-
  |-
  | Review    || under review, include link to review BZ
  | '''<span style="color:orange">Review</span>'''   || under review, include link to review BZ
  |-
  |-
  | Complete  || woohoo!
  | '''<span style="color:green">Complete</span>''' || woohoo!
  |}
  |}


Line 84: Line 85:
  | <div id="0">0</div>
  | <div id="0">0</div>
  | hadoop
  | hadoop
  | Blocked
  | '''<span style="color:red">Blocked</span>'''
  |
  |
  | [[User:rrati|rrati]],[[User:pmackinn|pmackinn]]
  | [[User:rrati|rrati]],[[User:pmackinn|pmackinn]]
Line 91: Line 92:
  | <div id="1">1</div>
  | <div id="1">1</div>
  | ant
  | ant
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 98: Line 99:
  | <div id="2">2</div>
  | <div id="2">2</div>
  | apache-commons-daemon
  | apache-commons-daemon
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 105: Line 106:
  | <div id="3">3</div>
  | <div id="3">3</div>
  | apache-commons-math
  | apache-commons-math
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 112: Line 113:
  | <div id="4">4</div>
  | <div id="4">4</div>
  | bookkeeper
  | bookkeeper
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 119: Line 120:
  | <div id="5">5</div>
  | <div id="5">5</div>
  | gmaven
  | gmaven
  | Available (but has a broken deps. depend on RHBZ#914056)
  | '''<span style="color:darkviolet">Available</span>''' (but has a broken deps. depend on RHBZ#914056)
  |
  |
  |
  |
Line 126: Line 127:
  | <div id="6">6</div>
  | <div id="6">6</div>
  | glassfish-gmbal
  | glassfish-gmbal
  | Review
  | '''<span style="color:orange">Review</span>'''
  | {{bz|859112}}
  | {{bz|859112}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]
Line 133: Line 134:
  | <div id="7">7</div>
  | <div id="7">7</div>
  | grizzly
  | grizzly
  | Review
  | '''<span style="color:orange">Review</span>'''
  | {{bz|859114}}
  | {{bz|859114}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]
Line 140: Line 141:
  | <div id="8">8</div>
  | <div id="8">8</div>
  | glassfish-management
  | glassfish-management
  | Review
  | '''<span style="color:orange">Review</span>'''
  | {{bz|859110}}
  | {{bz|859110}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]
Line 147: Line 148:
  | <div id="9">9</div>
  | <div id="9">9</div>
  | groovy
  | groovy
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  | {{bz|858127}}
  | {{bz|858127}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]
Line 154: Line 155:
  | <div id="10">10</div>
  | <div id="10">10</div>
  | hsqldb
  | hsqldb
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 161: Line 162:
  | <div id="11">11</div>
  | <div id="11">11</div>
  | javax-servlet
  | javax-servlet
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 168: Line 169:
  | <div id="12">12</div>
  | <div id="12">12</div>
  | jersey
  | jersey
  | Complete
  | '''<span style="color:green">Complete</span>'''
  | {{bz|825347}}
  | {{bz|825347}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]
Line 175: Line 176:
  | <div id="13">13</div>
  | <div id="13">13</div>
  | jets3t
  | jets3t
  | Review
  | '''<span style="color:orange">Review</span>'''
  | {{bz|47109}}
  | {{bz|47109}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]
Line 182: Line 183:
  | <div id="14">14</div>
  | <div id="14">14</div>
  | jetty
  | jetty
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 189: Line 190:
  | <div id="15">15</div>
  | <div id="15">15</div>
  | jspc-compiler
  | jspc-compiler
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 196: Line 197:
  | <div id="16">16</div>
  | <div id="16">16</div>
  | kfs
  | kfs
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 203: Line 204:
  | <div id="17">17</div>
  | <div id="17">17</div>
  | maven-native
  | maven-native
  | Review
  | '''<span style="color:orange">Review</span>'''
  | {{bz|864084}}
  | {{bz|864084}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]
Line 210: Line 211:
  | <div id="18">18</div>
  | <div id="18">18</div>
  | slf4j
  | slf4j
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 217: Line 218:
  | <div id="19">19</div>
  | <div id="19">19</div>
  | tomcat-jasper
  | tomcat-jasper
  | Available
  | '''<span style="color:darkviolet">Available</span>'''
  |
  |
  |
  |
Line 224: Line 225:
  | <div id="20">20</div>
  | <div id="20">20</div>
  | zookeeper
  | zookeeper
  | Review
  | '''<span style="color:orange">Review</span>'''
  | {{bz|823122}}
  | {{bz|823122}}
  | [[User:gil|gil]]
  | [[User:gil|gil]]

Revision as of 05:04, 4 April 2013

Apache Hadoop 2.0

Summary

Bring Apache Hadoop, the hottest open source big data platform, to Fedora, the hottest open source distribution. Fedora should be the best distribution for using Apache Hadoop.

Owner

People involved

Name IRC Focus Additional
Matthew Farrellee mattf keeping track, integration testing UTC-5
Peter MacKinnon pmackinn packaging UTC-5
Rob Rati rsquared packaging UTC-5


Current status

  • Targeted release: Fedora 20
  • Last updated: 3 Apr 2013
  • Percentage of completion: 5%


Detailed Description

Apache Hadoop is a widely used, increasingly complete big data platform, with a strong open source community and growing ecosystem. The goal is to package and integrate the core of the Hadoop ecosystem for Fedora, allowing for immediate use and creating a base for the rest of the ecosystem.


Benefit to Fedora

The Apache Hadoop software will be packaged and integrated with Fedora. The core of the Hadoop ecosystem will be available with Fedora and provide a base for additional packages.


Scope

  • Package the Apache Hadoop 2.0.2 software
  • Package all dependencies needed for Apache Hadoop 2.0.2
  • Skip package dependencies required for unit testing, record them in a dependency backlog for later cleanup

Approach

We are taking an iterative, depth-first approach to packaging. We do not have all the dependencies mapped out ahead of time.

Anyone who wants to help should find an available dependency below, edit the table changing the state to Active and packager to yourself.

While packaging a dependency, test dependencies can be skipped. Testing will be done via integration testing periodically during packaging and then after packaging completes. Test dependencies that are skipped must be added to the Skipped dependencies table below.

If you are lucky enough to pick a dependency that itself has unpackaged dependencies, identity the sub-dependencies and add them to the bottom of the Dependencies table below, change your current dependency to Blocked and repeat.

If your dependency is already packaged but the version is incompatible, contact the package owner and resolve the incompatibility in a mutually satisfactory way. For instance:

  • If the version available in Fedora is older, explore updating the package. If that is not possible, explore creating a package that includes a version in its name, e.g. pkgnameXY. Ultimately, the most recent version in Fedora should have the name pkgname while older versions have pkgnameXY. It may take a full Fedora release to rationalize package names. Make a note in the Dependencies table.
  • If the version you need is older than the packaged version, consider creating a patch to use the newer version. If a patch is not viable, proceed by packaging the dependency with a version in its name, e.g. pkgnameXY. Make a note in the Dependencies table.
Table legend
State Notes
Available free for someone to take
Active actively being packaged
Blocked pending packages for dependencies
Review under review, include link to review BZ
Complete woohoo!
Dependencies
# Project State Review BZ Packager Notes
0
hadoop Blocked rrati,pmackinn
1
ant Available VERSION INCOMPATIBILITY: Version 1.6 asked for, 1.8 currently packaged in Fedora. Needs to be inspected for api/functional incompatibilities
2
apache-commons-daemon Available Java import compilation error with existing package. Needs inspection
3
apache-commons-math Available Current apache-commons-math uses math3 in pom instead of math. Hadoop requests version 2.x. Needs inspection
4
bookkeeper Available Version 4.0 requested
5
gmaven Available (but has a broken deps. depend on RHBZ#914056) Version 1.0 requested
6
glassfish-gmbal Review RHBZ #859112 gil
7
grizzly Review RHBZ #859114 gil
8
glassfish-management Review RHBZ #859110 gil
9
groovy Available RHBZ #858127 gil 1.5 requested, but 1.8 packaged in fedora. Possible moving forward 1.8 series will be known as groovy18 and groovy will be 2.x.
10
hsqldb Available 1.8 in fedora, 2.0 requested. 2.2.8 packaged by gil, but seemingly no review request. Needs followup.
11
javax-servlet Available 3.0 requested for glassfish groupID. Failed resolution. Needs investigation since other javax.servlet API packages should be available in Fedora (tomcat?).
12
jersey Complete RHBZ #825347 gil
13
jets3t Review RHBZ #47109 gil
14
jetty Available VERSION INCOMPATIBILITY: jetty8 packaged in Fedora, but 6.x requested. 6 and 8 are incompatible. Needs investigation
15
jspc-compiler Available
16
kfs Available gil has packaged 0.5, but no review request. kfs has become Quantcast qfs
17
maven-native Review RHBZ #864084 gil Needs patch to build with java7
18
slf4j Available Package in fedora fails to match in dependency resolution. Needs inspection
19
tomcat-jasper Available Version 5.5.x requested
20
zookeeper Review RHBZ #823122 gil
22
[package name] Available RHBZ #XYZ noone [notes]
# JAR Project State Packager Notes
0 [jar name] [package name] Available noone Needed for tests by #N

Packager tips

  • mvn-rpmbuild utility will ONLY resolve from system repo
  • mvn-local will resolve from system repo first then fallback to maven if unresolved
  • can be used to find the delta between system repo packages available and missing dependencies that can be viewed in the .m2 local maven repo (find *.jar)
  • -Dmaven.local.debug=true
    • reveals how JPP lookups are executing per dependency -> useful for finding gId,aId mismatches
  • -Dmaven.test.skip=true
    • tells maven to skip test compilation

TODO: Template spec files to work from

TODO: Setup staging repository for sharing packages under review


How To Test

  1. TODO: NEEDS MORE DEFINITION
  2. yum install X Y Z across one or more nodes
  3. Setup a simple cluster by following TBD
  4. Run http://hadoop.apache.org/docs/stable/gridmix.html


User Experience

For users who are interested in running Apache Hadoop on Fedora, they will find it available from Fedora Project yum repositories.

TODO: SPECIFICALLY PACKAGES X Y Z


Dependencies

No other packages currently depend on Apache Hadoop.

Completion of this feature will involve packaging numerous dependencies, see the Dependencies table. Some of the dependencies are already being packaged by others in the Fedora community. Where dependency overlap is found, a negotaition must occur to ensure a satisfactory version and package is available to all parties.

TODO: Is https://fedoraproject.org/wiki/Hypertable ?


Contingency Plan

With no packages depending on Apache Hadoop, none is necessary. The biggest risk is not completing packages for all dependencies. In that case, the feature can be removed from the release notes. The packaged dependencies should remain in the distribution. The feature can be pushed to the next Fedora release.


Documentation


Release Notes

  • TODO


Comments and Discussion