From Fedora Project Wiki

< SIGs‎ | bigdata‎ | packaging

No edit summary
No edit summary
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This is an summary as a part of the [[Big_data_SIG|Big Data SIG]] of the issues uncovered during Mesos packaging.
This is an summary as a part of the [[SIGs/bigdata|Big Data SIG]] of the issues uncovered during Mesos packaging.


= Resolved Issues =
= Resolved Issues =
Line 8: Line 8:
* Updates to follow standard conventions around include dir (e.g. process->libprocess)  
* Updates to follow standard conventions around include dir (e.g. process->libprocess)  
* Updates to follow standard #ifndef __LIB_HEADER__ checks (some were missing, or not prefixed _LIB_).
* Updates to follow standard #ifndef __LIB_HEADER__ checks (some were missing, or not prefixed _LIB_).
* System integration files. (inside package)  
* System integration files. (inside package)
* env deployment overloads moved from /var/mesos to /etc/sysconfig/mesos, more appropriate for env settings
* libmesos now a properly versioned library


= Review Tickets =  
= Upstream Tracking =  
* stout  
* https://issues.apache.org/jira/browse/MESOS-543
** forked-repo: https://github.com/besser82/stout
* Patches: https://issues.apache.org/jira/browse/MESOS-543
** rpm-spec: https://github.com/ignatenkobrain/stout-rpm
 
** packaging BZ: https://bugzilla.redhat.com/show_bug.cgi?id=988545
= Packaging Breakdown =
* libprocess
* <strike> stout </strike>
** forked-repo: https://github.com/ignatenkobrain/libprocess  
** <strike> forked-repo: https://github.com/besser82/stout </strike>
** rpm-spec: https://github.com/ignatenkobrain/libprocess-rpm
** <strike> rpm-spec: https://github.com/ignatenkobrain/stout-rpm </strike>
** packaging BZ: https://bugzilla.redhat.com/show_bug.cgi?id=994152  
** <strike> packaging BZ: https://bugzilla.redhat.com/show_bug.cgi?id=988545 </strike>
* mesos
* <strike> libprocess </strike>
** forked-repo: https://github.com/timothysc/mesos
** <strike> forked-repo: https://github.com/ignatenkobrain/libprocess </strike>
** <strike> rpm-spec: https://github.com/ignatenkobrain/libprocess-rpm </strike>
** <strike> packaging BZ: https://bugzilla.redhat.com/show_bug.cgi?id=994152 </strike>
* mesos '''(per conversations with upstream libraries have been subpackaged)'''
** forked-repo: https://github.com/timothysc/mesos '''(upstream_shuffle branch)'''
** rpm-spec: https://github.com/timothysc/mesos-rpm  
** rpm-spec: https://github.com/timothysc/mesos-rpm  
**  
** https://bugzilla.redhat.com/show_bug.cgi?id=1010512


= Open Issues =
= Open Issues =
I've tried to list the open issues in priority order.  All recommendations listed here should be construed as developer opinion, and the reader should choose whatever path makes the most sense.   
I've tried to list the open issues in priority order.  All recommendations listed here should be construed as developer opinion, and the reader should choose whatever path makes the most sense.   
 
== more autotools cleaning ==
Currently I have checks that will match the packages I've created, and will likely work if folks want to pull and rebuild the .srpm for their own koji setup, but there may be check mismatch on other distros.  I also may have cheated in places by ensuring their proper rpm dependencies.
 
== JNI Linkage ==
Ideally switch to dlopen() around jvm interface.   


== Canonical source for dependencies ==  
== .jar(s) not installed? ==  
Both libprocess and stout are considered separable libraries with their own repos on github but are not setup as submodules within the main repository thus confusing "What is the definitive source?" and whether those libraries should be broken out separately or not?  Currently I've tried to sync with the mesos repository and update the packages in fedora while supplying a proper autoconf check in mesos for the dependency.
I don't understand why mesos.jar is not installed.


== JNI reaching into zookeeper utility classes ==  
== JNI reaching into zookeeper utility classes ==  
Line 38: Line 50:
    
    
== Failing Tests ==  
== Failing Tests ==  
Given the stack changes there are approximately 2 dozen tests that have been temporarily disabled, many of them are Zookeeper integration issues.
Given the stack changes there are approximately 2 dozen tests that have been temporarily disabled, many of them are Zookeeper integration issues.  We will carry this patch in our packaging, but the afore mentioned issue should be addressed first.
 
Some tests also require DNS, which causes issues in a mock/koji env, but we can carry those patches too.


== Boost / TR1 / C++11 ==  
== Boost / TR1 / C++11 ==  
There appears to be jumbled dep-graph here, and reliance on C++11 is preferred going forwards.  Adding checks where possible and leverage semantics there (e.g. locking).
There appears to be deps on tr1 but no checks for C++11.  Adding checks where possible and leverage semantics C++11 where possible is preferred for long term maintenance.  This applies to several elements in the code.. threading (pthread), variadic templates (tuple), etc.

Latest revision as of 16:17, 22 October 2013

This is an summary as a part of the Big Data SIG of the issues uncovered during Mesos packaging.

Resolved Issues

  • Removal of hard 3rdparty dependencies and replaced with autoconf checks where possible.
  • Updated #define CHECKS on library interface dependencies
    • http_parser
    • Zookeeper
  • Updates to follow standard conventions around include dir (e.g. process->libprocess)
  • Updates to follow standard #ifndef __LIB_HEADER__ checks (some were missing, or not prefixed _LIB_).
  • System integration files. (inside package)
  • env deployment overloads moved from /var/mesos to /etc/sysconfig/mesos, more appropriate for env settings
  • libmesos now a properly versioned library

Upstream Tracking

Packaging Breakdown

Open Issues

I've tried to list the open issues in priority order. All recommendations listed here should be construed as developer opinion, and the reader should choose whatever path makes the most sense.

more autotools cleaning

Currently I have checks that will match the packages I've created, and will likely work if folks want to pull and rebuild the .srpm for their own koji setup, but there may be check mismatch on other distros. I also may have cheated in places by ensuring their proper rpm dependencies.

JNI Linkage

Ideally switch to dlopen() around jvm interface.

.jar(s) not installed?

I don't understand why mesos.jar is not installed.

JNI reaching into zookeeper utility classes

The reliance on internal utility classes inside of zookeeper can be quite problematic. Applications developers will usually supply a contract around the interface but the inner-workings are usually subject to churn. During the packaging it became apparent of the drift.

Recommendations:

  • Remove the dependencies and try to stick with the interface where possible.
  • Use a pImpl idiom || some factory mechanism and hide the JNI goo below such that autoconf can enable impl1 impl2.

Failing Tests

Given the stack changes there are approximately 2 dozen tests that have been temporarily disabled, many of them are Zookeeper integration issues. We will carry this patch in our packaging, but the afore mentioned issue should be addressed first.

Some tests also require DNS, which causes issues in a mock/koji env, but we can carry those patches too.

Boost / TR1 / C++11

There appears to be deps on tr1 but no checks for C++11. Adding checks where possible and leverage semantics C++11 where possible is preferred for long term maintenance. This applies to several elements in the code.. threading (pthread), variadic templates (tuple), etc.