From Fedora Project Wiki
No edit summary
 
(3 intermediate revisions by one other user not shown)
Line 14: Line 14:
== Schedule ==
== Schedule ==


We'll have one all-day session with the following topics (feel free to add more):
{|
! Friday !! Saturday !! Sunday !! Unscheduled
|-
| F18 retrospective || Releng meeting? || Bug review|| EPEL discussion
|-
| F19 supported hardware || Secondary Arch process || Releng meeting? || Collaborative Articles
|-
| grub2 on media || Arch:PPC page rewrite|| || Test Templates
|-
| || SDK tutorial || || More optimization opportunities?
|-
| || Infra review || || Write F18 release notes
|- ||
|}
 
Topics:


* Introductions, not everyone from the PPC team has been at the last meeting in Blacksburg
* Introductions, not everyone from the PPC team has been at the last meeting in Blacksburg
Line 31: Line 46:
* EPEL discussion
* EPEL discussion
* should we install 32bit packages at all or shall we copy x86_64 and leave 32bit for the user to install manually after the system install ?
* should we install 32bit packages at all or shall we copy x86_64 and leave 32bit for the user to install manually after the system install ?
* Releng discussion w/ primary (process & schedule, checklist items)
* Bug review
* Bug review
== Full notes of discussions at the meeting ==
Hi everyone.
As the Fedora Secondary Arch Team for Power met again this year at the North
American FUDCon, this year in Lawrence we've put together a pretty long list
of topics we wanted to cover and discuss during those days. Details were   
available here                                                             
https://fedoraproject.org/wiki/Architectures/PowerPC/Meetings/FUDCon_Lawrence_2013
which was linked from the main secondary arch wiki for Power on fp.org the       
last few months.                                                                 
The team meeting at Lawrence were:
Phil Knirsch
Karsten Hopp
David Aquilina
Brent Baude 
Mark Hamzy   
Gustavo Luiz Duarte
+ special guests appearances from the Fedora release engineering,
infrastructure and the ARM teams                                 
Lets dive into the various topics we dicussed and what came out of it.
F18 retrospective
-----------------
The main point here was to list the several good and bad things we noticed
during the F18 development so we could repeate the good ones and fix the bad
ones. Here the final list:                                                 
Good:
- Managed to hit almost all target dates
- Kept schedule in sync with primary   
- Almost passed final RC checklist for our RC4 (just 2 bugs)
- Overall only 2 power specific bugs                       
Bad:
- Signing was/is still an issue:
  o Reliability               
  o Process                   
- Detected Custom paritioning bug specific for Power very late during beta
  testing, resulting in a delay there                                   
- Multipath needed lots of cycles and attention to get working properly 
Regarding the bad things we've had several discussions:
For the signing issues we had we've already started to address some of the
know issues during the development for F18. The main things that still bugged
us were the reliability and the overall process. For reliability we'll be   
working with Miloslav Trmac to provide network dumps and error messages.   
Several from the team will also be at the DevConf in Brno in February to work
with him on this.                                                           
Regarding the process issues we've discussed several options and came up with
the following flow:                                                         
1) Sign packages for tag X. Returns list of all signed packages for tag X
2) Using this list run mesh to compose tree                             
3) Depending on tree (updates or release) create images and push bits   
This will ensure that all packages that get pushed will always be signed on
the first go. It will run the chance that a push on a certain day might miss a
few packages that were just built, but we think the benefit from always only 
pushing signed bits more than outweights this.                               
Next up the late discovery of a critical bug for Beta. I really helped to
begin with that we stuck to the primary schedule as closely as possible. That
allowed us to get composes and images really early on. But in this specific 
instance this didn't really help as it took us several days until we realized
that this was actually a Power specific issue (with Prep partitions), which 
meant the fix for it would be late for Beta. On the other hand it allowed us
to test the updates.img functionality even on the install media. To ensure we
cover the most important scenarios we'll be moving towards using the testing
templates as primary already does, of course with slight modifications and 
additions. Additionally providing a one-stop pointer on the arch wiki to learn
about the current state of builds, signing, composes and install tests using a
dashboard style will help keep everyone informed about all important stages. 
Last up was multipath. We've hit that during the late Alpha testing and
noticed that multipath wasn't on any of the criterias during Beta testing.
We'll propose to add those to the criteria also for primary for upcoming 
Fedora releases.                                                         
Supported archs/hardware for F19
--------------------------------
We discussed this for a bit and at the end came to the conculsion not to
do to much changing there compared to Fedora 18:                       
- Compiler flags just as before for ppc64, generic 64bit optimization 
- For ppc we only provide Everything and Fedora trees, no more install media
  or images                                                               
- Following primary make ppc64 pure, meaning installs won't contain ppc   
  packages anymore                                                         
grub2 as bootloader on install media (DVD)
------------------------------------------
The driving factor here was that with switching to grub2 for Fedora 18 we now
have a modern bootloader for Power that allows lots of cool things, but for 
the media we've still used yaboot (similar to primary using syslinux as the 
media bootloader). As it turns out we already are doing that on primary     
for UEFI images, so this has already gotten some testing and the necessary 
code is already available. The plan is now to create experimental boot ISOs 
with grub2 to see if they work properly and write a proposal post Fedora 19 to
switch to grub2 and touch base with primary releng & anaconda in the time in 
between to coordinate the efforts.                                           
Discuss use of test templates as used on primary:
https://fedoraproject.org/wiki/QA:Fedora_18_Install_Results_Template
--------------------------------------------------------------------
This went pretty quickly as we all agreed that this was a really good idea.
The list will certainly need some review and weeding/additions based on what
we've seen (e.g. multipath install test case). Additionally the final goal is
to link the release criteria with specific test cases where applicable. This
will make testing and checking if we meet the release criteria much easier. 
Ideas on how to improve the visibility of the whole process (koji-shadow logs,
signing progress, updates, etc.) aka Dashboard                               
----------------------------------------------                               
This topic was related to some of the issues we faced during Alpha and Beta
testing. The main goal for the discussion about this was to improve information
availability and visibility to everyone. So the idea is to create a dashboard 
online that contains all following current information:                       
- Compose Status (last ran, logs)                                           
- koji-shadow status (last ran, logs)                                       
- Signing Status (last run, signed vs. unsigned #s)                         
- buildbranched status                                                       
- Failed builds                                                             
- Testing Status (IBM QA? AutoQA? )                                         
Work on that has been started and David Aquilina has some of it already
available in text form.                                               
Infrstructure review
--------------------
As we've had quite a few infrastructure changes over the last year we wanted
to take the time and review the current state and if there are any areas where
we need changes or improvements. Kevin Fenzi from the infrastructure team came
over to talk with us about it in an overall fashion. Main points coming out if
it were:                                                                     
- EPEL needs action (new machine), especially to support RHEL-7 in the future
- Overall no issues otherwise, capacity and setup works really well         
- Storage might become an issue again, but NAS upgrade planed anyway       
- Backups for important boxes (koji & composer)                             
Continued optimization opportunities
------------------------------------
Just a brief look at what packages we currently compile as ppc64p7 and
brainstormed about a possible switch to ppc64p8 at some point.       
SDK tutorial/demo - discussion on applicability for Fedora
----------------------------------------------------------
Unfortunately had to skip that due to lack of time.
Collaberative articles (brainstorm session)
-------------------------------------------
Talked about more articles for the next releases, e.g. followup articles for
mock: What are repositories? among others things.                         
Project page rewrite/re-do/redesign
-----------------------------------
As the main wiki for Power in fp.org got a bit cluttered over the past 3
releases we thought it was time to clean that up. Basic idea is to just have a
bit of immediate information on the main page and have separate sub pages for
specific details.                                                           
EPEL discussion
---------------
Related to Infrstructure review there seems to be the wish to drop Power
support for EPEL at some point. Discussed again with Kevin what we'd need to
do to keep that working. Mainly need a current HW for the buildsystem for it.
Releng discussion w/ primary (process & schedule, checklist items)
------------------------------------------------------------------
We discussed various topics around release engineering with Dennis Gilmore
from rel-eng and Brendan Conoboy from the ARM teams.
Here the summary of the subtopics we had:
- Checklist/release criteria: Which does primary care about, which not?
        - As long as it's sane, primary doesn't care about our checklist
- Scheduling - what are the milestones we need to sync on? how long does it
  take for content to hit the mirrors?
        - Content has to be staged for mirrors on Friday for a Tuesday release
        - Must release on a Tuesday (preferrably announcing at 10a Eastern)
- Signing - messy & unreliable. What can we do to help debug?
        - Packet trace between client & bridge needed
        - bodhi messier than we thought (requires multiple sign & mash
          attempts even on primary)
        - just deal with it.
- koji garbage collection?
        - script still needs to be written to sync stages of gc b/w primary &
          secondary
- koji-shadow strategy?
        - shadowing stable and -updates-testing simultaneously is still sound
- -override tag sync between primary and secondary?
        - no complaints if we run our own version of the sync tag script for
          the override tags, but listening on the fedmsg message bus might be
          better (but would need new tooling)
- (infra?) how does primary monitor koji, storage, buildbranched, etc?
        - infra might be able to hook up basic system monitoring for us
        - cronjob output goes to dgilmore, should probably set up an
          arch-releng list.
Bug review
----------
Started doing that early Sunday but only did a brief overview and some initial
pruning of old bugs or bugs that we knew were already fixed or weren't
relevant anymore. Will do more individual pruning in the upcoming weeks via
irc meetings.
Statememt of Direction
----------------------
The Fedora Secondary Arch for Power team intends to focus and deliver the
following items for Fedora 19:
- Deliver a release for Power systems installable on typical currently
  available 64bit hardware
- All packages will be built with general 64bit Power support
- Higher optimized builds for a select list of components available as
  additional ppc64p7 arch packages
- Deliver 32bit compatibility packages for all components
- No 32bit install trees and DVD ISOs
- Sync development with the primary schedule
- Ensure release criteria are met for all milestones




[[Category:Arch-specific SIGs]][[Category:SIGs]]
[[Category:Arch-specific SIGs]][[Category:SIGs]]
[[Category:Fedora special-interest groups|PowerPC]]
[[Category:Fedora special-interest groups|PowerPC]]

Latest revision as of 13:40, 31 January 2013

Summary

Members of the Fedora on PowerPC project will be attending the 2013 FUDCon event in Lawrence, KS. Currently there aren't any real sessions planned, it'll be more like a discussion. This page is intended to gather topics we'd like to cover during the FUDCo. Please see below for further details.

Audience

Everyone who is interested in PowerPC hardware, i.e. IBM PowerPC hardware. Even though the Fedora on PPC project focuses on IBM hardware, we'd like to also invite people with Apple PPC hardware to join us to discuss how they can help us with the project. If we don't get some developers who are willing to provide patches soon, we might have to drop support for these older hardware due to lack of manpower to maintain it.

Contacts

We're online most of the time in #fedora-ppc on Freenode IRC, just post your question there and wait for someone to answer.

Schedule

Friday Saturday Sunday Unscheduled
F18 retrospective Releng meeting? Bug review EPEL discussion
F19 supported hardware Secondary Arch process Releng meeting? Collaborative Articles
grub2 on media Arch:PPC page rewrite Test Templates
SDK tutorial More optimization opportunities?
Infra review Write F18 release notes

Topics:

  • Introductions, not everyone from the PPC team has been at the last meeting in Blacksburg
  • Retrospective F18, what do we need to improve for F19
    • more communication, i.e. more communication on the fedora-ppc mailinglist
    • more communication with the other secondary arch maintainers about common problems, tools, processes, involvement etc.
  • Supported archs/hardware for F19
  • grub2 as bootloader on install media (DVD)
  • Discuss use of test templates as used on primary: https://fedoraproject.org/wiki/QA:Fedora_18_Install_Results_Template
  • Ideas on how to improve the visibility of the whole process (koji-shadow logs, signing progress, updates, etc.)
  • Infrstructure review
  • Continued optimization opportunities
  • SDK tutorial/demo - discussion on applicability for Fedora
  • Collaberative articles (brainstorm session)
  • Project page rewrite/re-do/redesign
  • EPEL discussion
  • should we install 32bit packages at all or shall we copy x86_64 and leave 32bit for the user to install manually after the system install ?
  • Releng discussion w/ primary (process & schedule, checklist items)
  • Bug review

Full notes of discussions at the meeting

Hi everyone.

As the Fedora Secondary Arch Team for Power met again this year at the North American FUDCon, this year in Lawrence we've put together a pretty long list of topics we wanted to cover and discuss during those days. Details were available here https://fedoraproject.org/wiki/Architectures/PowerPC/Meetings/FUDCon_Lawrence_2013 which was linked from the main secondary arch wiki for Power on fp.org the last few months.

The team meeting at Lawrence were:

Phil Knirsch Karsten Hopp David Aquilina Brent Baude Mark Hamzy Gustavo Luiz Duarte

+ special guests appearances from the Fedora release engineering, infrastructure and the ARM teams


Lets dive into the various topics we dicussed and what came out of it.


F18 retrospective


The main point here was to list the several good and bad things we noticed during the F18 development so we could repeate the good ones and fix the bad ones. Here the final list:

Good:

- Managed to hit almost all target dates
- Kept schedule in sync with primary    
- Almost passed final RC checklist for our RC4 (just 2 bugs)
- Overall only 2 power specific bugs                        

Bad:

- Signing was/is still an issue:
  o Reliability                 
  o Process                     
- Detected Custom paritioning bug specific for Power very late during beta
  testing, resulting in a delay there                                     
- Multipath needed lots of cycles and attention to get working properly   

Regarding the bad things we've had several discussions:

For the signing issues we had we've already started to address some of the know issues during the development for F18. The main things that still bugged us were the reliability and the overall process. For reliability we'll be working with Miloslav Trmac to provide network dumps and error messages. Several from the team will also be at the DevConf in Brno in February to work with him on this. Regarding the process issues we've discussed several options and came up with the following flow:

1) Sign packages for tag X. Returns list of all signed packages for tag X 2) Using this list run mesh to compose tree 3) Depending on tree (updates or release) create images and push bits

This will ensure that all packages that get pushed will always be signed on the first go. It will run the chance that a push on a certain day might miss a few packages that were just built, but we think the benefit from always only pushing signed bits more than outweights this.

Next up the late discovery of a critical bug for Beta. I really helped to begin with that we stuck to the primary schedule as closely as possible. That allowed us to get composes and images really early on. But in this specific instance this didn't really help as it took us several days until we realized that this was actually a Power specific issue (with Prep partitions), which meant the fix for it would be late for Beta. On the other hand it allowed us to test the updates.img functionality even on the install media. To ensure we cover the most important scenarios we'll be moving towards using the testing templates as primary already does, of course with slight modifications and additions. Additionally providing a one-stop pointer on the arch wiki to learn about the current state of builds, signing, composes and install tests using a dashboard style will help keep everyone informed about all important stages.

Last up was multipath. We've hit that during the late Alpha testing and noticed that multipath wasn't on any of the criterias during Beta testing. We'll propose to add those to the criteria also for primary for upcoming Fedora releases.


Supported archs/hardware for F19


We discussed this for a bit and at the end came to the conculsion not to do to much changing there compared to Fedora 18:

- Compiler flags just as before for ppc64, generic 64bit optimization  
- For ppc we only provide Everything and Fedora trees, no more install media
  or images                                                                 
- Following primary make ppc64 pure, meaning installs won't contain ppc     
  packages anymore                                                          


grub2 as bootloader on install media (DVD)


The driving factor here was that with switching to grub2 for Fedora 18 we now have a modern bootloader for Power that allows lots of cool things, but for the media we've still used yaboot (similar to primary using syslinux as the media bootloader). As it turns out we already are doing that on primary for UEFI images, so this has already gotten some testing and the necessary code is already available. The plan is now to create experimental boot ISOs with grub2 to see if they work properly and write a proposal post Fedora 19 to switch to grub2 and touch base with primary releng & anaconda in the time in between to coordinate the efforts.


Discuss use of test templates as used on primary: https://fedoraproject.org/wiki/QA:Fedora_18_Install_Results_Template


This went pretty quickly as we all agreed that this was a really good idea. The list will certainly need some review and weeding/additions based on what we've seen (e.g. multipath install test case). Additionally the final goal is to link the release criteria with specific test cases where applicable. This will make testing and checking if we meet the release criteria much easier.


Ideas on how to improve the visibility of the whole process (koji-shadow logs, signing progress, updates, etc.) aka Dashboard


This topic was related to some of the issues we faced during Alpha and Beta testing. The main goal for the discussion about this was to improve information availability and visibility to everyone. So the idea is to create a dashboard online that contains all following current information:

- Compose Status (last ran, logs)                                             
- koji-shadow status (last ran, logs)                                         
- Signing Status (last run, signed vs. unsigned #s)                           
- buildbranched status                                                        
- Failed builds                                                               
- Testing Status (IBM QA? AutoQA? )                                           

Work on that has been started and David Aquilina has some of it already available in text form.


Infrstructure review


As we've had quite a few infrastructure changes over the last year we wanted to take the time and review the current state and if there are any areas where we need changes or improvements. Kevin Fenzi from the infrastructure team came over to talk with us about it in an overall fashion. Main points coming out if it were:

- EPEL needs action (new machine), especially to support RHEL-7 in the future
- Overall no issues otherwise, capacity and setup works really well          
- Storage might become an issue again, but NAS upgrade planed anyway         
- Backups for important boxes (koji & composer)                              


Continued optimization opportunities


Just a brief look at what packages we currently compile as ppc64p7 and brainstormed about a possible switch to ppc64p8 at some point.


SDK tutorial/demo - discussion on applicability for Fedora


Unfortunately had to skip that due to lack of time.


Collaberative articles (brainstorm session)


Talked about more articles for the next releases, e.g. followup articles for mock: What are repositories? among others things.


Project page rewrite/re-do/redesign


As the main wiki for Power in fp.org got a bit cluttered over the past 3 releases we thought it was time to clean that up. Basic idea is to just have a bit of immediate information on the main page and have separate sub pages for specific details.


EPEL discussion


Related to Infrstructure review there seems to be the wish to drop Power support for EPEL at some point. Discussed again with Kevin what we'd need to do to keep that working. Mainly need a current HW for the buildsystem for it.


Releng discussion w/ primary (process & schedule, checklist items)


We discussed various topics around release engineering with Dennis Gilmore from rel-eng and Brendan Conoboy from the ARM teams.

Here the summary of the subtopics we had:

- Checklist/release criteria: Which does primary care about, which not?

       - As long as it's sane, primary doesn't care about our checklist

- Scheduling - what are the milestones we need to sync on? how long does it

 take for content to hit the mirrors?
       - Content has to be staged for mirrors on Friday for a Tuesday release
       - Must release on a Tuesday (preferrably announcing at 10a Eastern)

- Signing - messy & unreliable. What can we do to help debug?

       - Packet trace between client & bridge needed
       - bodhi messier than we thought (requires multiple sign & mash
         attempts even on primary)
       - just deal with it.

- koji garbage collection?

       - script still needs to be written to sync stages of gc b/w primary &
         secondary

- koji-shadow strategy?

       - shadowing stable and -updates-testing simultaneously is still sound

- -override tag sync between primary and secondary?

       - no complaints if we run our own version of the sync tag script for
         the override tags, but listening on the fedmsg message bus might be
         better (but would need new tooling)

- (infra?) how does primary monitor koji, storage, buildbranched, etc?

       - infra might be able to hook up basic system monitoring for us
       - cronjob output goes to dgilmore, should probably set up an
         arch-releng list.

Bug review


Started doing that early Sunday but only did a brief overview and some initial pruning of old bugs or bugs that we knew were already fixed or weren't relevant anymore. Will do more individual pruning in the upcoming weeks via irc meetings.


Statememt of Direction


The Fedora Secondary Arch for Power team intends to focus and deliver the following items for Fedora 19:

- Deliver a release for Power systems installable on typical currently
  available 64bit hardware
- All packages will be built with general 64bit Power support
- Higher optimized builds for a select list of components available as
  additional ppc64p7 arch packages
- Deliver 32bit compatibility packages for all components
- No 32bit install trees and DVD ISOs
- Sync development with the primary schedule
- Ensure release criteria are met for all milestones