From Fedora Project Wiki

Fedora Release Engineering Meeting :: Monday 2008-03-03

Beta Freeze

  • unresolved issues in X, perl, and mkinitrd
  • koji slowness also contributing to slip
  • delaying freeze and corresponding beta release by one week
  • no proposed change to GA date at this time--discuss next week

IRC Transcript

2008-03-03-14:07-meeting-snapshot

-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Roll Call13:00
f13ping: notting jeremy rdieter wwoods lmacken warren jwb poelcat spot13:00
warrenmeh13:00
* spot is here13:01
* lmacken is here13:01
* jeremy 13:01
* rdieter here kinda sorta13:01
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Fedora 9 Beta13:01
f13Tonight, we freeze.13:01
* poelcat here13:01
nottingf13: tomorrow, we dine in hell?13:02
f13I already have f9-beta being populated from dist-f9.  I'll commit a change tonight so that dist-rawhide pulls from it13:02
f13notting: pretty much.13:02
f13tomorrow's rawhide will be the frozen content13:02
* wwoods here13:02
f13I'm also going to start signing later today (once the updates push is done)13:02
f13since people have been slamming planes into the runway the last few days (and today) the next couple days are likely going to be sweeping the decks of all the wreckage.13:03
f13we're keeping our eyes on the X stuff, as we may just maybe have to delay the release a bit for good X bits to land13:03
f13but delaying the release won't be the same as keeping rawhide frozen.  If we get to the point where we're just waiting for X, we'll let rawhide move on and just wait for X to land to do the final compose+push13:04
f13We'll ask wwoods to hook us up with a wiki page to start tracking testing and issues13:05
f13and we already have an F9Beta bug13:05
wwoodsyup13:05
jeremywith a fair number of open things on it already13:05
f13right13:05
f13we should be adding more as we see them.13:05
f13I'm asking jwb to do test composes of ppc, but I will take care of pushing the final buttons to make the content show up13:07
f13we may use machines in PHX for it, if mmcgrath has the resources, otherwise we'll do it in BOS again.13:07
f13but since the netapp drop point is in PHX, PHX would be preferred.13:07
f13that's about all I have initially for Beta stuff, are there any questions/issues that you'd like discussed?13:08
nottingX bad13:09
f13yes13:09
spotf13: any chance of perl 5.10.0 slipping in?13:09
* notting looks at the clock13:09
f13spot: how close are you?13:09
wwoodsOh uh. Someone should probably mention that boot.iso is gone baby gone13:09
f13oh yeah13:09
jeremywwoods: yeah, need to do some release note stuff13:09
wwoodsand that netinst.iso and/or SNAKE are the easy ways to do rawhide installs13:10
f13I need to throw some code at pungi to move/copy netinst.iso from images/ out to iso/13:10
nottingneed to add relnotes about upstart13:10
spotf13: maybe 330 packages done at this point13:10
spotout of about 80013:10
spotslowdown is koji, not me.13:11
sticksternotting: http://fedoraproject.org/wiki/Docs/Beats/SystemDaemons13:11
sticksterI added a short note there, but it might be able to use clarification.13:11
spotkoji just can't build fast enough, its doing one an hour, on average13:11
nottingf13:  i may be of surprisingly little use this week, i appear to have caught the flu13:11
warrennotting, Sinucleanse!13:12
f13spot: there aren't any places you can make better chain groups instead of single links?13:12
nottingwarren: um, yeah. i'm sure that will do wonders for aches and pains ;)13:12
spotf13: not until i get about 400 done.13:12
nottingstickster: oh dear. someone changed a script to be started from upstart? must beat them ;)13:12
sticksternotting: I thought someone mentioned zeroconf in some idle chitchat here or f-devel-l13:13
sticksterIt's not firsthand knowledge, feel free to delete.13:13
wwoodsI wonder if initscripts can send out upstart events when they do their thing13:13
nottingwwoods: see the linked bug from the 'upstart' tracker about cross-system deps13:14
* stickster & # derailing meeting--13:14
nottingwwoods: sysv -> upstart is doable. upstart -> sysv is hard13:14
f13spot: ok, it's going to be close.  I don't like our options here.  Slip beta, land it during the freeze and cross fingers, land it after beta and cross fingers, delay until F1013:14
wwoodsyeah, sorry, abandoning tangent13:14
* jeremy is strongly against "land it after beta"13:14
spoti would really hate to delay13:14
spotif i had a few more days, i could get it done13:14
f13that leaves us either with slipping beta freeze, or landing it after the freeze and hoping beta will be ok13:15
poelcatf13: do we know any more about the state of X which could affect beta?13:15
* jwb is here now13:15
f13poelcat: many drivers aren't working right now, ajax is working on it as fast as he can13:15
spotX is having rather annoying keymapping issues13:16
f13poelcat: but ajax/jrb will know more details.13:16
warrenHave we had installable rawhide at all in the past weeks?13:16
spot(hint: don't select multiple items in evolution)13:16
f13warren: yes13:16
jeremyspot: the keymap problems are worse on upgrades than new installs -- and you can change your config to make it work.  but yeah, still not pretty13:16
wwoodstoday's is installable AFAIK13:16
spotjeremy: any workarounds would be good to document13:16
wwoods650/1050 packages installed; I'll let you know if it reboots OK13:17
jeremywwoods: still not rebootable due to the mkinitrd thing.  but I guess I'm going to end up looking at that this afternoon since peter is hosed by rhel13:17
f13provided you don't use lvm or label= mounting methods13:17
wwoodsfucking seriously? still no progress on that mkinitrd bug?13:17
f13to be fair, it was the weekend13:18
warrenjeremy, what needs to be done?13:18
warrenbug #?13:18
nottingwwoods: iscsi and 5.2 happened13:19
wwoodswarren: bug 43522813:19
bzbotBug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=435228 low, low, ---, Peter Jones, NEW , mkinitrd doesn't grab dm modules for LVs listed by LABEL or UUID13:19
warrenis the nash and dhcp libs dep problem fixed?13:19
f13wasn't aware of such a problem.13:19
wwoodsthe second patch, by the way, does *not* fix the problem13:20
jeremywarren: yes.13:20
f13right, so we clearly need work on this issue13:21
f13that's not all that surprising13:21
warrenIsn't it pretty clear that we have to slip the freeze?13:21
wwoodsso: a bug I submitted a fix for last week has made all rawhide installs non-bootable and therefore we have no pre-freeze testing.13:21
warrenperl is a huge change13:21
f13warren: slipping the freeze just means more shit gets broken.13:22
f13this happens /every/ /single/ /freeze/13:22
jwbwhen is freeze date again?13:22
wwoodsI don't think slipping the freeze is a good idea. Lengthening it might be necessary.13:22
f13jwb: tonight13:22
jwbi agree with wwoods13:22
f13right, we always have the option of lengthening the freeze13:23
f13and that doesn't seem unreasonable in this case.13:23
f13especially for perl to land13:23
f13bah13:23
* f13 wipes wagamamas off his laptop13:24
wwoodsthat's my gut feeling on the matter, anyway13:24
wwoodsanyway I spent a couple of days hacking on the mkinitrd thing so feel free to ask me if you want logs / debugging info /etc13:24
nottingif we want perl 5.10 in f9, we should land it now13:24
f13so here is my take on it13:25
f13I don't like announcing ahead of time that we're slipping a release date, or lengthening the freeze13:25
f13this just invites people to take their time and land stuff at the last possible moment, not giving us any time to verify integration and test.13:25
f13so I'd rather announce that we're going ahead as scheduled, and only announce changes if they become absolutely necessary13:26
f13notting: I think we all agree on that13:26
* spot wants to land it, its just a matter of getting the builds done13:26
wwoodsis it just a matter of rebuilding, or are there lots of fixes necessary?13:26
jeremyf13: the flip side is that people pull of heroics only to get told "haha, just kidding.  you had more time"13:27
f13Proposal: Maintain schedule as is.  Keep eye on Perl builds for timing.  Also let ajax know if our contengency plan of delaying the release ( he should already be aware of this via jrb ).  Delay the release of the beta for Perl/X/Whatever else only when it becomes necessary.13:27
warrenjeremy, +113:28
warrenjeremy, that also means that some people might give up when they really could have fixed something in time.13:28
f13it is a balancing act.13:28
spotwwoods: at this point, almost no fixes13:29
spotjust rebuilds13:29
jwbspot, is this being done on the side still?13:30
spotjwb: in dist-f9-perl13:30
jwband the plan is to mass tag those builds to dist-f9?13:30
* spot nods13:30
jwbk...13:30
jeremyf13: -1.13:30
warrenf13, -113:31
warrenjeremy, what do you suggest instead though?13:31
jwbi'll refrain from voting at the moment13:31
jeremyI think that we need to take a real look at where we are and actually adjust the schedule based on what we really think rather than saying "lalalalala" when we know that there's really _zero_ chance of us having a real freeze of any kind tonight13:31
poelcatjeremy: +1 13:32
f13jeremy: my only problem with that is the schedule has been known well in advance.  Without any sort of freeze, I have very little confidence that anything will actually improve13:32
f13because, this shit has been /bad/ for a long time13:32
jwbf13 has a point13:33
f13we've really only had two relatively stable installable points.  Just after F8 release, and the Alpha freeze~13:33
jwbwould it be possible to grant tag privileges to ajax for X stuff?13:34
jeremyand being unrealistic changes anything how exactly?13:34
f13jeremy: I'm not sure of any other way to get the message across to fix shit and stop breaking things13:34
wwoodsdoesn't a freeze do that?13:35
f13perhaps we need to move the feature freeze back before the beta freeze13:35
f13because beta just tends to be the crap people land last minute13:35
jwbthis isn't a beta13:35
f13but I'm game for jeremy's suggestion13:35
* warren notes that there are different kinds of "features", many don't threaten stability.13:35
f13jeremy: Looking at what we have now, what would you consider realistic to do?13:35
* f13 notes that feature freezes aren't just for stability either.13:36
wwoodsyes, but many of the things that *are* causing instability are feature-related (xserver 1.5, encrypted filesystems, etc)13:36
f13It seems that the two main pain points are X and mkinitrd13:37
jeremywwoods: what instability is encrypted filesystems causing?13:37
f13and then the desire to see perl land.13:37
f13do we have a good idea of what other than those things are threatening beta?13:37
lmacken13:30  spot> jwb: in dist-f9-perl13:37
lmackenerm13:37
warrenthe size of perl's changes doesn't worry you?13:37
wwoodsjeremy: it kept people from booting for a while and broke rescue. current mkinitrd junk isn't encryption-related13:38
warrenit might build but how well does it work?13:38
spotthey really only affect perl, and about 10 other packages.13:38
wwoodsso I guess I should have said "*have been* causing instability"13:38
f13warren: that's what beta is /for/13:38
wwoodssorry for confusion!13:38
spotin my testing, those other packages work fine with 5.1013:38
warrennice13:38
f13warren: to get testing on these features, not to say "everything is fine, have fun!"13:38
wwoodsalso encryption in rescue mode totally works now, so yeah.. encryption is totally A-OK fine13:40
f13so what we're really talking about is X, perl, and mkinitrd.13:41
f13I'd be willing to bet we have a working mkinitrd before the end of the day, if people don't bug jeremy for the rest of the day13:41
f13perl is going as fast as it can, and ajax is working on X, we need to leave him alone too13:41
f13what else is going on?13:42
* spot could be done in a week, assuming the builders keep going13:42
spot(worst case)13:42
jeremyperl needing a week sounds like we either need to a) drop perl 5.10 or b) slip a week.  I don't really see how we're having any other conversation13:42
spotit only needs a week because the builders only spit out a package an hour13:43
jwbif we drop perl, is it dropped for RC/Final as well?13:43
wwoodscan't we freeze now and just lengthen the freeze for perl and xserver?13:44
f13jwb: yes, if it doesn't go into beta, it doesn't go in.13:44
wwoodsI'm trying to find some way to express how desperately I want to spend some time stabilizing rawhide so people can actually test it13:44
jwbwwoods, that's what i was getting at with giving ajax tagging privileges13:44
jeremywwoods: I'd rather let us be honest.  there are some other things which are almost certainly on the fence or people pushing hard at the last minute13:44
wwoodsWhich is kind of the point of the freeze13:44
f13well, I'm feeling of the opinion that my lofty goal of 1 week freezes is pretty trashed by now13:45
f13and I think we need to extend the amount of days we spend in freeze mode anyway13:45
f13because the 1 week stuff is killing me.13:45
f13so either way I'd want to extend freezes by a week~13:46
warrenf13, prior to you arriving at RH, very long freezes seemed to kill people's motivation too.13:46
warrenf13, but back then there were fewer people to tag things in, so maybe that is less of an issue now.13:46
jwbi think he knows that13:46
f13warren: that was when all freezes were blocking, and there was less communication13:46
jeremylong freezes just lead to more problems after the freeze13:46
wwoodsI know long freezes suck. Also, on second thought: most of the instability we're dealing with is in things like kernel, X, etc13:47
f13alright, so it seems like all my ideas are shit today, jeremy, what are we going to do?13:47
wwoodsstuff we'd be accepting during a freeze anyway13:47
wwoodsso maybe a longer freeze - while sounding good in theory - would just piss off all the devs that are trying to work on otherwise-stable stuff13:47
warrenlonger freeze pisses off devs who work on stuff that is irrelevant to stability13:48
* jeremy really thinks we just need to bite the bullet and push back the freeze. the stuff that's causing us the pain is the stuff that's working on being stabilized 13:48
wwoodswarren: right13:48
f13jeremy: I'm ok with that, I'm talking about something else now.  The fact that freeze on tuesday && have tree ready for mirrors by next Monday is just unrealistic13:48
f13especially if we want any sort of testing beyond "hey look, it installed" on the bits we hand to the mirrors13:49
jeremyf13: I think that's one of the points of being a community-focused distro.  we can't do all the testing before we call it a beta13:49
jeremywhich, yes, leads to the beta having more problems than in the Old Days (tm)13:50
f13jeremy: thus far, we can't really do /any/ testing either.13:50
f13but now we're just arguing because people are pissed at the situation.13:50
jeremyf13: that's a little unfair -- there are a number of things that are getting tested.  even on the install side of things  (cf I fixed several bugs yesterday that people were finding and reporting)13:50
f13so.13:51
f13Proposal: Delay freeze by a week, keep it the same length as previously scheduled.13:51
warrenproposed amendment: announce it today, but warn people to not add things that could cause further instability13:51
jwbor what?13:52
warrenjwb, pointy sticks13:52
wwoodsyelldown warhellride13:52
jeremyjwb: those who add instability will be called out in public13:52
jeremyjwb: that's the only real "or what" that we have -- peer pressure13:52
warrenjwb, Point to the crater that used to be the orphanarium.13:52
wwoodswe will seriously doubt your commitment to sparklemotion13:52
jwbi'm just going to stay silent because i have nothing nice to say about any of that13:53
wwoodsfair 'nuff13:54
wwoodsyou may still reserve the right to say "toldja so" if something really terrible actually happens13:54
jwbno, that's not what i'm thinking13:54
f13can we at least agree to the proposal?13:56
jwbbasically, public shaming didn't work for the first beta freeze date.  what makes us think it'll work now?13:56
jwbf13, +113:56
jeremyf13: +113:56
wwoods+1 to the proposal13:56
nottingf13: +1, i suppose. not really any other option13:57
spot+113:57
lmackenf13: +113:57
f13alright, that's probably good for Beta.13:57
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Open Discussion13:57
nottingadjusting final as well?13:57
f13the floor is open13:57
f13notting: I wish we wouldn't have to13:58
f13or I'd rather not that is13:58
jwbi wanted to talk about spins, but that's going to take too long13:58
f13notting: if we don't slip final, that still gives us 3 weeks from beta release to final freee13:59
warrenlet's discuss the final schedule next week?13:59
f13that doesn't seem terribly unreasonable.13:59
jeremyyeah, I'd like to get to looking at the doom that is sysfs14:01
jwbhuh?14:01
nottingjeremy: that's a little large of a project for a few weeks14:01
nottingdoom_level(sysfs) == MAX_INT14:02
jeremytrue... and "want" is probably strong14:02
jeremybut I'd like to get the @!#$ bug fixed14:02
f13alright, close of meeting14:03

Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!