From Fedora Project Wiki

Fedora Release Engineering Meeting - 2009-04-06

Rawhide status

  • PPC is broken due to sqlite causing yum to crash
  • x86 (32-bit) is broken due to weird rpm error when building images

These issues would threaten the availability of a snapshot for this week.

Beta wrapup

Known issues:

  • KDE x86_64 live was accidentally x86_32
    • new one being built, not uploaded yet
  • Source ISOs were wrong on torrent
    • new scripts will double-check for this
  • XFCE is broken, but not a process issue
  • similarly, PPC


Current plan is to just ship Live, Live-KDE, and Live-XFCE for snapshot release; shipping everything is just too much data to push around. Proposal will be taken to the Spins SIG.


We are not using sigul yet. Volnteers to help with deployment of it welcome. Repodata signing is messy.

IRC Log transcript

---f13 has changed the topic to: Fedora Release Engineering Meeting - Roll Call Apr 06 14:11
-->sdziallas ( has joined #fedora-meeting Apr 06 14:11
fhornain Sorry for that guys. Apr 06 14:11
vincentvdk bye Apr 06 14:12
f13 ping: notting dgilmore jwb lmacken wwoods poelcat rdieter warren spot Apr 06 14:12
*wwoods here Apr 06 14:12
warren pong Apr 06 14:12
*notting is here Apr 06 14:12
*poelcat here Apr 06 14:13
*dgilmore is here Apr 06 14:13
*jwb Apr 06 14:13
f13 k, we need somebody to be the note taker today Apr 06 14:13
*spot is here Apr 06 14:13
f13 Agenda is rawhide issues, beta wrapup, snapshot 1, signing, open floor Apr 06 14:14
-->ajoian (n=ajoian@ has joined #fedora-meeting Apr 06 14:14
*poelcat would like to propose Apr 06 14:14
f13 ok, we can tack that on too Apr 06 14:15
notting i can take notes Apr 06 14:15
f13 thanks bill Apr 06 14:15
---f13 has changed the topic to: Fedora releng - rawhide Apr 06 14:16
f13 We've got a number of issues going on in rawhide that is preventing the tree from being installable. Apr 06 14:16
f13 On ppc we've got a sqlite segfault in any yum action Apr 06 14:16
-->sdziallas_ ( has joined #fedora-meeting Apr 06 14:16
dgilmore fun Apr 06 14:17
notting oof. can we block that version? Apr 06 14:17
f13 Apr 06 14:17
buggbot Bug 494396: urgent, urgent, ---, pmatilai, NEW, 3.6.12 causes segfaults in yum on ppc Apr 06 14:17
f13 blocking it would essentially be a downgrade which is a FESCo no no Apr 06 14:17
f13 we'd have to epoch the previous version and build it Apr 06 14:17
f13 even if we cleared that, today we had libdrm and xorg-x11-drv-ati packages requiring a kernel that wasn't even built yet at the time of the rawhide compose Apr 06 14:18
notting right, but any ppc rawhide user is now dead in the water, unless i misunderstood Apr 06 14:18
wwoods so.. me and jwb? Apr 06 14:18
f13 notting: they were pretty dead in the water before this, at least as far as new installs were concerned. Apr 06 14:19
jwb wwoods, and j-rod Apr 06 14:19
jwb he converted Apr 06 14:19
poelcat and jlaska Apr 06 14:19
wwoods and jlaska too Apr 06 14:19
notting f13: but the buildsystem is only working because the repos aren't built on rawhide? Apr 06 14:19
f13 right, once inside the chroot we don't use the repodata for anything Apr 06 14:19
f13 only composes do Apr 06 14:19
f13 the kernel dep issue is resolved now that the kernel is built, but we need to make airlied aware of the damage he caused Apr 06 14:20
<--ajoian has quit (Remote closed the connection) Apr 06 14:20
-->dkovalsk ( has joined #fedora-meeting Apr 06 14:20
dgilmore f13: he needs to coordinate that much betetr Apr 06 14:20
jwb how about partioning? Apr 06 14:20
dgilmore better Apr 06 14:20
f13 and even if /that/ was fixed, we still have a problem with creating i386 install trees Apr 06 14:21
f13 every x86 host I've tried to run buildinstall on results in some very bizarre rpmdb corruption Apr 06 14:21
f13 OSError: Got an error from /usr/lib/anaconda-runtime/buildinstall: error: Unterminated {: {_%{_keyringpath}/*.k Apr 06 14:21
f13 0< / Apr 06 14:21
f13 error: /: reading of public key failed. Apr 06 14:21
f13 error: Unterminated {: {_%{_keyringpath}/*.k Apr 06 14:21
f13 0< / Apr 06 14:21
f13 error: /: reading of public key failed. Apr 06 14:21
f13 wait a tick, this may not be rpmdb now that I think about it Apr 06 14:22
dgilmore looks kinda like a typo Apr 06 14:23
f13 I don't see it on other arches though Apr 06 14:23
f13 and "keyringpath" doesn't appear anywhere in the anaconda git tree Apr 06 14:23
wwoods that looks like rpm macro badness Apr 06 14:24
-->lfoppiano (n=lfoppian@fedora/lfoppiano) has joined #fedora-meeting Apr 06 14:24
f13 anyway, that needs more investigation Apr 06 14:25
f13 it only seems to happen with i386, not with x86_64 Apr 06 14:25
f13 and not with ppc, once ppc has the right sqlite Apr 06 14:25
f13 so that's the extent of the known rawhide issues. Any other comments on rawhide? Apr 06 14:26
-->mdomsch ( has joined #fedora-meeting Apr 06 14:26
notting wwoods: macro's only defined in /usr/lib/rpm/macros - nothing else that i can find references it Apr 06 14:26
wwoods misparsing? mangled on installation? Apr 06 14:26
notting wwoods: or memory corruption in librpm! Apr 06 14:26
wwoods dun dun dunnnn Apr 06 14:26
notting f13: just to state for the record - that means no snapshot last week? Apr 06 14:27
f13 we don't do a snapshot on the same week as beta Apr 06 14:27
dgilmore ./lib/rpmts.c: char *pkpath = rpmGetPath(ts->rootDir, "%{_keyringpath}/*.key", NULL); Apr 06 14:27
<--fhornain has quit ("Leaving") Apr 06 14:28
f13 notting: so there was no snapshot last week, that is correct Apr 06 14:28
<--vincentvdk has quit ("Leaving.") Apr 06 14:28
f13 we'd have to clear up these issues in order to have a snapshot this week Apr 06 14:28
<--benedictus has quit (Client Quit) Apr 06 14:30
notting ok. next issue? Apr 06 14:31
<--sdziallas has quit (Read error: 101 (Network is unreachable)) Apr 06 14:32
-->sonargal (n=Test@fedora/SonarGal) has joined #fedora-meeting Apr 06 14:32
f13 I think that's the end of the known rawhide issues, aside from the continued anaconda storage work Apr 06 14:33
---f13 has changed the topic to: Fedora releng - Beta wrapup Apr 06 14:33
f13 tail end of last week uncovered a number of issues with the beta compose Apr 06 14:33
f13 seems no matter how much time we have, we (I mean I) screw something up Apr 06 14:33
notting aside from ppc being generally fubar, what was up? Apr 06 14:34
f13 the source isos on torrent were the wrong isos, and the source checksum on the mirrors was the wrong checksum file Apr 06 14:34
f13 and the Live KDE x86_64 image was actually built with 32bit packages. Apr 06 14:34
f13 I fixed the checksum on the mirrors, and was in the process of uploading the correct source isos to torrent Apr 06 14:34
-->cebbert (n=cebbert@fedora/cebbert) has joined #fedora-meeting Apr 06 14:34
f13 I built a new live-KDE as well, but haven't uploaded it anywhere yet Apr 06 14:35
f13 I added a test case to verify the checksums file in the tree, important to do since we shuffle them to get them signed Apr 06 14:37
-->thomasj_ ( has joined #fedora-meeting Apr 06 14:37
f13 and I need to re-look at livecd-creator to patch it to that we can use the same cachedir for both 32bit composes and 64bit composes, instead of using a different cachedir for each (which is what led to the miscompose) Apr 06 14:37
<--CheekyBoinc has quit (Remote closed the connection) Apr 06 14:38
---sdziallas_ is now known as sdziallas Apr 06 14:38
-->gregdek (n=gdk@nat/redhat/x-d7ff5ad07fd6dd6f) has joined #fedora-meeting Apr 06 14:39
f13 not much else to say here from me. Anybody else? Apr 06 14:39
dgilmore nope Apr 06 14:40
-->CheekyBoinc (n=CheekyBo@fedora/CheekyBoinc) has joined #fedora-meeting Apr 06 14:40
---f13 has changed the topic to: Fedora releng - Snapshot 1 Apr 06 14:40
f13 so we're supposed to do a snapshot this week. Apr 06 14:40
notting f13: oh, didn't nirik say something about xfce being busted? Apr 06 14:40
f13 I'll admit, I'm a bit fuzzy on these, in what all we're supposed to deliver Apr 06 14:40
f13 notting: yeah, its fubar due to something in the package set. Apr 06 14:40
f13 notting: when he composes with the same package set, he is able to reproduce the problem. Apr 06 14:41
nirik yeah, i686 seems to be busted, but I can't figure out why yet. Apr 06 14:41
notting f13: ah, ok. not a compose issue Apr 06 14:41
nirik it works fine with post beta rawhide. Apr 06 14:41
notting f13: erm, process issue Apr 06 14:41
nirik it doesn't seem to be. Apr 06 14:41
poelcat f13: there's also a list here Apr 06 14:42
f13 poelcat: yeah, but in the past I don't think we delivered every single spin in a snapshot Apr 06 14:43
f13 that's just entirely too much work/data to push around Apr 06 14:43
*nirik wonders if we should disable/remove the i686 Xfce from torrents. I guess I don't know for sure it fails for everyone. Apr 06 14:45
f13 looking at old torrent configs Apr 06 14:45
f13 nirik: is it only broken to install, or is it just not launchable? Apr 06 14:45
f13 what is broken in other words. Apr 06 14:45
f13 Looks like for snapshots we only did live images, live, live KDE Apr 06 14:45
nirik it boots, the kernel loads and then it hangs. You can control-alt-del and it will reboot, but it never gets to a desktop here at least. Apr 06 14:45
f13 and XFCE for spins Apr 06 14:45
wwoods maybe we can spread out the tasks over the snapshots Apr 06 14:46
f13 we only get 2 snapshots this time around Apr 06 14:46
wwoods e.g. snapshot1 we do live images, snapshot2 is DVDs, 3 is CD sets.. oh Apr 06 14:46
*nirik doesn't understand why i686 gnome would be working if it was the same packageset. Oh well, it's wacky. Apr 06 14:46
f13 and then the <bleep>ing preview release Apr 06 14:46
<--mitr ( has left #fedora-meeting ("Leaving") Apr 06 14:47
-->mitr ( has joined #fedora-meeting Apr 06 14:47
f13 wwoods: I just don't think there is enough time/bandwidth/people for that to do much good Apr 06 14:47
*poelcat notes there is no room for snap2 with final freeze 2009-04-14 Apr 06 14:47
poelcat resulting from slipped beta Apr 06 14:48
f13 hrm. we only slipped a week though Apr 06 14:48
f13 did we only have two snapshots scheduled to begin with? Apr 06 14:48
poelcat yes Apr 06 14:48
f13 interesting Apr 06 14:48
f13 so we've got one snapshot. Apr 06 14:49
f13 and then the fuller preview release after that Apr 06 14:49
f13 I'm OK with just doing the Live, Live-KDE, and Live-XFCE for the snapshot, unless any of the spins owners have a real desire to see a snapshot of their spins Apr 06 14:50
notting wfm Apr 06 14:51
<--fbijlsma has quit ("Leaving") Apr 06 14:52
wwoods sounds fine Apr 06 14:53
---knurd is now known as knurd_afk Apr 06 14:53
f13 ok. We should probably get the spins sig pinged with this plan Apr 06 14:54
f13 jwb: you go to that meeting right? Apr 06 14:54
jwb i got to very few meetings due to $work atm Apr 06 14:54
<--thomasj has quit (Connection timed out) Apr 06 14:54
jwb (for example, i haven't hardly been able to pay attention here) Apr 06 14:54
f13 poelcat: do you go to the spins meeting? Apr 06 14:55
poelcat f13: usually Apr 06 14:55
*nirik tries to always attend as well. Apr 06 14:57
jlaska poelcat: wwoods: sorry guys, was in meeting Apr 06 14:57
f13 could one of you take that proposal to the meeting? Apr 06 14:57
nirik sure, or how about an email to the spins list? Apr 06 14:58
-->mizmo (n=duffy@ has joined #fedora-meeting Apr 06 14:58
<--mizmo (n=duffy@ has left #fedora-meeting ("Leaving") Apr 06 14:59
---Marflow_afk is now known as Marflow Apr 06 14:59
<--mether_ has quit (Client Quit) Apr 06 14:59
f13 nirik: either way, so long as they get a chance to speak up Apr 06 15:00
<--drago01 has quit (Remote closed the connection) Apr 06 15:01
f13 anything else on snapshot1 ? Apr 06 15:01
nirik Does everything get composed for PR? Apr 06 15:02
f13 I think so, sadly Apr 06 15:03
---f13 has changed the topic to: Fedora releng - signing Apr 06 15:04
f13 we still aren't using sigul, and we should be Apr 06 15:04
f13 if anybody wants to help mitr and me with that, please let us know, otherwise it's going to be a "when we get time" kind of thing Apr 06 15:04
mitr I'll try to add key import this week Apr 06 15:05
f13 Even with out sigul, we have pretty heavy pressure to start signing repodata files Apr 06 15:06
f13 which means an extra step in updates pushing Apr 06 15:06
f13 we'd have to get the repomd.xml file to the signing box, create a detached sig for it, then get the detached sig over into the repodata/ dir for the repo in question Apr 06 15:08
f13 as the current updates pusher, jwb, that would add a rather annoying step in what you are doing Apr 06 15:08
f13 when I agreed to this, I was doing the updates pushes, and was willing to take on the extra monkey work. Apr 06 15:08
f13 I think jwb should have hte right to say yes/no on this matter Apr 06 15:08
notting pressure from....? Apr 06 15:10
*jwb reads Apr 06 15:10
jwb can i have more background on this? Apr 06 15:11
f13 notting: RH security team, yum folks. Apr 06 15:12
jwb we want to sign them because... Apr 06 15:12
f13 jwb: right now, repodata isn't signed, there is no real guarentee that the repodata on a mirror is the /correct/ repodata and not forged Apr 06 15:12
jwb yes. but that isn't a problem because the packages are signed, or? Apr 06 15:13
jwb or are you saying it is a problem Apr 06 15:13
f13 the security and yum guys are saying that this is a problem Apr 06 15:13
jwb you don't have to rehash here, but can you point me to lists to read? Apr 06 15:13
f13 bressers and geppeto I think are the two we'd want to talk to Apr 06 15:14
jwb ok Apr 06 15:14
jwb also, if repodata is signed, do the packages need to be? Apr 06 15:14
dgilmore f13: would it mean signing rawhide repodata also? Apr 06 15:14
mitr AFAICS: Apr 06 15:14
f13 I originally thought it was to avoid the stale repodata trick, keeping stale repodata in place to point to a signed but known vulnerable packages Apr 06 15:14
mitr 1) repo points to metalink Apr 06 15:14
mitr 2) metalink is served over https Apr 06 15:14
mitr 3) metalink contains repodata hash Apr 06 15:14
f13 but even with signed repodata you could still do that Apr 06 15:14
mitr => all we are (were?) missing is certificate verification in yum. Apr 06 15:15
mitr (and SHA-256 hashes in metalink, but that's not nearly as important) Apr 06 15:16
f13 mitr: well, we are using ssl to get the metalink Apr 06 15:16
f13 I'm not sure what added value the repodata being gpg signed adds Apr 06 15:16
mitr f13: ssl without certificate verification does not provide authenticity. Apr 06 15:17
f13 aren't we doing cert verification? Apr 06 15:17
mitr Last time I looked ( <= 1 month ago) we were not. I can recheck tomorrow. Apr 06 15:17
jwb i don't have a problem doing more monkey steps if they actually fix something Apr 06 15:17
jwb but i'd like to know what that something is Apr 06 15:18
f13 right, I would too Apr 06 15:18
jwb it will likely slow down the frequency of updates pushes to a degree, however that is not a bad thing Apr 06 15:18
f13 and we'll have to come up with a good easy way of doing the extra signing. Apr 06 15:18
jwb and with the ability to push security updates alone, not a huge deal Apr 06 15:18
notting f13: how would it get shoehorned into the push process, as it stands. wouldn't that require bodhi hackery? Apr 06 15:19
f13 it would require slight bodhi hackery Apr 06 15:19
mitr One risk that is fixed by signing/authenticating repodata is modifying the repodata to pull in known vulnerable packages as additional dependencies. Apr 06 15:19
f13 right now bodhi waits for the compose to finish and then rsyncs everythign to /pub/. It could wait for a .asc file to show up in each composed repo before doing the rsync Apr 06 15:19
notting ew. Apr 06 15:20
-->rdieter ( has joined #fedora-meeting Apr 06 15:21
f13 it already has a wait mode on waiting for repodata to hit the public mirror before it does announcements Apr 06 15:21
f13 in fact Apr 06 15:21
jwb well Apr 06 15:24
jwb it does, but it doesn't work Apr 06 15:24
f13 Apr 06 15:24
jwb it waits for it to show up on some netapp, but that netapp doesn't currently exist. so it waits for 5min for no real reason Apr 06 15:24
f13 that's news to me Apr 06 15:25
jwb we discovered that about a month ago when bodhi was in some kind of wait loop and DNS was fubar Apr 06 15:25
jwb mmcgrath told us the netapp was being shipped somewhere Apr 06 15:25
jwb i dunno if it's back Apr 06 15:25
<--Sonar_Guy has quit ("Leaving") Apr 06 15:25
f13 oh interesting Apr 06 15:26
jwb anyway, your point still stands. it already has logic to wait for $something Apr 06 15:26
mmcgrath it's not, I requested an ETA and they just told me the disks haven't arrived yet. Apr 06 15:26
f13 ok, so lets get you (jwb) and the yum folks together to work out what it is we're actually fixing and how to integrate it into our current push process Apr 06 15:29
-->josedamiangarri1 (n=damian@ has joined #fedora-meeting Apr 06 15:29
---thomasj_ is now known as thomasj Apr 06 15:29
f13 I think I was supposed to generate .asc files for the beta tree, or preview tree, or some tree Apr 06 15:29
<--RadicalRo has quit (Remote closed the connection) Apr 06 15:31
-->giallu (n=giallu@fedora/giallu) has joined #fedora-meeting Apr 06 15:31
<--linuxguru has quit ("linuxguru crashes") Apr 06 15:31
f13 We're over time, the other topics were F12 schedule and open floor Apr 06 15:33
f13 F12 schedule would take a while, probably not good to get into now. Apr 06 15:33
f13 anything pressing before we call it a meeting? Apr 06 15:33
f13 guess not Apr 06 15:35
---f13 has changed the topic to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See for meeting schedule Apr 06 15:35
f13 thanks all! Apr 06 15:35

Generated by 2.7 by Marius Gedminas - find it at!