m (1 revision(s)) |
(→IRC Transcript: Removed some HTML markup.) |
||
Line 28: | Line 28: | ||
== IRC Transcript == | == IRC Transcript == | ||
<table class="irclog"> | <table class="irclog"> | ||
<tr id="t12:57"><td class="other" colspan="2">{{Template:Tip}} f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Buildroot contents for updates - JesseKeating</td><td><a href="#t12:57" class="time">12:57</a></td></tr> | <tr id="t12:57"><td class="other" colspan="2">{{Template:Tip}} f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Buildroot contents for updates - JesseKeating</td><td><a href="#t12:57" class="time">12:57</a></td></tr> | ||
Line 440: | Line 427: | ||
<div class="generatedby"> | <div class="generatedby"> | ||
<p>Generated by irclog2html.py 2.3 by | <p>Generated by irclog2html.py 2.3 by [mailto:marius@pov.lt Marius Gedminas] | ||
- find it at | - find it at [http://mg.pov.lt/irclog2html/ mg.pov.lt]!</p> | ||
</div> | </div> | ||
Latest revision as of 14:51, 18 February 2014
Fedora Release Engineering Meeting :: Monday 11-Jun-2007
Buildroot contents for updates
- The buildroot used for fedora 7 updates building is not self updating. It only contains things from f7-gold, and stable released updates. This means that one update candidate cannot be built against another update candidate without rel-eng interaction. We haven't been very vocal about this yet, and it isn't documented anywhere.
- Do we want to adjust things or leave them be?
- See IRC log for discussion points
Decision: Talk to lmacken about modifying Bodhi to do a sanity check on published buildroot contents before allowing an update be pushed. After that we will make the dist-fc7-build buildroot auto-update with -candidate builds and see what happens.
Expand Standard Operating Procedures (SOPs)
- f13
- rel-eng has more and more people helping out so we want to make sure that things are well documented
- Infrastructure team started doing SOP/ pages (Standard Operating Procedure) and they are proving very useful. I'd like to start doing them for Release Engineering too
- thinking of ReleaseEngineering/SOP/<task> as the layout, the SOP page itself could be a list of pages and info on how to add an SOP
- Proposal: As a release engineer, as you do tasks for Fedora, check to see if there is an SOP/<task> page. If there isn't, create one and poke rel-eng folks for review.
- This is more of a mandate than a vote item.
Early Torrent Release - Rahul Sundaram
- explore possibility of doing early torrent releases.
- See IRC log for discussion of pros and cons and affect on mirrors
Decision: Investigate with Infrastructure team ways of getting more mirrors to participate in seeding the torrent (early?). Do not release torrent to general public before the agreed upon coordinated release date/time.
Upgrade path enforcement - Rahul Sundaram
- Discussion surrounding policy that upgrade path should not break from a particular point on, for example, after Test1 or Test2--enforced by Release Engineering.
- See IRC log for discussion details
Decision: Recommend to FESCo policy that upgrade path never be compromised either by removing builds or NEVR regressions. File RFE in Koji to enforce rule at build time
IRC Transcript
![]() | <a href="#t12:57" class="time">12:57</a> | |
f13 | Meeting time coming up. | <a href="#t12:57" class="time">12:57</a> |
---|---|---|
f13 | Release Engineering meeting starting now. | <a href="#t12:59" class="time">12:59</a> |
f13 | jwb_gone: notting: jeremy: rdieter: warren: wwoods: ping | <a href="#t13:00" class="time">13:00</a> |
notting | yes? | <a href="#t13:00" class="time">13:00</a> |
rdieter | here | <a href="#t13:00" class="time">13:00</a> |
* nirik sits in the bleachers with his coffee. | <a href="#t13:00" class="time">13:00</a> | |
f13 | notting: just role call. | <a href="#t13:00" class="time">13:00</a> |
wwoods | ptchu | <a href="#t13:00" class="time">13:00</a> |
wwoods | curmudgeonly grammar teacher living in my frontal lobe requires me to say: it's roll call. | <a href="#t13:01" class="time">13:01</a> |
* jeremy is here | <a href="#t13:01" class="time">13:01</a> | |
wwoods | like honor roll. | <a href="#t13:01" class="time">13:01</a> |
notting | wwoods: cinnamon roll? | <a href="#t13:01" class="time">13:01</a> |
notting | pork roll? | <a href="#t13:01" class="time">13:01</a> |
jeremy | my clock is just slow :) | <a href="#t13:01" class="time">13:01</a> |
f13 | wwoods: but what if I'm asking to see who's here to fill what role? | <a href="#t13:01" class="time">13:01</a> |
f13 | role roll call? | <a href="#t13:02" class="time">13:02</a> |
* mmcgrath pongs | <a href="#t13:02" class="time">13:02</a> | |
wwoods | f13: totally | <a href="#t13:02" class="time">13:02</a> |
wwoods | and if you're playing back a filmstrip of this event, you can say: roll role roll call | <a href="#t13:02" class="time">13:02</a> |
f13 | ok, first topic is kind of fun. | <a href="#t13:02" class="time">13:02</a> |
warren | fun | <a href="#t13:02" class="time">13:02</a> |
f13 | As it currently stands, the buildroot used for fedora 7 updates building is not self updating. It only contains things from f7-gold, and stable released updates. | <a href="#t13:03" class="time">13:03</a> |
f13 | this means that one update candidate cannot be built against another update candidate without rel-eng interaction. | <a href="#t13:03" class="time">13:03</a> |
f13 | We haven't been very vocal about this yet, and it isn't documented anywhere. I have to ask if we want to adjust things or leave them be. | <a href="#t13:03" class="time">13:03</a> |
mmcgrath | f13: was this by design or an oversight? | <a href="#t13:04" class="time">13:04</a> |
f13 | mmcgrath: design | <a href="#t13:04" class="time">13:04</a> |
wwoods | I think we want to adjust that. People have been complaining about how huge a PITA it is to get several interdependent packages built | <a href="#t13:04" class="time">13:04</a> |
f13 | it's how things work internally since before I got here. | <a href="#t13:04" class="time">13:04</a> |
rdieter | might be worth trying, anyone against self-updated buildroots? | <a href="#t13:04" class="time">13:04</a> |
* mmcgrath just wonders what we might break if we change it. | <a href="#t13:04" class="time">13:04</a> | |
wwoods | oh. then how did updated stuff get put into the buildroot for interdependent builds? | <a href="#t13:04" class="time">13:04</a> |
notting | nothing would break until someone puts something broke in updates-candidate and someone builds against it | <a href="#t13:05" class="time">13:05</a> |
f13 | mmcgrath: we run the risk of broken buildroots, things built against things that don't go out as updates, etc... | <a href="#t13:05" class="time">13:05</a> |
warren | Here's one possible solution, similar to the fedora-cvs work queue. Create a fedora-rel-eng work queue, where people can ask for that kind of request in a controlled manner. We can use that until we have a more automated process in packagedb. | <a href="#t13:05" class="time">13:05</a> |
f13 | wwoods: we have a 'dist-fc7-override' tag that we can tag builds with to make them available. | <a href="#t13:05" class="time">13:05</a> |
nirik | how many requests has releng had to add something to the buildroots? | <a href="#t13:05" class="time">13:05</a> |
jeremy | warren: you more want to eventually get to where people can specify "use these unreleased updates" for my build... not something in the packagedb | <a href="#t13:05" class="time">13:05</a> |
f13 | I'm bringing pjones in, I know he's had some opinions on this in the past. | <a href="#t13:05" class="time">13:05</a> |
wwoods | f13: right, which requires rel-eng intervention (and extra action by maintainers --> OMGWTFREGRESSION) | <a href="#t13:06" class="time">13:06</a> |
warren | f13, oops, I shouldn't have added a package to dist-fc7-build I'm guessing. I'll have to undo that. | <a href="#t13:06" class="time">13:06</a> |
pjones | hey, so where are we? | <a href="#t13:06" class="time">13:06</a> |
wwoods | how did this work for Core / Extras? | <a href="#t13:06" class="time">13:06</a> |
rdieter | wwoods: Extras self-updated, Core, dunno (not?) | <a href="#t13:06" class="time">13:06</a> |
nirik | extras always got everything after builing. | <a href="#t13:06" class="time">13:06</a> |
notting | wwoods: extras always built against latest, even unreleased | <a href="#t13:06" class="time">13:06</a> |
notting | wwoods: core required intervention | <a href="#t13:06" class="time">13:06</a> |
f13 | pjones: basically I've stated the current situation (not updating) and gathering thoughts/opinions on either keeping it like it is, or changing it and risking the breakage. | <a href="#t13:06" class="time">13:06</a> |
f13 | mschwendnt (hwoever the hell you spell that) and dgilmore had some stats for how often the buildroot got fubar in Extras | <a href="#t13:07" class="time">13:07</a> |
nirik | I know it will be fun for me on the next Xfce update... there are lots of dependency chains there... 3 inital packages, then everythign needs them and each one in the row after that... | <a href="#t13:07" class="time">13:07</a> |
wwoods | It would seem like we need some kind of a hybrid where people can create private branched buildroots for building their stuff, or something? | <a href="#t13:07" class="time">13:07</a> |
pjones | f13: at the very least, if we leave the default like it is, we need the ability to make it pull from updates and such via BuildRequires: | <a href="#t13:07" class="time">13:07</a> |
f13 | pjones: well, the buildroot currently is populated with f7-gold and stable updates | <a href="#t13:07" class="time">13:07</a> |
wwoods | I admit not having deep knowledge of how much overhead that would require. All I know is that a bunch of maintainers are unhappy about it. Which is not news to anyone. | <a href="#t13:08" class="time">13:08</a> |
f13 | pjones: it just doesn't have any updates-testing or -candidate builds. | <a href="#t13:08" class="time">13:08</a> |
pjones | so if, for example, I know a new anaconda needs a newer kudzu which I've just built, then I can BR: it and have that work. | <a href="#t13:08" class="time">13:08</a> |
nirik | how about a bodhi tag/box? "add to buildroot for more builds"? | <a href="#t13:08" class="time">13:08</a> |
mdomsch | pjones, BR with version? | <a href="#t13:08" class="time">13:08</a> |
f13 | pjones: in the current situation, unless kudzu has been pushed out as stable, you'd have to request rel-eng interaction to tag the kudzu build. | <a href="#t13:08" class="time">13:08</a> |
pjones | mdomsch: yes | <a href="#t13:09" class="time">13:09</a> |
mdomsch | pjones, even then, you need to have the repo available for mock to use it | <a href="#t13:09" class="time">13:09</a> |
pjones | mdomsch: agreed | <a href="#t13:09" class="time">13:09</a> |
f13 | mdomsch: if the buildroot was self updating, chain-build would work | <a href="#t13:09" class="time">13:09</a> |
f13 | almost. | <a href="#t13:09" class="time">13:09</a> |
notting | f13: that's a big win for peopel | <a href="#t13:09" class="time">13:09</a> |
rdieter | I'm of the KISS mind, auto-update buildroot, rel-eng can step in and fix any breakage (ie, untag bustedness) if need be. | <a href="#t13:09" class="time">13:09</a> |
pjones | f13: chain-build has other problems though. not the least of which being the syntax is weird | <a href="#t13:09" class="time">13:09</a> |
f13 | I don't think chainbuild has the concept of 'this was already built, but wait unti it's available to build me' | <a href="#t13:09" class="time">13:09</a> |
f13 | pjones: yeah, syntax is interesting. NOt sure how to make it better. Open to suggestions | <a href="#t13:10" class="time">13:10</a> |
rdieter | f13: chainbuild worked for me that way, when I tried it. | <a href="#t13:10" class="time">13:10</a> |
pjones | rdieter: *nod*. We should be able to tell "oh, kudzu is fucked, we need to roll it out as well as ${dependent packages that got built}" | <a href="#t13:10" class="time">13:10</a> |
f13 | rdieter: oh cool, maybe it is smart enough (: | <a href="#t13:10" class="time">13:10</a> |
f13 | pjones: my worry is that $(dependent packages that got buitl) may have made it out to stable updates | <a href="#t13:10" class="time">13:10</a> |
f13 | and then it's not so easy/forgiving to just make it go away | <a href="#t13:11" class="time">13:11</a> |
pjones | rdieter: I think we have problems where an update isn't available to build with more often than we have problems where an update was bad. | <a href="#t13:11" class="time">13:11</a> |
pjones | f13: that seems like something we should be able to preclude from happening | <a href="#t13:11" class="time">13:11</a> |
rdieter | pjones: agreed. | <a href="#t13:11" class="time">13:11</a> |
f13 | pjones: with some work sure, bodhi could query to see if any component in the buildroot for an update is as of yet unreleased. | <a href="#t13:11" class="time">13:11</a> |
pjones | f13: nod. | <a href="#t13:11" class="time">13:11</a> |
warren | f13, good idea. | <a href="#t13:11" class="time">13:11</a> |
* f13 signs lmacken up for more work. | <a href="#t13:12" class="time">13:12</a> | |
rdieter | A compromise maybe would be to enable updates-testing in the buildroot? | <a href="#t13:12" class="time">13:12</a> |
f13 | rdieter: that's not good enough I don't thik. | <a href="#t13:12" class="time">13:12</a> |
f13 | think. | <a href="#t13:12" class="time">13:12</a> |
jeremy | f13: with that, I'd be okay with allowing updates-candidates in buildroots. but it's going to cause problems | <a href="#t13:12" class="time">13:12</a> |
f13 | rdieter: you may want to issue a big stack as one update. | <a href="#t13:12" class="time">13:12</a> |
jeremy | eg, jakub builds new glibc, it's in updates-testing and he wants it to be there a week | <a href="#t13:12" class="time">13:12</a> |
rdieter | f13: ok, nevermind. | <a href="#t13:12" class="time">13:12</a> |
jeremy | people that want to build and immediately push (or security updates, etc) will get flagged | <a href="#t13:12" class="time">13:12</a> |
f13 | jeremy: well, jakub builds into dist-fc7-candidate anyway, something completely not imported for his own personal tests, then tags it for updates-cnadidate by hand when he knows it's good | <a href="#t13:13" class="time">13:13</a> |
f13 | but yes, I can see that causing problems, so perhaps an override option. | <a href="#t13:13" class="time">13:13</a> |
f13 | unless that will break deps :/ | <a href="#t13:13" class="time">13:13</a> |
f13 | So I'm willing to experiment with Fedora 7 updates to see how it goes, once we get some bodhi love. | <a href="#t13:14" class="time">13:14</a> |
warren | f13, how could bodhi know if something that was in the buildroot is not yet pushed? Does mock output list *all* packages that were installed? | <a href="#t13:14" class="time">13:14</a> |
pjones | f13: perhaps allow package builders to specifically tag their updates-testing things for building? you'd probably still need the "this can't be released until that is" infrastructure, but it wouldn't have the problem jeremy mentioned | <a href="#t13:14" class="time">13:14</a> |
f13 | warren: koji tracks buildroot components for a build. | <a href="#t13:15" class="time">13:15</a> |
warren | f13, ah. | <a href="#t13:15" class="time">13:15</a> |
wwoods | I thought we were going to start allowing updates to have a set of packages? | <a href="#t13:16" class="time">13:16</a> |
nirik | yeah, I think it would make sense to leave default as it is now, and allow a checkbox for maintainers in bodhi to add their update-testing update to the buildroot? | <a href="#t13:16" class="time">13:16</a> |
f13 | pjones: that's a possiblity. Would have to think about how to construct that workflow | <a href="#t13:16" class="time">13:16</a> |
notting | nirik: that really doesn't help with things like chain-build | <a href="#t13:16" class="time">13:16</a> |
warren | nirik, oh, so a third unpublished layer? | <a href="#t13:16" class="time">13:16</a> |
f13 | ah right. | <a href="#t13:16" class="time">13:16</a> |
f13 | that doesn't help chain-build. | <a href="#t13:16" class="time">13:16</a> |
nirik | true. | <a href="#t13:16" class="time">13:16</a> |
f13 | so... | <a href="#t13:16" class="time">13:16</a> |
warren | bodhi could push to THREE layers instead of the current two. | <a href="#t13:16" class="time">13:16</a> |
warren | unpushed -> buildrootonly -> updates-testing -> updates | <a href="#t13:17" class="time">13:17</a> |
nirik | well, it already has pending/updates-testing/updates | <a href="#t13:17" class="time">13:17</a> |
f13 | Proposal: We talk to lmacken about modifying bodhi to do a sanity check on published buildroot contents before allowing an update be pushed. We then make the buildroot self updating and see what happens. | <a href="#t13:17" class="time">13:17</a> |
warren | ah | <a href="#t13:17" class="time">13:17</a> |
rdieter | f13: +1 | <a href="#t13:17" class="time">13:17</a> |
f13 | warren: you'd still introduce a break in the workflow where you can't rely upon chain-build. | <a href="#t13:17" class="time">13:17</a> |
warren | nirik, what does pending mean? | <a href="#t13:17" class="time">13:17</a> |
warren | f13, +1 | <a href="#t13:17" class="time">13:17</a> |
f13 | warren: it means it hasn't gone out in updates-testing yet | <a href="#t13:17" class="time">13:17</a> |
nirik | warren: someone added that they want it as an update, but hasn't pushed it to updates-testing yet | <a href="#t13:18" class="time">13:18</a> |
wwoods | f13: update pushed live, or pushed to -testing? | <a href="#t13:18" class="time">13:18</a> |
pjones | f13: as an aside, the problem with chain-build is that it doesn't really act on a checked out tree | <a href="#t13:19" class="time">13:19</a> |
pjones | f13: +1 on your proposal above, btw. | <a href="#t13:19" class="time">13:19</a> |
f13 | pjones: lets talk about chain-build over in #fedora-devel after this with mbonnet ok? | <a href="#t13:20" class="time">13:20</a> |
pjones | ok | <a href="#t13:20" class="time">13:20</a> |
f13 | wwoods: I don't understand the question? | <a href="#t13:20" class="time">13:20</a> |
abadger1999 | f13: +1 | <a href="#t13:20" class="time">13:20</a> |
f13 | wwoods: oh I see. Well, it shouldn't allow something to go into -testing unless what was used to produce it is on the way to -testing, ditto final. | <a href="#t13:21" class="time">13:21</a> |
wwoods | f13: I'm trying to clarify "pushed" in your proposal - pushed where? | <a href="#t13:21" class="time">13:21</a> |
f13 | although we could be more relaxed with -testing. | <a href="#t13:21" class="time">13:21</a> |
wwoods | ah, gotcha | <a href="#t13:21" class="time">13:21</a> |
wwoods | +1 either way, but I wanted to be clear | <a href="#t13:21" class="time">13:21</a> |
f13 | k. I'll work with lmacken to see which is more feasable. | <a href="#t13:22" class="time">13:22</a> |
f13 | This may take a while to implement so we'll still need documented workflow for people to ask rel-eng to tag things for them. | <a href="#t13:22" class="time">13:22</a> |
f13 | jeremy: notting: your votes? | <a href="#t13:23" class="time">13:23</a> |
notting | f13: open the floodgates! | <a href="#t13:23" class="time">13:23</a> |
jeremy | let's see what we can do... I'm open to trying things, though still a little concerned for the obvious reasons | <a href="#t13:23" class="time">13:23</a> |
f13 | jeremy: ditto (: | <a href="#t13:23" class="time">13:23</a> |
f13 | Ok, so that passed. | <a href="#t13:23" class="time">13:23</a> |
f13 | poelcat: Decision: We talk to lmacken about modifying bodhi to do a sanity check on published buildroot contents before allowing an update be pushed. We then make the dist-fc7-build buildroot auto-update with -candidate builds and see what happens. | <a href="#t13:24" class="time">13:24</a> |
![]() | <a href="#t13:24" class="time">13:24</a> | |
f13 | so rel-eng has more and more people doing things for us, we'll want to make sure that things are well documented. | <a href="#t13:25" class="time">13:25</a> |
f13 | The Infrastructure team started doing SOP/ pages (Standard Operating Proceedure) and they are proving very useful. I'd like to start doing them for Release Engineering too | <a href="#t13:25" class="time">13:25</a> |
* mclasen mumbles about silly acronyms | <a href="#t13:26" class="time">13:26</a> | |
f13 | I'm thinking of ReleaseEngineering/SOP/<task> as the layout, the SOP page itself could be a list of pages and info on how to add an SOP | <a href="#t13:26" class="time">13:26</a> |
f13 | mclasen: suggestions? | <a href="#t13:26" class="time">13:26</a> |
warren | f13, sounds good to me. | <a href="#t13:26" class="time">13:26</a> |
mclasen | since you spell out release engineering there, why not spell out procedure as well | <a href="#t13:27" class="time">13:27</a> |
f13 | *shrug* I'm lazy, and there is already precidence for SOP/ in the Infrastructure pages | <a href="#t13:27" class="time">13:27</a> |
* mclasen relurks | <a href="#t13:27" class="time">13:27</a> | |
f13 | I don't really think it matters if the page is linked to from ReleaseEngineering. | <a href="#t13:27" class="time">13:27</a> |
f13 | Proposal: As a release engineer, as you do tasks for Fedora, check to see if there is an SOP/<task> page. If there isn't, create one and poke rel-eng folks for review. | <a href="#t13:28" class="time">13:28</a> |
f13 | I guess this is more of a mandate than a vote item. | <a href="#t13:28" class="time">13:28</a> |
f13 | Ok, moving on | <a href="#t13:29" class="time">13:29</a> |
![]() | <a href="#t13:29" class="time">13:29</a> | |
f13 | mether wanted us to talk about doing early torrent releases. | <a href="#t13:29" class="time">13:29</a> |
mmcgrath | f13: I found out at linux tag that OpenSuSE actually has some of their mirrors setup as torrent seeds. We could also do that. | <a href="#t13:30" class="time">13:30</a> |
f13 | I'm personally not a huge fan of this, last time the mirror admins caught wind of something like this they got kinda pissy. | <a href="#t13:30" class="time">13:30</a> |
wwoods | like.. torrents open up before the mirrors finish synching? | <a href="#t13:30" class="time">13:30</a> |
jeremy | mmcgrath: some of our mirrors act as torrent seeds | <a href="#t13:30" class="time">13:30</a> |
jeremy | mmcgrath: eg, I know kernel.org does | <a href="#t13:30" class="time">13:30</a> |
* mmcgrath didn't know that. | <a href="#t13:30" class="time">13:30</a> | |
f13 | wwoods: yeah, I don't recall from the email how soon he wanted it open. | <a href="#t13:30" class="time">13:30</a> |
jeremy | f13: I'm not a fan, so -1 from me | <a href="#t13:30" class="time">13:30</a> |
mmcgrath | how do we get the seed to them? | <a href="#t13:30" class="time">13:30</a> |
f13 | mmcgrath: it's done after release though, not before release. | <a href="#t13:30" class="time">13:30</a> |
mmcgrath | ahh | <a href="#t13:30" class="time">13:30</a> |
f13 | mmcgrath: they probably just copy the mirrored isos into place then fire up bittorrent | <a href="#t13:31" class="time">13:31</a> |
wwoods | I think I kind of fail to see the point | <a href="#t13:31" class="time">13:31</a> |
jeremy | yeah, believe so | <a href="#t13:31" class="time">13:31</a> |
* f13 tries to find the posting | <a href="#t13:31" class="time">13:31</a> | |
warren | A mirror with lots of bandwidth doesn't need the data to be useful to a torrent cloud. | <a href="#t13:31" class="time">13:31</a> |
mmcgrath | wwoods: I think the point is to encourage torrent usage. | <a href="#t13:31" class="time">13:31</a> |
warren | They will rapidly obtain it and become a seed within 10-20 minutes. | <a href="#t13:31" class="time">13:31</a> |
mmcgrath | did anyone try the torrent right after release this time around? I wonder what the speeds were. | <a href="#t13:31" class="time">13:31</a> |
notting | f13: right, we don't actually publish the layout for torrent, though | <a href="#t13:31" class="time">13:31</a> |
f13 | Torrent only release | <a href="#t13:32" class="time">13:32</a> |
f13 | -------------------- | <a href="#t13:32" class="time">13:32</a> |
f13 | I remember this was discussed before but when a new release is made and | <a href="#t13:32" class="time">13:32</a> |
f13 | we are waiting for the mirrors to sync why not do a torrent only release | <a href="#t13:32" class="time">13:32</a> |
f13 | immediately? | <a href="#t13:32" class="time">13:32</a> |
rdieter | mmcgrath: I did, speed was good (not great). | <a href="#t13:32" class="time">13:32</a> |
f13 | The advantage is that we won't suffer as much from infrastructure and | <a href="#t13:32" class="time">13:32</a> |
f13 | mirror problems with the mad rush in everybody trying to get it at the | <a href="#t13:32" class="time">13:32</a> |
f13 | same time. Not sure if this has any disadvantages. | <a href="#t13:32" class="time">13:32</a> |
mmcgrath | rdieter: but you didn't think "gosh this is so terrible I'll just go to download.fedora.redhat.com and get it from there" ? :) | <a href="#t13:32" class="time">13:32</a> |
poelcat | mmcgrath: speed was not that great for at least 24 hours, maybe longer | <a href="#t13:32" class="time">13:32</a> |
warren | why would it be terrible? | <a href="#t13:32" class="time">13:32</a> |
warren | The more people that download a torrent, the faster it goes. A mirror server's bandwidth only accelerates it. | <a href="#t13:32" class="time">13:32</a> |
rdieter | mmcgrath: nope, I think it took ~2 hrs for me. | <a href="#t13:33" class="time">13:33</a> |
f13 | Well, bittorrent is pretty damn dumb in how it works | <a href="#t13:33" class="time">13:33</a> |
poelcat | warren: but not if everyone is starting at zero to begin with | <a href="#t13:33" class="time">13:33</a> |
mmcgrath | warren: not if no one has it and everyone is hitting torrent.fp.o to get it in the first many hours. | <a href="#t13:33" class="time">13:33</a> |
f13 | so if there are a lot of seeders in the other side of hte world from you, and a couple local, torrent will try all of them anyway and you'll be stuck getting bits slowly | <a href="#t13:33" class="time">13:33</a> |
notting | f13: so, disadvantages: 1) annoy mirrors 2) at most, you're getting a day or two extra release | <a href="#t13:33" class="time">13:33</a> |
f13 | notting: that was my thought too. | <a href="#t13:34" class="time">13:34</a> |
mmcgrath | really we should ask ourselves "Is increasting torrent speed over the first 24 hours something we need to fix" | <a href="#t13:34" class="time">13:34</a> |
f13 | plus confusion in the masses as to what the release date really is | <a href="#t13:34" class="time">13:34</a> |
f13 | mmcgrath: We should always consider making things faster. | <a href="#t13:34" class="time">13:34</a> |
mmcgrath | but at what cost! Oh the humanity! | <a href="#t13:34" class="time">13:34</a> |
f13 | I think we could publish torrent layouts early or ask for mirrors officailly to help torrent. | <a href="#t13:34" class="time">13:34</a> |
f13 | so I'm interested in looking at improving that. I don't thin kjust releasing the torrent to the general public is the way to achieve this though. | <a href="#t13:35" class="time">13:35</a> |
wwoods | I think getting some mirrors to seed the torrents might be the smarter thing to do | <a href="#t13:35" class="time">13:35</a> |
mmcgrath | <nod> | <a href="#t13:35" class="time">13:35</a> |
wwoods | I don't think breaking the release dates is a good idea | <a href="#t13:35" class="time">13:35</a> |
mmcgrath | we could limit early release to people with a fedoraproject.org account and ask them to start seeding early. | <a href="#t13:35" class="time">13:35</a> |
warren | mmcgrath, how do you limit that? | <a href="#t13:35" class="time">13:35</a> |
f13 | Proposal: Investigate with Infrastructure team ways of getting more mirrors to participate in seeding the torrent (early?). Do not release torrent to general public before the agreed upon coordinated release date/time. | <a href="#t13:36" class="time">13:36</a> |
warren | mmcgrath, torrent has no login | <a href="#t13:36" class="time">13:36</a> |
poelcat | how much impact would there be if 20-30 individ people had adv access to GA IOS and started their own seed at the minute release went live? not before | <a href="#t13:36" class="time">13:36</a> |
mmcgrath | warren: by restricting access to the .torrent file | <a href="#t13:36" class="time">13:36</a> |
warren | mmcgrath, so it could be leaked easily | <a href="#t13:36" class="time">13:36</a> |
mmcgrath | warren: your right, lets do nothing. | <a href="#t13:37" class="time">13:37</a> |
warren | that'll work though | <a href="#t13:37" class="time">13:37</a> |
notting | poelcat: such as.. mirrors? | <a href="#t13:37" class="time">13:37</a> |
warren | mmcgrath, I'm not suggesting that we do nothing. | <a href="#t13:37" class="time">13:37</a> |
nirik | with f7 it was leaked and appeared on some of the torrent sites... torrentspy, etc. | <a href="#t13:37" class="time">13:37</a> |
poelcat | notting: no idividual people like you and me... running from home | <a href="#t13:37" class="time">13:37</a> |
nirik | (before release) | <a href="#t13:37" class="time">13:37</a> |
rdieter | f13: +1 | <a href="#t13:37" class="time">13:37</a> |
warren | Leaks are OK IMHO, but we better make sure the ACTUAL release is leaked. | <a href="#t13:38" class="time">13:38</a> |
warren | (People don't receive a trojaned copy, or not quite final.) | <a href="#t13:38" class="time">13:38</a> |
notting | i think mirrors would do better | <a href="#t13:38" class="time">13:38</a> |
f13 | I do too | <a href="#t13:38" class="time">13:38</a> |
mmcgrath | f13: sounds good to me. | <a href="#t13:38" class="time">13:38</a> |
f13 | mirrors can make use of symlinks/hardlinks whatever to participate, mirrormanager could potentially help | <a href="#t13:38" class="time">13:38</a> |
warren | When I ran a mirror with DS-3 bandwidth (45mbit) and I joined the user swarm, I was downloading FC1 ISO's at like 7MB/sec. | <a href="#t13:39" class="time">13:39</a> |
warren | hmm... that doesn't add up. we might have had an OC-3 by then. | <a href="#t13:39" class="time">13:39</a> |
rdieter | afk 15 mins... | <a href="#t13:40" class="time">13:40</a> |
* f13 pushes for more votes. | <a href="#t13:40" class="time">13:40</a> | |
warren | f13, +1 | <a href="#t13:41" class="time">13:41</a> |
f13 | warren, notting, jeremy, wwoods ? | <a href="#t13:41" class="time">13:41</a> |
jeremy | sure! | <a href="#t13:41" class="time">13:41</a> |
wwoods | +1 | <a href="#t13:41" class="time">13:41</a> |
f13 | poelcat: Decision: Investigate with Infrastructure team ways of getting more mirrors to participate in seeding the torrent (early?). Do not release torrent to general public before the agreed upon coordinated release date/time. | <a href="#t13:42" class="time">13:42</a> |
![]() | <a href="#t13:42" class="time">13:42</a> | |
f13 | Another mether topic | <a href="#t13:42" class="time">13:42</a> |
f13 | Supported live upgrade from last test to general release | <a href="#t13:42" class="time">13:42</a> |
f13 | --------------------------------------------------------- | <a href="#t13:42" class="time">13:42</a> |
f13 | This was agreed upon for Fedora 7 test 4 in QA meetings but we didn't | <a href="#t13:42" class="time">13:42</a> |
f13 | actually announce it. It would encourage us to be more careful in | <a href="#t13:42" class="time">13:42</a> |
f13 | pushing out any updates after the last test release and it would | <a href="#t13:42" class="time">13:42</a> |
f13 | encourage more people to try it out the development branch and provide | <a href="#t13:42" class="time">13:42</a> |
f13 | us feedback. | <a href="#t13:42" class="time">13:42</a> |
f13 | I think all he's looking for here is that we set policy that upgrade path shouldn't break from that point on, and we enforce that. | <a href="#t13:43" class="time">13:43</a> |
warren | This means don't pull packages from rawhide in "oops" cases? | <a href="#t13:43" class="time">13:43</a> |
f13 | THere has been some discussion around making it so that upgrade path works from day one of a release cycle though, pre-test1 even. | <a href="#t13:43" class="time">13:43</a> |
f13 | warren: or revert with epoch | <a href="#t13:43" class="time">13:43</a> |
wwoods | at the very least, we're saying that official support for (Last Test Release) -> (Final Release) upgrades should be considered | <a href="#t13:44" class="time">13:44</a> |
warren | I'm all for disallowing pulling of packages from rawhide *always*. | <a href="#t13:44" class="time">13:44</a> |
f13 | I'm somewhat lazy and and would like to set policy only from say test2 or test3 on. | <a href="#t13:44" class="time">13:44</a> |
warren | If you need to undo a mistake, use epoch. People are really overly emotional about epoch. | <a href="#t13:44" class="time">13:44</a> |
f13 | but I can be convinced otherwise. | <a href="#t13:44" class="time">13:44</a> |
wwoods | in theory there would only be critical fixes in that time period anyway | <a href="#t13:44" class="time">13:44</a> |
f13 | wwoods: "this time" we'll actually be good about that! | <a href="#t13:44" class="time">13:44</a> |
warren | We lose testers whenever we require users to undo a mistake manually. | <a href="#t13:44" class="time">13:44</a> |
f13 | </straight_face> | <a href="#t13:44" class="time">13:44</a> |
wwoods | right, we're saying test3 (where test3 is final) - I'm sure mether would be happy to have it at test2 or earlier | <a href="#t13:44" class="time">13:44</a> |
notting | f13: an addict will never change until *they* decide to change | <a href="#t13:45" class="time">13:45</a> |
wwoods | I'll leave it to you all to determine whether that's feasible or not | <a href="#t13:45" class="time">13:45</a> |
wwoods | test2 would be even better because in theory that's the feature freeze | <a href="#t13:45" class="time">13:45</a> |
warren | I would be OK with a compromise of test2. | <a href="#t13:45" class="time">13:45</a> |
wwoods | so guaranteeing that users will be able to easily upgrade from test2 to final | <a href="#t13:45" class="time">13:45</a> |
wwoods | will mean getting lots more testers by test2. or so we'd hope. | <a href="#t13:45" class="time">13:45</a> |
f13 | wwoods: that's not what we're saying. | <a href="#t13:45" class="time">13:45</a> |
warren | f13, part of this is a rule, "After test2, rawhide yum update should not require manual intervention."? | <a href="#t13:46" class="time">13:46</a> |
f13 | wwoods: all I'm talking about right now is enforceing upgrade paths. | <a href="#t13:46" class="time">13:46</a> |
f13 | warren: I don't know if we can really ensure that either | <a href="#t13:46" class="time">13:46</a> |
notting | f13: how is that different? why wouldn't upgrades work? | <a href="#t13:46" class="time">13:46</a> |
* lmacken strolls in | <a href="#t13:46" class="time">13:46</a> | |
f13 | notting: %post fuckups | <a href="#t13:46" class="time">13:46</a> |
wwoods | yeah, why wouldn't that work? assuming that you've already installed test2 by a supported method.. | <a href="#t13:46" class="time">13:46</a> |
f13 | other such things. | <a href="#t13:46" class="time">13:46</a> |
f13 | beyond the "fixing" of here's a new package | <a href="#t13:46" class="time">13:46</a> |
f13 | I don't want to put us on the line for some things that may be out of our control. I'm happy to enforce upgrade paths, which goes a long long way to this, but I'm not going to say we can prevent manual intervention completely | <a href="#t13:47" class="time">13:47</a> |
warren | f13, i'm unclear on what exactly this proposal would mean. | <a href="#t13:47" class="time">13:47</a> |
f13 | warren: from say test2 on, any new package build has to have a clean upgrade path from the previous released package | <a href="#t13:48" class="time">13:48</a> |
warren | f13, will we stop the insanity of undoing a %{version} upgrade by removing it from rawhide and expecting users to manually downgrade? | <a href="#t13:48" class="time">13:48</a> |
f13 | no making builds dissappear, no downgrades without epoch | <a href="#t13:48" class="time">13:48</a> |
abadger1999 | Right. things like broken deps could also need manual intervention. | <a href="#t13:48" class="time">13:48</a> |
warren | f13, is that the entire scope of this proposal? | <a href="#t13:48" class="time">13:48</a> |
f13 | warren: are you looking for something else? | <a href="#t13:49" class="time">13:49</a> |
warren | no | <a href="#t13:49" class="time">13:49</a> |
warren | I'd be happy with that. | <a href="#t13:49" class="time">13:49</a> |
warren | +1 for after test2 | <a href="#t13:49" class="time">13:49</a> |
f13 | Can we come up with a reasonable cutoff point? forever? test1/2/3? | <a href="#t13:49" class="time">13:49</a> |
warren | I would prefer forever, but test2 is a good compromise. | <a href="#t13:49" class="time">13:49</a> |
warren | or even test1 | <a href="#t13:49" class="time">13:49</a> |
wwoods | test2 seems like a reasonable compromise for the first run for this | <a href="#t13:50" class="time">13:50</a> |
wwoods | test1 is a bit early - it's pre-feature freeze so we can't guarantee there won't be big changes between test1 and test2 | <a href="#t13:50" class="time">13:50</a> |
f13 | perhaps rel-eng can suggest test2 to FESCo, and they'll ratify/adjust as necessary. | <a href="#t13:50" class="time">13:50</a> |
f13 | wwoods: good point, like packages going away | <a href="#t13:50" class="time">13:50</a> |
warren | Who opposes "forever"? | <a href="#t13:50" class="time">13:50</a> |
wwoods | since the goal of this change is to make upgrades from test->final much easier, we can't necessarily hold those promises for anything pre-feature-freeze | <a href="#t13:51" class="time">13:51</a> |
f13 | warren: I do. | <a href="#t13:51" class="time">13:51</a> |
warren | f13, do you similarly oppose test1? | <a href="#t13:51" class="time">13:51</a> |
f13 | yeah, pretty much the same reasons. | <a href="#t13:51" class="time">13:51</a> |
f13 | up to test2 things are going to be somewhat fast and loose. | <a href="#t13:51" class="time">13:51</a> |
warren | but you're OK with the test2 compromise? | <a href="#t13:51" class="time">13:51</a> |
f13 | yeah, I'm fine with test2 on | <a href="#t13:52" class="time">13:52</a> |
warren | OK, let's do it. | <a href="#t13:52" class="time">13:52</a> |
* f13 is interested in notting and jeremy's opinion. | <a href="#t13:52" class="time">13:52</a> | |
warren | jeremy said "sure!" | <a href="#t13:53" class="time">13:53</a> |
notting | i'd sort of like earlier, but there are occasionally going to be things that have to 'go away' | <a href="#t13:53" class="time">13:53</a> |
jeremy | as a general thing, it's fine... but as notting says, there are always going to be exceptions | <a href="#t13:53" class="time">13:53</a> |
mclasen | of course, it doesn't mean that we shouldn't strive to have it at all times | <a href="#t13:54" class="time">13:54</a> |
notting | exactly | <a href="#t13:54" class="time">13:54</a> |
notting | we need to make things easier for users, not harder | <a href="#t13:55" class="time">13:55</a> |
jeremy | yep, I'm 110% agreed. just want to make sure we're realistic about the fact that we're not going to hit 100% | <a href="#t13:55" class="time">13:55</a> |
f13 | well, if that | <a href="#t13:55" class="time">13:55</a> |
* warren tries to understand the logic of 110% not reaching 100%. | <a href="#t13:56" class="time">13:56</a> | |
f13 | 's the desire, should we just set policy to have upgrade paths from day one, and only make exceptions when necessary? | <a href="#t13:56" class="time">13:56</a> |
jeremy | f13: yep | <a href="#t13:56" class="time">13:56</a> |
warren | f13, +1 | <a href="#t13:57" class="time">13:57</a> |
warren | f13, does this mean builds in F7 should fail if it isn't already superceded by a build in devel now? | <a href="#t13:57" class="time">13:57</a> |
jwb_gone | what do you do with packages that don't make it | <a href="#t13:58" class="time">13:58</a> |
f13 | jwb_gone: untag them! | <a href="#t13:58" class="time">13:58</a> |
f13 | oh wait... | <a href="#t13:58" class="time">13:58</a> |
warren | jwb_gone, concrete example? | <a href="#t13:58" class="time">13:58</a> |
jwb_gone | how does that fix the repo? | <a href="#t13:58" class="time">13:58</a> |
notting | warren: huh? | <a href="#t13:58" class="time">13:58</a> |
jwb_gone | foo-1.1.2.fc6 > foo-1.1.1.fc7 | <a href="#t13:58" class="time">13:58</a> |
jwb_gone | what do you do in that case? | <a href="#t13:58" class="time">13:58</a> |
f13 | we're talking about devel, not f7 | <a href="#t13:58" class="time">13:58</a> |
f13 | warren: ^^ | <a href="#t13:59" class="time">13:59</a> |
warren | jwb_gone, epoch | <a href="#t13:59" class="time">13:59</a> |
jwb_gone | was an example. s/fc6/fc7 s/fc7/fc8 | <a href="#t13:59" class="time">13:59</a> |
warren | People really need to get over their fear of epoch. It is better to use epoch than to break automated upgrades and to lose testing. | <a href="#t13:59" class="time">13:59</a> |
jwb_gone | mether's option was to pull that package from the repo in that case. i think that is wrong | <a href="#t13:59" class="time">13:59</a> |
warren | Yes, mether is wrong in that case. | <a href="#t13:59" class="time">13:59</a> |
f13 | who does the epoch and build though? Should we just file bugs and expect people to fix it, or will rel-eng have to go on a warpath at some point and fix all these things ourselves? | <a href="#t14:00" class="time">14:00</a> |
f13 | policy is one thing, but how does one enforce policy? | <a href="#t14:00" class="time">14:00</a> |
warren | f13, before brew, beehive would reject builds if N-V-R went backwards. | <a href="#t14:01" class="time">14:01</a> |
warren | We could do something like that. | <a href="#t14:01" class="time">14:01</a> |
wwoods | at some point we should have an rpm-diff tool running for proposed updates | <a href="#t14:02" class="time">14:02</a> |
wwoods | and that's something it should test for | <a href="#t14:02" class="time">14:02</a> |
warren | f13, I don't see anything wrong with enforcing N-V-R progression in the buildsystem itself. | <a href="#t14:02" class="time">14:02</a> |
wwoods | but yeah, putting that into the buildsys works | <a href="#t14:03" class="time">14:03</a> |
mclasen | warren: you mean E-V-R ? | <a href="#t14:03" class="time">14:03</a> |
f13 | ok, so we set policy now, and ask for koji enhancements to block a build of nevr goes backwards, for some value of backwards. | <a href="#t14:04" class="time">14:04</a> |
warren | "N-V-R ADVANCEMENT ERROR: foo-1.2.3-2.fc8 is older than foo-1.2.4-1.fc8, please consider releasing a new version upstream or use an Epoch." | <a href="#t14:04" class="time">14:04</a> |
warren | mclasen, yeah | <a href="#t14:04" class="time">14:04</a> |
f13 | Proposal: Recommend to FESCo policy that upgrade path never be compromised either by removing builds or NEVR regressions. File RFE in Koji to enforce rule at build time | <a href="#t14:05" class="time">14:05</a> |
warren | "E-V-R ADVANCEMENT ERROR: foo-1.2.5-1.fc7 is newer than foo-1.2.4-1.fc8, please upgrade the version in F8 before F7." | <a href="#t14:06" class="time">14:06</a> |
notting | sorry. gotta head out | <a href="#t14:06" class="time">14:06</a> |
mclasen | f13: does that affect our flexibility wrt to building updates in different tags, if they inherit from each other ? | <a href="#t14:07" class="time">14:07</a> |
f13 | mclasen: example? | <a href="#t14:07" class="time">14:07</a> |
warren | mclasen, you mean FC6, F7 and devel for example? | <a href="#t14:07" class="time">14:07</a> |
mclasen | yeah | <a href="#t14:07" class="time">14:07</a> |
f13 | warren: I'm not quite sure I want to do forward blocking yet. | <a href="#t14:07" class="time">14:07</a> |
warren | mclasen, version-release-disttag should be sufficient to prevent problems | <a href="#t14:08" class="time">14:08</a> |
f13 | warren: because it's more important to get a security update out for a releaesd product than it is for rawhide. | <a href="#t14:08" class="time">14:08</a> |
mclasen | I was thinking about a situation where I build something for f7 first, and then my next devel build has to be newer than that | <a href="#t14:08" class="time">14:08</a> |
mclasen | probably not a problem in practise | <a href="#t14:08" class="time">14:08</a> |
warren | f13, if both F7 and F8 are out as stable distros that might become a good idea. | <a href="#t14:08" class="time">14:08</a> |
f13 | warren: that's more enforcement at update push time, rather than build time | <a href="#t14:08" class="time">14:08</a> |
warren | mclasen, I'm thinking the buildsys only enforces forward E-V-R progression for a "stable" distro build if both distros are stable. | <a href="#t14:08" class="time">14:08</a> |
f13 | warren: lets not dictate what order a maintainer builds things in. | <a href="#t14:09" class="time">14:09</a> |
warren | f13, I don't see a problem with doing it between two stable releases, when F7 and F8 are both released. | <a href="#t14:09" class="time">14:09</a> |
f13 | warren: and really, upgrading from a releaed f7 to a released f8 is going to have issues anyway | <a href="#t14:09" class="time">14:09</a> |
warren | sure, but there shouldn't be any reason why a package in F8 should be < than F7 | <a href="#t14:09" class="time">14:09</a> |
f13 | becuase currently your upgrade to f8 via supported methods is anaconda, and you can't add update repos then. | <a href="#t14:10" class="time">14:10</a> |
f13 | warren: at release of hte udpate time sure. At build time, no. | <a href="#t14:10" class="time">14:10</a> |
warren | f13, how is this a problem really? | <a href="#t14:10" class="time">14:10</a> |
f13 | warren: what right do we have to dictate what order people do their builds in? | <a href="#t14:10" class="time">14:10</a> |
warren | f13, most cases where N+1 < N in extras were accidents. | <a href="#t14:10" class="time">14:10</a> |
warren | f13, OK, I guess we can consider this later. | <a href="#t14:11" class="time">14:11</a> |
f13 | so long as the update requests land at the right time I'm happy. I could give a shit what order they built the updates in. | <a href="#t14:11" class="time">14:11</a> |
jwb_gone | f13, there's nothing to prevent fc-6 builds from going higher | <a href="#t14:11" class="time">14:11</a> |
jwb_gone | it doesn't use koji | <a href="#t14:11" class="time">14:11</a> |
f13 | jwb_gone: not forever | <a href="#t14:11" class="time">14:11</a> |
f13 | so I didn't see any votes on the last proposal | <a href="#t14:12" class="time">14:12</a> |
f13 | <f13> Proposal: Recommend to FESCo policy that upgrade path never be compromised either by removing builds or NEVR regressions. File RFE in Koji to enforce rule at build time | <a href="#t14:12" class="time">14:12</a> |
* bpepple gives a +1 from the peanut gallery. | <a href="#t14:13" class="time">14:13</a> | |
warren | f13, what happened to the test2 part? | <a href="#t14:13" class="time">14:13</a> |
warren | f13, or at least the mention the "get an exception"? | <a href="#t14:13" class="time">14:13</a> |
warren | who grants exceptions? who makes that choice? | <a href="#t14:13" class="time">14:13</a> |
f13 | warren: I adjusted it to be forever. | <a href="#t14:13" class="time">14:13</a> |
wwoods | we were convinced that "forever" is workable and useful | <a href="#t14:14" class="time">14:14</a> |
f13 | warren: rel-eng grants exception, dealt with on case by case basis. | <a href="#t14:14" class="time">14:14</a> |
warren | f13, ok. | <a href="#t14:14" class="time">14:14</a> |
rdieter | f13: +1 proposal | <a href="#t14:14" class="time">14:14</a> |
warren | f13, +1 | <a href="#t14:14" class="time">14:14</a> |
wwoods | +1 | <a href="#t14:15" class="time">14:15</a> |
f13 | jeremy: ? | <a href="#t14:15" class="time">14:15</a> |
f13 | well, he was pretty happy with this before. | <a href="#t14:15" class="time">14:15</a> |
f13 | poelcat: Decision: Recommend to FESCo policy that upgrade path never be compromised either by removing builds or NEVR regressions. File RFE in Koji to enforce rule at build time | <a href="#t14:16" class="time">14:16</a> |
![]() | <a href="#t14:16" class="time">14:16</a> | |
f13 | that's the end of the agenda. We're over time. ANy pressing things? | <a href="#t14:16" class="time">14:16</a> |
wwoods | do we have an official (or draft) policy for updates-testing stuff? | <a href="#t14:17" class="time">14:17</a> |
warren | f13, until we're able to enable more rel-eng people to do pushing, could we have pushing done a little more often? | <a href="#t14:18" class="time">14:18</a> |
f13 | wwoods: all we decided on last week wwas to allow maintainers to skip updates-testing if they wanted to. | <a href="#t14:18" class="time">14:18</a> |
warren | f13, sometimes pushing has waited on 2-3 days for example. That has to be frustrating to our contributors. | <a href="#t14:18" class="time">14:18</a> |
f13 | warren: yeah, I'm trying to do the best we can here. Once a day isn't totally unreasonable. | <a href="#t14:18" class="time">14:18</a> |
f13 | um, since release of f7 I don't believe I've gone for more than 2 days without a push | <a href="#t14:19" class="time">14:19</a> |
warren | As long as we don't go beyond 1 day I think we're OK. | <a href="#t14:19" class="time">14:19</a> |
![]() | <a href="#t14:20" class="time">14:20</a> | |
f13 | thanks all. | <a href="#t14:20" class="time">14:20</a> |
Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!