13:04:41 <oschreib> #startmeeting ovirt 3.3 go/no-go
13:04:41 <ovirtbot> Meeting started Tue Aug 27 13:04:41 2013 UTC.  The chair is oschreib. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:04:41 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:05:00 <oschreib> #topic agenda
13:05:13 * masayag is here
13:05:15 <oschreib> #info Release tracker review
13:05:38 <oschreib> #info Communications
13:05:47 <oschreib> #info other topics
13:05:56 <oschreib> #info Go/No-Go decision
13:05:58 * mskrivanek here
13:05:59 * lhornyak is here
13:06:21 * oschreib is waiting for people to come
13:06:23 * tal here
13:06:25 * ecohen here
13:06:53 * lvernia here...
13:07:16 * doron here
13:07:37 <oschreib> ok, before reviewing the tracker
13:07:43 <oschreib> mburns: you had something to say?
13:08:25 <mburns> oschreib: ovirt-node builds are coming today with latest vdsm (built last week)
13:08:40 <mburns> but looking at above conversations, there are still patches being done for engine-3.3
13:08:51 <mburns> seems that we can't release if that's still happening
13:09:06 <oschreib> mburns: patches are fine, if they're not urgent we can still release
13:09:07 <oschreib> BUT
13:09:49 <oschreib> if the node will be ready only today (and TBH, we have rc2 build of engine which will be uploaded later today), so we might want to wait a bit.
13:10:09 <mburns> sounds like it
13:10:13 <mburns> if you have an rc2
13:10:24 <mburns> unless all the packages are low-risk
13:10:27 * apuimedo here as well (forgot to say earlier)
13:10:27 <mburns> err...
13:10:30 <mburns> all the changes...
13:11:05 <oschreib> engine isn't low risk
13:11:07 <oschreib> so is node.
13:11:17 <oschreib> no idea about other packages
13:11:35 <danken> vdsm is quite quiet I believe
13:12:26 <danken> but I think we should have a nicely-tagged minor version in ovirt-3.3. Not the current v4.12.0 +14 patches.
13:13:19 <mburns> oschreib: ok, then we've already said no-go, i think
13:13:28 <mburns> but please proceed with the status of other things
13:13:53 <oschreib> the other important topic is communication
13:14:01 <oschreib> I'm +1 on no-go
13:14:13 <oschreib> any other thoughts?
13:14:58 <danken> for those in the cold for two weeks, it's not yet clear what are the Engine patches that cause the delay.
13:15:12 <danken> I'd love to see a link/explanation in the meeting minutes.
13:15:15 <danken> oschreib: ^^
13:15:51 <apuimedo> could we have a wiki page (if it doesn't exist and I just ignore its existence)
13:16:05 <apuimedo> with the things that are red lighted
13:16:33 <oschreib> danken: apuimedo: none of the patches in engine (afaik) isn't red light, but bug fixes in several areas.
13:16:34 <apuimedo> split by package, of course
13:17:15 <apuimedo> so we're stalling for them to get merged and thats it?
13:17:19 <apuimedo> *that's
13:17:47 <oschreib> apuimedo: well, we have a new node, and I prefer it to be in the repo for ~week before moving to GA (same for engine, btw)
13:17:59 <mskrivanek> oschreib: so aren't they supposed to be on the tracker?
13:18:18 <apuimedo> oschreib: indeed, a new node with last week vdsm is a thing that should be tested at least a week
13:18:37 <apuimedo> make sure nothing is amiss
13:18:39 <oschreib> mskrivanek: As I noted couple of times (in the mailing list), we track critical bugs, that MUST get inside the release
13:19:02 <oschreib> mskrivanek: but bug fixes in the branch are still something we want
13:19:35 <apuimedo> danken: since there is a no-go, maybe the editBondings improvement could be backported to ovirt-3.3 branch
13:19:48 <apuimedo> (and moving some func tests, maybe)
13:19:58 <oschreib> mskrivanek: from my POV, every bug fix that gets into master should get into ovirt-engine-3.3 as well
13:20:07 <mskrivanek> oschreib: okay, so what are we trying to do..release or keep fixing indefinitely? IMHO if it's not a critical bug for 3.3 we should go ahead and release
13:20:15 <oschreib> apuimedo: easy on improvements
13:20:30 <oschreib> mskrivanek: you're right.
13:20:36 <apuimedo> oschreib: understood
13:21:14 <oschreib> and that what we would do, but since we have new node, that would delay the GA anyway, I don't care to build new engine with patches.
13:21:45 <oschreib> mburns: so, are we talking about one or two weeks of delay?
13:22:07 <mskrivanek> so if there is any bug you think it should be in 3.3 let's put it on the tracker. But currently there is only one open. Which still means we delay..but I wouldn't delay on any patches not on the tracker
13:22:37 <dneary> Is jbrooks here?
13:22:45 <dneary> He has been working on communication plan
13:22:54 <oschreib> mskrivanek: we can't put everything in the tracker, it's up to the maintaners to track trivial bug fixes.
13:23:03 <oschreib> dneary: we will talk about that in few minutes.
13:23:15 <mskrivanek> oschreib: then it doesn't belong to 3.3. We decided to branch several weeks back
13:23:49 <mburns> oschreib: i think 1 week
13:23:57 <oschreib> mskrivanek: why not? if we know of a bug, I see no reason for it not getting into 3.3
13:24:10 <mburns> assuming all patches deemed critical are ready for builds in the next 24 hours
13:24:22 <danken> oschreib: how a probably further delay is not a reason?!
13:24:35 <danken> we are late already. we should push only blockers.
13:24:42 <danken> otherwise we would never converge.
13:25:00 <danken> For example, http://gerrit.ovirt.org/#/c/18587/ is an ugly vdsm bug.
13:25:00 <mskrivanek> danken: yes!
13:25:06 <oschreib> danken: I disagree. bug fixes should get inside. not every bug fix is a reason for a re-spin.
13:25:17 <danken> but it's not new, so it should NOT be pushed for ovirt-3.3.0.
13:25:26 <apuimedo> minor bugs can wait for a later subrelease
13:25:42 <danken> oschreib: you should be the BAD cop!
13:25:43 <apuimedo> i.e., what's not on the tracker
13:25:51 <doron> apuimedo: so we have sub-releases?
13:26:09 <mskrivanek> oschreib: we decided to branch 3.3 and we already have RC...we should not take random new patches, whatever they are, if they are not critical. And if they are critical they belong to the tracker...
13:26:14 <danken> you must do everything to stop this constant slip of timing.
13:26:15 <doron> ie- since 3.2.1 (respin) we did not have a release
13:26:22 <apuimedo> doron: well, if there is something really outstanding found out during after release, we could have 3.3.1
13:26:38 <doron> apuimedo: that's the problem.
13:26:41 <danken> exactly, apuimedo. Like your http://gerrit.ovirt.org/#/c/18587/.
13:27:04 <apuimedo> danken: yes, that is a good example of something that should not block
13:27:06 <danken> fsimonce: do you agree with my request to tag vdsm-4.12.1?
13:27:21 <fsimonce> danken, yes, I just emailed you about it
13:27:25 <apuimedo> and should wait, if the conditions are good, for a 3.3.1 in case it is deemed necessary
13:27:29 <alonbl> can somoene point me to the developer that worked toward a milestone for upstream 3.3.0?
13:28:02 <danken> alonbl: ?
13:28:09 <alonbl> danken: yes?
13:28:17 <danken> alonbl: I did not get your question.
13:28:21 <apuimedo> me neither
13:28:52 <tal> Same here
13:28:54 <alonbl> danken: apuimedo: which developer on engine/tools worked toward a milestone at ovirt-engine-3.3 before freeze?
13:29:06 <mskrivanek> of course if there are other bugs which mainteners feel should get in let's put them on tracker and delay...but I'd agree with apuimedo and others here to finally release. Bear in mind we have ~500 bugs fixed since 3.2.1 ...
13:29:09 <alonbl> apuimedo: maybe in vdsm, I do not know.
13:29:16 <mburns> yes, we did a bad job tracking toward milestones in 3.3
13:29:26 <mburns> hopefully we'll do better in 3.4/4.0
13:29:42 <alonbl> it will be never better, unless we stop disrespect upstream
13:30:03 <alonbl> we should undertand by now (after the invalid 3.2 and 3.1 handing)
13:30:18 <alonbl> that the upstream release is a random point in time, completely broken product
13:30:33 <alonbl> as nobody actually work toward the milestone, nor we provide support
13:30:44 <danken> alonbl: migration network, multiple gateways, Neutron integration, were all ovirt-3.3 milestones.
13:30:52 <danken> and we worked hard to get them in on time.
13:30:57 <apuimedo> indeed
13:31:01 <alonbl> the example is 3.2.3... any reason why all z-stream bugs are not released upstream?
13:31:09 <alonbl> danken: no
13:31:20 <alonbl> danken: you worked hard to commit then for some random point in time
13:31:28 <alonbl> danken: but nobody actually fixed anything upstream
13:31:55 <danken> alonbl: huh? are you aware of a devestating bug in the network features?
13:31:58 <alonbl> danken: the feature freeze was code freeze for some reason, which lasted 24 hours
13:32:27 <apuimedo> alonbl: the delays we got in multiple gateways were upstream only, and we spent a lot of time getting it fixed
13:32:29 <alonbl> danken: I am seeing the master branch post split
13:32:44 <apuimedo> for example
13:32:57 <mburns> alonbl: danken:  yes, we had some broken processes, and we'll have to deal with that for the next release
13:32:59 <mskrivanek> alonbl: and are you aware of the fixes resulting from the oVirt Test Day...there were quite a few
13:33:14 <apuimedo> more than quite a few
13:33:16 <alonbl> mskrivanek: so... we have one day...
13:33:17 <mburns> but there is no reason to deal with that in this meeting
13:33:20 <alonbl> mskrivanek: great!
13:33:30 <alonbl> mskrivanek: one day of fixing random point in time.
13:33:33 <apuimedo> alonbl: that day spanned more than a week for some
13:33:43 <mburns> we need to deal with the go/no-go and status of blocking issues
13:34:02 <alonbl> mburns: no.... we need to understand that we care enough about upstream like we care about downstream
13:34:02 <mskrivanek> alonbl: no, we have (or better say had) a version which was +- usable, we fixed what was broken and since then we should have not accept random engine or whatver patches. only critical onces. And that's what the tracker is supposed to be for
13:34:17 <mburns> alonbl: yes, but not a topic for this meeting
13:34:22 <alonbl> mburns: I am glad.
13:34:35 <alonbl> mburns: so if we cannot release downstream today, we cannot release upstream as well
13:35:03 <mburns> alonbl: can you define what you mean about upstream/downstream?
13:35:09 <mburns> you mean master vs ovirt-3.3?
13:35:18 <mskrivanek> alonbl: that's not related
13:35:19 <oschreib> alonbl: bottom line, what is your suggestion?
13:35:21 <alonbl> mburns: the difference between upstream and downstream are ~10 oatches
13:35:40 <alonbl> mburns: if QA is not passed on downstream, how can it pass on upstream?
13:35:42 <mburns> alonbl: or do you mean downstream, as in rhev?
13:35:46 <alonbl> mburns: yes
13:36:38 <danken> fsimonce: ok, I'll tag current ovirt-3.3 as vdsm-4.12.1.
13:36:45 <mburns> rhev should have no impact on this discussion
13:36:52 <fsimonce> danken, ok
13:36:57 <mburns> i understand the point that there are bugs (there always are)
13:37:21 <alonbl> mburns: has nothing to do with bugs... it is the concept...
13:37:49 <alonbl> mburns: can you please show me a decent process in which developer releases something to public and then continue to perfect it to release something to its own customers?
13:38:20 <mburns> alonbl: aside from something like RHEL?
13:38:32 <alonbl> mburns: rhel is taking upstream (RELEASED)
13:38:55 <mburns> alonbl: and then patching and making better
13:39:01 <alonbl> mburns: means that apache, util-linux, etc... are all reached to conclusion that the are stable
13:39:18 <mburns> those patches go back upstream, but not necessarily on the version that gets released
13:39:20 <alonbl> mburns: different senario.... as packages are not pre-release at upstream
13:39:36 <alonbl> mburns: again... apache-2.2.0 was feature closed
13:39:45 <alonbl> mburns: develoeprs of apache worked toward the apache-2.2 goal
13:40:12 <alonbl> mburns: our developer works toward rhevm-3.3.0 goal not ovirt-engine-3.3.0 goal
13:40:16 <mburns> alonbl: and we were supposed to work towards a 3.3 goal
13:40:28 <mburns> alonbl: right, and that has to be fixed going forward
13:40:36 <mburns> (and not everyone worked toward that goal)
13:40:40 <alonbl> mburns: we have the chance to fix it now
13:41:09 <alonbl> mburns: we worked hard to make the 10 patches diff between upstream/downstream at the major component : ovirt-engine
13:41:27 <mburns> alonbl: then what are you proposing we do now?
13:42:10 <alonbl> mburns: I supposing to rebase on master and wait until all is stable and realase product which is feature close and stable.
13:43:00 <mburns> we are feature closed, currently (iinm)
13:43:06 <mburns> we're working on stability now
13:43:28 <mburns> bugs that are blockers should get attached to the tracker and only those bugs should get in
13:43:44 <mburns> it's my understanding that's the process we're following now...
13:46:09 <mburns> oschreib: have we gone through blockers yet?
13:46:23 <mskrivanek> alonbl: so you're suggesting to delay by several months? I don't really follow....we declared ovirt features complete what, a month ago? And stabilizing. downstream has little to do with it...and indeed should be based on ovirt 3.3 - but that's RHEV issue, not ovirts and out of todays discussion
13:46:24 <danken> alonbl: oh dear. I feel that vdsm's master branch is much less stable than its ovirt-3.3 branch. please don't take master to ovirt-3.3
13:46:34 <oschreib> mburns: no, we haven't
13:46:36 <danken> mburns: oschreib: there should be a longer bugfix-only phase before release, not a shorter one.
13:47:06 <mburns> danken: agreed
13:47:21 <oschreib> danken: I never claimed otherwise.
13:47:25 <mburns> danken: but we can fix that in the next release
13:47:44 <mburns> for now, it's just figuring out what blockers we have
13:47:56 <mburns> and when we can conceivably get them fixed and merged
13:48:02 <danken> oschreib: of course you have - few minutes ago you said that minor bug fixes should go in. they should not. we must stop the 3.3.0 train.
13:48:13 <oschreib> alonbl: mburns: do we have any bottom line or suggestion from the above discussion?
13:48:14 <apuimedo> danken: very much agreed on the longer bugfix only phase
13:48:30 <apuimedo> and two test days one week apart
13:48:30 <mburns> oschreib: from me, blockers only in 3.3 builds
13:48:33 <oschreib> something we can perhaps vote, or just a suggestion to do something better next time.
13:48:52 <mburns> oschreib: my proposal:
13:48:57 <danken> but please. let us release 3.3.0 according to the defined process.
13:49:14 <mburns> blockers only, build asap, release approx 1 week later (assuming no new blockers)
13:49:25 <danken> for 3.4.0 we should have a longer stabling phase, with two test days, maybe.
13:50:01 <danken> I must vote against rebasing now on top of master. that's a too huge slip imo.
13:50:10 <mburns> oschreib: for other suggestions (test days, timelines, etc), those should be 3.4 suggestions, not 3.3
13:50:21 <danken> UNLESS current 3.3.0 is inedible.
13:50:23 <apuimedo> mburns: yes, of course they are for 3.4
13:50:53 <oschreib> danken: I prefer bug fixes will get in, instead of shipping something with ~100 known bugs.(minor ones, even)
13:50:59 * mburns very much against rebasing at this stage
13:51:06 * jbrooks is here
13:51:26 <mburns> oschreib: i'm not against putting those 100 bug fixes into 3.3.1
13:51:38 <apuimedo> mburns: I agree with you
13:52:00 <mburns> but where we are right now, if they're not critical, then the risk of regression isn't worth the value of the bugfix
13:52:15 <oschreib> ok.
13:52:23 <danken> oschreib: that's a non-converging process. you must have a stop sign somewhere, and you must slow down before it.
13:52:46 <mburns> oschreib: what's the tracker bz #?
13:52:59 <oschreib> danken: well, if we have so many bugs, maybe alon's suggestion isn't that bad for engine.
13:53:10 <oschreib> mburns: https://bugzilla.redhat.com/show_bug.cgi?id=918494
13:53:59 <oschreib> OK, lets review the tracker (non ON_QA bugs)
13:54:19 <mburns> 987939 is the only one that's not at least modified
13:54:20 <oschreib> and than decide on delay/etc
13:54:30 * mburns assumes modified should be in the build coming today, right?
13:54:44 <oschreib> mburns: I'm not sure, that's what I want to check.
13:54:53 <mburns> ok
13:55:01 <oschreib> #topic blokcer review
13:55:20 <oschreib> #info Bug 988299 - Impossible to start VM from Gluster Storage Domain (gluster, MODIFIED)
13:55:43 <oschreib> sahina: any updates on the bug above for ovirt-engine-3.3?
13:56:16 <sahina> oschreib, the vdsm patch for this has been merged to 3.3
13:56:45 <mburns> sahina: in the latest vdsm build?
13:57:05 <mburns> (4.12.0-2)
13:57:18 <mburns> danken: did you say you're rebuilding with 4.12.1?
13:57:26 <danken> mburns: that patch forces us to ship gluster 3.4.0 within ovirt repos
13:57:43 <oschreib> danken: what's the problem about that?
13:57:43 <danken> fsimonce or dougsland would build it.
13:57:45 <mburns> danken: right, i worked that out with fsimonce last week
13:57:58 <mburns> we're shipping current 3.4.0 from gluster in f19 and el6
13:58:07 <mburns> and we updated ovirt-release to include the 3.4.0 repos
13:58:19 <danken> oschreib: no problems, just wanted to make sure mburns is aware of it.
13:58:25 <oschreib> sahina: so it sounds we can move this bug from modified, right?
13:58:33 <mburns> should be on_qa now, i think
13:58:42 <mburns> fsimonce: built it last week, iiuc
13:58:56 <fsimonce> mburns, yes
13:59:06 <sahina> oschreib, yep, will move to ON_QA
13:59:50 <oschreib> #info bug should be ON_QA
13:59:56 <oschreib> so we need a new vdsm, right?
14:00:14 <mburns> no, that was included in the build last week
14:00:38 <mburns> there is consideration for a new vdsm build, but not for this bug
14:00:56 <oschreib> ok
14:00:57 <oschreib> next
14:01:24 <oschreib> #info Bug 997362 - [engine-config] /var/log/ovirt-engine/engine-config.log doesn't exist (engine, MODIFIED)
14:01:29 <danken> fsimonce: v4.12.1 tag pushed. Would you or dougsland build it?
14:01:34 <oschreib> anyone representing infra?
14:01:51 <fsimonce> danken, for what? there's nothing new... just for the name?
14:01:58 <oschreib> I see http://gerrit.ovirt.org/#/c/18075/ is merged into ovirt-engine-3.3
14:02:06 <oschreib> so it's probably fixed
14:02:13 <oschreib> mperina: ping
14:03:10 <oschreib> no one here
14:03:24 <mskrivanek> it was merged 14 days ago, i think it's not an issue
14:03:39 <oschreib> mskrivanek: so someone should handle the bug.
14:03:43 <oschreib> next
14:03:54 <mskrivanek> bug says ON_QA
14:04:15 <oschreib> #info Bug 987939 - engine-setup -> engine-cleanup -> engine-setup -> fails (engine, NEW)
14:04:42 <oschreib> mskrivanek|away: no it's not....
14:04:56 <oschreib> about the bug above, which is in my domain
14:05:04 <oschreib> I suggest removing it from the tracker.
14:05:12 <oschreib> it won't be fixed in the next days
14:05:32 <oschreib> it's indeed important, but not critical, as it fails after a partial cleanup
14:05:37 <oschreib> objections?
14:05:40 <danken> fsimonce: yes, I think that the name is important enough. don't you?
14:05:50 <mburns> the 997362 is a rhev bug, fwiw, there is no ovirt bug, that i can see
14:06:01 <fsimonce> danken, just making sure I'm not missing something
14:06:02 <mskrivanek> oschreib: just for the sake of correctness - review http://gerrit.ovirt.org/#/c/18075/ is merged to ovirt-3.3 and the ovirt bug is 996962 and that is ON_QA
14:06:07 <danken> fsimonce: hopefully we can have a clear vdsm-4.12.1.tag.gz released, too.
14:06:16 <mburns> oschreib: as for 987939, i'd argue against it being a blocker anyway, so i'd agree to remove it from the tracker
14:07:11 <oschreib> mskrivanek: we we're talking about 987939 (which is the one blocks ovirt, although it's an rhev bug)
14:07:13 <oschreib> ok, removed
14:07:21 <mskrivanek> mburns: oschreib: btw seems to be the only open bug on tracker...but let's wait till we go through all
14:07:22 <oschreib> #info bug removed from blocker list.
14:07:39 <mburns> ok, so that's it, right?
14:07:43 <oschreib> #info review done
14:07:57 <mburns> only question is a rhev bug which is merged in ovirt already
14:08:06 <mburns> and shouldn't block
14:08:18 <mburns> so i think we're good once we have new engine and node builds
14:08:35 <mburns> (and vdsm if they decide to increase the version...)
14:09:01 <mskrivanek> mburns: it is merged
14:09:06 <mskrivanek> mburns: it's a wrong bug on tracker
14:09:08 <mburns> so let's get builds done in the next 24 hours and then we'll release next week
14:09:12 <mburns> mskrivanek: yep
14:09:18 <oschreib> mburns: +1
14:09:23 <mskrivanek> instead of 987939 there should be 996962
14:09:38 <oschreib> mburns: last topic is communication
14:09:39 <mskrivanek> mburns: +1
14:10:11 <mburns> ok, let's go to communication
14:10:18 <oschreib> mburns: one minutes
14:10:22 <oschreib> #chair mburns
14:10:22 <ovirtbot> Current chairs: mburns oschreib
14:10:38 <oschreib> #info agreed: 3.3 Release is no-go for this week
14:11:03 <oschreib> #info one week delay, with new node and engine builds within 24 hours
14:11:08 <oschreib> next topic
14:11:27 <oschreib> #topic release communication
14:11:49 <oschreib> so we need announce, press, etc
14:11:59 <oschreib> dneary: mburns: who's handling that?
14:12:42 <mburns> jbrooks: you've done a lot of that work already, right?
14:13:30 <dneary> oschreib, That will come down to myself and jbrooks
14:13:36 <jbrooks> mburns, Yes, I've been working mostly on the release notes, I will help w/ the announcement, I have a list of ppl to email about the release, too, press
14:13:45 <dneary> oschreib, We need a week's lead time to get our ducks in a row
14:13:53 <dneary> What's the current release date?
14:13:59 <mburns> dneary: target is next wednesday
14:14:12 <jbrooks> That's plenty of time for what I'm doing
14:14:32 <oschreib> 2013-09-04
14:14:52 <mburns> jbrooks: ok, let me know if you need anything else
14:14:54 <oschreib> mburns: please note that it's a half day off here in Israel btw.
14:15:11 <mburns> oschreib: on 04-Sep?
14:15:40 <oschreib> mburns: indeed
14:15:50 <dneary> jbrooks, Let's have a talk about sites to target, do some seeding of the pool for some tech journalists between now & then
14:16:00 <mburns> oschreib: ok, so let's have go/no-go on tuesday, and i'll push content around on wednesday, as needed
14:16:01 <oschreib> mburns: we might want to make sure everything is ok until the 03
14:16:05 <jbrooks> dneary, sounds good
14:16:05 <dneary> I bet we could put together a nice talking points/briefing for the release
14:16:10 <oschreib> mburns: cool
14:16:29 <oschreib> dneary: for sure.
14:17:05 <dneary> Right - let's take that to the users list, then
14:17:21 <oschreib> #info communication should be ready until 03-Sep
14:17:38 <oschreib> anything else?
14:17:58 <mburns> nothing from me
14:18:14 <oschreib> endmeeting in 3
14:18:29 <oschreib> 2
14:18:39 <oschreib> 1
14:18:46 <oschreib> #endmeeting