14:20:37 <dcaro> #startmeeting
14:20:37 <ovirtbot> Meeting started Mon Oct 27 14:20:37 2014 UTC.  The chair is dcaro. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:20:37 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:20:49 <dcaro> #chair eedri bkorren bkp sbonazzo
14:20:49 <ovirtbot> Current chairs: bkorren bkp dcaro eedri sbonazzo
14:21:23 <dcaro> #topic Hosting
14:21:42 <dcaro> #info Moving away from rackspace
14:22:07 <dcaro> I'm glad to announce that we are ready to decommission the rackspace servers
14:22:11 <eedri> \0/
14:22:12 <eedri> whooho
14:22:19 <bkorren> \0/
14:22:26 <eedri> +1 .. on a great effort finally finished
14:22:31 <eedri> dcaro, ^
14:22:39 <sbonazzo> great :-)
14:22:41 <dcaro> they have been out already almost a week, we moved all the servers from there to phoenix lab
14:22:43 <eedri> dcaro, that's for the effort and leading this project
14:22:50 <dcaro> welcome :)
14:22:54 <eedri> thanks also for bkorren in asissting
14:23:26 <YamakasY> ofrenkel: indeed!
14:23:42 <YamakasY> ofrenkel: BUT my centos 6 box is a 3.4 host
14:23:45 <dcaro> hopefully we can start stabilizing the envs to get reliable builds
14:23:50 <YamakasY> eedri: brought cookies ?
14:24:05 <eedri> YamakasY, brownies
14:24:13 <YamakasY> eedri: ++++++++++!
14:26:15 <bkorren> eedri, neah, credit is all for dcaro I just got in his way ^^;;
14:26:52 <dcaro> bkorren: don't underestimate your getting in the way ;)
14:28:06 <outi> this sucks :(
14:28:16 <outi> stumbling from one bz to another
14:28:23 <bkorren> dcaro, oh, btw, I've seen some stuff fail because of missing postgres, is the intention to have it installed everywhere or was it a badly labeled slave?
14:28:41 <dcaro> bkorren: I'm writing right now a patch to get it everywhere
14:30:00 <sbonazzo> dcaro are we moving to topic "jenkins slaves issues"? :-)
14:30:05 <bkorren> dcaro, via puppet? you might like to try puppetlabs` postgresql module - its reasonably good, and can save you time...
14:30:07 <dcaro> #topic jenkins
14:30:39 <dcaro> bkorren: I will :) thanks
14:31:01 <dcaro> #info Add postgresql confs tot he slaves, look into puppetlabs postgresql module
14:31:21 <dcaro> sbonazzo: feel free to expose issues :)
14:32:05 <sbonazzo> dcaro: we have vdsm-jsonrpc-java failing the build for F19 on F21 slaves, I guess it's something different than the rpm bug
14:32:20 <sbonazzo> dcaro looks something more network / jenkins related
14:33:11 <sbonazzo> dcaro http://jenkins.ovirt.org/job/vdsm-jsonrpc-java_master_create-rpms-fc19-x86_64_merged/71/console
14:33:15 <dcaro> sbonazzo: I just reconnected one of the slaves, that seemed to be online but failing to ping (not sure why jenkins still got it online)
14:33:23 <sbonazzo> dcaro oh!
14:33:34 <sbonazzo> dcaro ok, now we have the rpm bug on f21 :-)
14:33:41 <dcaro> sbonazzo: :/
14:34:16 <dcaro> sbonazzo: yep, I'll add the vdsm-jsonrpc labef filter too, I suppose that it will happen on all the fc19 jobs that run on fc21, I'll review all
14:34:20 <sbonazzo> dcaro but for vdsm-jsorpc-java on master we can drop f19 build
14:34:27 <sbonazzo> dcaro a 3.5 branch has been created
14:34:30 <dcaro> sbonazzo: that's better :)
14:34:35 <sbonazzo> dcaro we need new jobs for that branch
14:34:46 <dcaro> sbonazzo: so those are 2 different issues
14:34:52 <sbonazzo> dcaro yep
14:35:19 <dcaro> #action dcaro avoid running fc19 mock builds on fc21 hosts
14:35:41 <dcaro> #action dcaro add new 3.5 branch jobs for vdsm-jsonrpc-java project
14:35:49 <sbonazzo> dcaro +1
14:36:05 <dcaro> sbonazzo: do you know if there's already a ticket for the new jobs?
14:36:36 <sbonazzo> dcaro I'm not sure, pkliczew, did you open a ticket?
14:36:58 <pkliczew> sbonazzo, I forgot to open one, sorry
14:37:06 <sbonazzo> dcaro so, no, no ticket
14:37:21 <dcaro> pkliczew: can you open it please?
14:37:31 <bkp> No brownies for pkliczew :)
14:37:43 <pkliczew> dcaro, sure
14:37:55 <dcaro> pkliczew: thanks (:
14:39:27 <sbonazzo> dcaro lat issue on my side
14:39:42 <sbonazzo> dcaro http://jenkins.ovirt.org/job/ovirt-appliance_engine-3.5_master_merged/ is taking 3 days to finish
14:39:49 <sbonazzo> dcaro I guess it's stuck
14:40:05 <sbonazzo> 16:43:55 What do you want to do now? 16:43:55 1) Report Bug 16:43:55 2) Debug 16:43:55 3) Quit 16:43:55
14:40:06 <dcaro> sbonazzo: it's waiting user input :/
14:40:11 <dcaro> 16:43:55 What do you want to do now?
14:40:13 <dcaro> 16:43:55 1) Report Bug
14:40:15 <dcaro> 16:43:55 2) Debug
14:40:17 <dcaro> 16:43:55 3) Quit
14:40:48 <sbonazzo> dcaro yep, we need something to stop the job after some time of inactivity
14:41:12 <sbonazzo> dcaro (and maybe fix the job)
14:41:24 <dcaro> I think anaconda it's not handling correctly some exception, and generating one that hides the original one
14:41:41 <sbonazzo> dcaro so anaconda bug?
14:41:58 <dcaro> #action add a timeout to the job ovirt-appliance_engine-3.5_master_merged
14:42:07 <dcaro> fabiand: ^
14:42:12 <dcaro> fromani: have you seen that error?
14:42:25 <fabiand> dcaro, yes
14:42:33 <fabiand> I actually worked on that already, it depends on a fix for anaconda
14:42:36 <fabiand> let me dig out the fix
14:43:00 <fromani> dcaro: I don't
14:43:08 <fabiand> <fabiand> https://bugzilla.redhat.com/show_bug.cgi?id=1155979
14:43:08 <fabiand> <fabiand> https://bugzilla.redhat.com/show_bug.cgi?id=1155981
14:43:09 <fabiand> dcaro, ^^
14:43:19 * eedri here again
14:43:21 <dcaro> fromani: oops xd, that was for fabian
14:43:25 <fabiand> dcaro, these fixes are also for some other jobs which hang. .
14:43:26 <dcaro> strange that he answered...
14:43:46 <eedri> fabiand, dcaro do we have node nightly builds? someone asked me today
14:43:54 <eedri> sorry if offtopic
14:43:57 <fabiand> dcaro, and IO think it's a bug in jenkins that it sometimes does not kill stuck jobs ..
14:44:19 <fabiand> eedri, no not currently afaik
14:45:12 <fromani> dcaro: :)
14:45:59 <dcaro> fabiand: The timeout plugin was updated the last time we upgraded jenkins, and it seems it's not totally backwards compatible, as I just added an action to take (abort build).
14:46:18 <dcaro> fabiand: when was the first time you experienced those timeout errors?
14:47:02 <dcaro> eedri: http://resources.ovirt.org/pub/ovirt-master-snapshot/iso/ovirt-node-base-iso/ <- ?
14:47:43 <fabiand> dcaro, it's actually a bug in anaconda, I experienced it ~1week ago ..
14:47:58 <fabiand> dcaro, the timeout plugin never worked for those jobs ..
14:48:03 <fabiand> not that I can recall ..
14:48:17 <dcaro> fabiand: mmm, ok, I'll investigate
14:48:22 <fabiand> thanks
14:48:24 * fabiand too :)
14:48:29 <fabiand> but how to fix the stalling ;)
14:48:36 <dcaro> #action Investigate timeout plugin not working on node jobs (and possibly others)
14:48:53 <dcaro> fabiand: what is required for anaconda to work properly?
14:48:58 <fabiand> dcaro, timeout is actually configured for all the jobs that currently stall ..
14:49:07 <fabiand> dcaro, an anaconda fix, which landed
14:49:19 <fabiand> it will soon be ion one of their builds which the job will pick up
14:51:07 <dcaro> fabiand: ok, so can we just wait? Or must we work around it untill it's built?
14:51:49 <fabiand> I'll wait
14:51:58 <fabiand> I'll consider to disable the jobs if it will take longer
14:52:20 <dcaro> fabiand: okok
14:52:20 <fabiand> but the fix is merged, so I expect it to land soon (that is what the anaconda team told me)
14:52:32 <fabiand> but let me know if you want me to do sth with those jobs ..
14:54:39 <fabiand> rbarry, tlitovsk fabiand -- friendly reminder to update the team status
14:54:41 <fabiand> on the pad
14:54:43 <fabiand> http://cloudinf.etherpad.corp.redhat.com/RHEVHMinutes?
14:54:54 <dcaro> fabiand: I'll just readjusted the timeout settings there, let me run a couple builds to make sure it works as expected
14:55:07 <fabiand> dcaro, cool!
14:55:09 <dcaro> okok, any other jenkins issues around?
14:55:15 <fabiand> I'll be happy to see ifthat it works!
14:55:19 <fabiand> dcaro, otherwise I am good
14:55:30 <fabiand> Now that I've learned that I got a username+password ..
14:55:31 <fabiand> :)
14:56:35 <fabiand> dcaro, which job will you actually be trigegring?
14:56:41 <dcaro> fabiand: hehehe, i actually was thinking on add the option to the timeout plugin to reset the time when output is generated (for example if it's having network issues and has to retry a few yum mirrors), if the issue persists I'll dig in to fix it too
14:56:51 <dcaro> fabiand: http://jenkins.ovirt.org/job/ovirt-appliance_engine-3.5_master_merged/
14:57:00 <dcaro> fabiand: did not trigger it as it already has a pending build
14:57:10 <dcaro> fabiand: but reconfigured it
15:00:27 <dcaro> moving
15:00:36 <dcaro> #topic puppet & foreman
15:00:58 <dcaro> #Sent a patch to move the jenkins_slave related manifests to it's owwn module
15:01:04 <dcaro> oops...
15:01:08 <dcaro> #info Sent a patch to move the jenkins_slave related manifests to it's owwn module
15:01:33 <bkorren> dcaro, great, this leads me to question (not about the patch)
15:02:17 <bkorren> dcaro, we have r10k there right? any docs on how it works? can we use feature branches in puppet?
15:02:45 <ewoud> bkorren: every puppet branch is translated into a puppet environment by r10k
15:03:12 <dcaro> bkorren: we have a basic setup, I think there are no docs, we are using only the production branch, but all of them are created on foreman, so they will be deployed to the puppetmaster if created
15:03:40 <dcaro> bkorren: we don't follow any feature-branch flow actually as there's not too much development around them either and we did not agree on a common structure
15:04:16 <bkorren> dcaro, hmm... is it even possible to use feature branches with gerrit?
15:04:46 <dcaro> bkorren: yes you can, but there are no pull requests (if that'ss where you want to go)
15:05:26 <ewoud> bkorren: yes, you can create a review against a branch
15:05:47 <ewoud> it will still need to be merged, but in theory you could do that yourself
15:05:53 <ewoud> if you want it in practice is another
15:06:15 <bkorren> dcaro, I don't mind rebasing to the dev branch when the code is ready, but will gerrit actually let me push arbitrarily named branches?
15:06:39 <dcaro> bkorren: it's not configured to allow it right now, but you can configure it to allow that
15:08:18 <dcaro> bkorren: we can discuss that in the mailing list. If we are going to adopt a differwnt workflow thatn the rest of projects (and force a different gerrit configuration) I'd like to be sure it's better than the alternative
15:09:39 <bkorren> dcaro, ok, I think puppet is special in this case because you need a master to do interesting stuff, but I'll take it to the mailing list when it'll be more relevant
15:11:18 <dcaro> bkorren: ok, good, I'll be glad to discuss it :)
15:11:52 <dcaro> okok, we are almost out of time, any other issues that I forgot about?
15:12:04 <bkorren> dcaro, the wiki :)
15:12:52 <dcaro> bkorren: well, the main promoters to move the wiki are not around, want to talk about it anyhow?
15:13:35 <bkorren> dcaro, I just wanted to remind bkp to write some of what he told us this morning
15:13:51 <dcaro> bkorren: xd
15:13:55 <bkorren> e.g. in the mailing list
15:16:13 <bkp> bkorren: It's on my list!
15:16:22 <bkorren> bkp, thanks!
15:16:28 <bkp> Granted, my list is 5 km long...
15:16:44 <dcaro> I forgot!
15:17:27 <dcaro> due to some infra members being on the other side of the atlantic, we might want to change the meeting time to a later hour
15:18:02 <dcaro> I'll send an email to the list so everyone can comment, but for the people here, what do you think?
15:18:23 <dcaro> (The idea was to move it to ~6:00pm CEST)
15:19:48 <dcaro> Better? Worse? ideas?
15:20:09 <bkorren> dcaro, seems reasonable
15:20:16 <bkorren> dcaro, to me
15:21:54 <eedri> dcaro, no prob from my side
15:22:07 <eedri> ewoud, YamakasY Rydekull_ misc ?
15:22:28 * ewoud reads
15:22:56 <ewoud> since that would move it outside of office hours for me, that would be better to me
15:23:45 <dcaro> okok, out of time, i'll start the discussion on the list
15:24:01 <ewoud> so +1 from my side
15:26:09 <dcaro> #endmeeting