14:03:54 <ewoud> #startmeeting infra weekly
14:03:54 <ovirtbot> Meeting started Mon Aug  5 14:03:54 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:03:54 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:04:26 <ewoud> #chair dcaro_ obasan
14:04:26 <ovirtbot> Current chairs: dcaro_ ewoud obasan
14:04:45 <obasan> eedri will join in a minute
14:05:24 * ewoud is looking for the minutes of last week
14:06:36 <ewoud> http://ovirt.org/meetings/ovirt/2013/ovirt.2013-07-29-14.02.html
14:06:47 <ewoud> it looks like we didn't add any action items
14:07:06 <ewoud> #topic hosting
14:07:09 * eedri here
14:07:23 <ewoud> #chair eedri
14:07:23 <ovirtbot> Current chairs: dcaro_ eedri ewoud obasan
14:07:36 <ewoud> I think there was something about not adding more slaves to rackspace01
14:07:52 <dcaro_> ewoud: actually we had to delete one :S
14:08:07 <ewoud> I did see that
14:09:09 <eedri> ewoud, yes. space issues
14:09:10 <ewoud> #info one slave removed from rackspace01 because of disk issue limitations
14:09:20 <eedri> ewoud, so new vms must be added only to rackspace02
14:09:25 <ewoud> any plans for the external storage?
14:09:33 <ewoud> #info new VMs should be added only to rackspace02
14:09:40 <eedri> ewoud, i think i sent a query on that, didn't got reply yet
14:10:44 <ewoud> eedri: ok
14:10:59 <ewoud> #info eedri is inquiring about external storage
14:11:20 <ewoud> I don't think we had any other issues last week
14:12:48 <ewoud> anything else on hosting?
14:13:59 <obasan> ewoud, I wasn't available during the the whole week so I'm kinda out of the loop :)
14:14:11 <eedri> ewoud, not that i know of
14:14:20 <ewoud> #info itamar updated gerrit to 2.6.1
14:14:21 <eedri> ewoud, there were some jenkins issues
14:14:26 <ewoud> might be nice for the minutes
14:15:38 <ewoud> eedri: any more details?
14:15:50 <eedri> ewoud, yes.
14:16:06 <eedri> ewoud, it seems that post gwt upgrade all jobs running gwt compilation started to fail
14:16:14 <eedri> ewoud, and still do ocasionally i think
14:16:21 <eedri> ewoud, dcaro_ started looking at it
14:16:41 <eedri> ewoud, i believe the new compilations require more memory that existing vms might not have
14:16:54 <eedri> ewoud, or not able to run it in parallel to other jobs
14:17:11 <eedri> ewoud, so jobs like gwt_user/gwt_admin/create_rpms are affected
14:17:45 <dcaro_> ewoud: eedri: yep, they fail because the slaves get memory problems, I added more swap (1 gb) to each slave and I created some exclusion groups in jenkins to avoid them from running in parallel, but it's still fdailing sometimes. Have to tune it
14:18:11 <eedri> ewoud, i moved all these jobs to run on a specific vm that has only 1 executor - not sure if that solves it completely
14:18:41 <eedri> dcaro_, can you grab some of the gwt developers to help maybe?
14:18:47 <eedri> dcaro_, and close this issue?
14:19:08 <ewoud> #info some issues involving gwt build jobs, possibly memory related - it's being monitored
14:19:30 <dcaro_> I'll try, but I don't think that the problem is more than just lack of memory :S
14:19:49 <ecohen> dcaro_, feel free to grab awels to help you with these issues after the meeting (I recall that we had an e-mail thread on that matter?)
14:19:59 <YamakasY_Mobile> hheeeyyyaa!!! I can install vdsm using puppet :D
14:21:25 <ewoud> anything else on hosting?
14:21:56 <YamakasY_Mobile> oh almost fully
14:21:58 <eedri> dcaro_, we can try allocating more memory to f19 now, that we removed one vm
14:22:10 <dcaro_> eedri: that's true :)
14:24:11 <ewoud> #topic jenkins
14:24:23 <ewoud> we already mentioned some build issues on hosting
14:24:45 <ewoud> my jenkins slave is also being removed
14:24:57 <ewoud> I don't think we have to backup anything, do we?
14:25:07 <eedri> ewoud, no, just a slave afail
14:25:11 <eedri> ewoud, best to scan it
14:25:21 <eedri> ewoud, just to make sure we didn't back anything up there
14:25:26 * eedri is removing it from jenkins now
14:26:03 <ewoud> any other issues, news or other details?
14:26:14 <eedri> removed
14:26:29 <dcaro_> there's a new job for ovirt-scheduler, it's not working yet though
14:26:41 <eedri> dcaro_, yea.. i saw that - who requested it?
14:26:54 <dcaro_> lazslo
14:26:57 <eedri> dcaro_, i don't think it's wise to add jobs per patch unless they are very fast
14:27:04 <eedri> dcaro_, so per commit is OK
14:27:17 <dcaro_> eedri: it should be very fast
14:27:20 <eedri> dcaro_, ok
14:27:31 <dcaro_> when it works :)
14:28:03 <eedri> ewoud, what do you think on utilizing internal resources for testing patches? and publishing the results on upstream jenkins?
14:28:21 <eedri> ewoud, it's a possibility that jenkins publisher plugin provides
14:28:46 <eedri> ewoud, assuming we're not going to be able to support per patch jobs on existing infra
14:29:22 <ewoud> eedri: not sure
14:29:23 <eedri> ewoud, you think there might be security issues with that approach?
14:30:15 <obasan> eedri, the whole purpose of this plugin is to provide this exact service > private jenkins to public jenkins. I doubt that it's problematic security wise
14:30:45 <ewoud> eedri: security from what perspective? from ovirt or the one hosting the internal resources?
14:31:22 <eedri> ewoud, from an internal jenkins user modifying / reviewing upstream gerrit patches
14:31:29 <muertochongo> ello all
14:31:35 <eedri> ewoud, but now that i think on it, it should be a problem
14:31:36 <ydary> ovedo: ping
14:31:37 <eedri> shouldn'
14:31:39 <eedri> tt
14:31:56 <muertochongo> anyone know a reason NOT to rsync the /iso's from one manager to another?
14:32:03 <ovedo> ydary, already pinged you on personal message
14:33:22 <eedri> ewoud, just a thought that we need to think on it, cause from what i see, it might be that new vms on rackspace might not be able to support too much jobs per patch as well
14:33:42 <ewoud> eedri: I'm going to think about it, can't fully thing it through right now
14:33:43 <obasan> muertochongo, I don't see a problem with that...
14:33:58 <muertochongo> yeah just felt like I was missing somehting
14:34:02 <muertochongo> ok ty
14:34:16 <ewoud> eedri: I do think it's good if you also put it on the ML
14:34:22 <muertochongo> anyone here use the Dell HIT KIT with EQ San?
14:34:38 <eedri> ewoud, i can start a thread on it
14:35:57 <ewoud> eedri: cool
14:36:03 <obasan> eedri, +1
14:36:11 <ewoud> moving on to puppet if there's nothing else on jenkins
14:36:24 <ewoud> I don't see anything like backups on the slave, so I think we can delete it
14:36:28 <muertochongo> puppet has some kewl stuff
14:36:38 <muertochongo> especially for node deploys
14:36:42 <muertochongo> and nagios integration
14:36:50 <ewoud> #topic puppet
14:37:57 <ewoud> we should re-visit the deployment using jenkins
14:38:41 <dcaro_> ewoud: what do you mean?
14:38:57 <ewoud> dcaro_: you sent a mail about it, I think there's also a trac ticket open for it
14:39:27 <dcaro_> ewoud: ahhh, the deployment of the manifests xd
14:39:29 <dcaro_> yeah
14:39:31 <ewoud> currently we use a post-update hook in /var/lib/puppet/puppet.git which essentially checks out every branch to /etc/puppet/environments, does a git submodule update --init everywhere
14:42:53 <ewoud> is there an easy way to do this in jenkins (pretty sure there is) but also rsync it
14:43:22 <obasan> ewoud, my familiarity with git submodules is 0 :)
14:43:45 <dcaro_> Theres a trac on it. I think that the easyest way would be reusing the script the post-hook uses, but from a jenkins job
14:43:58 <dcaro_> (maybe modify it a little)
14:44:52 <dcaro_> jenkins has some submodule options if I remember correctly too... not sure if they do what we want though
14:45:16 <ewoud> I doub tit
14:45:27 <ewoud> because the script also removes removed branches
14:50:54 <ewoud> but in the mean time, can't we just have jenkins push it forward?
14:58:22 <ewoud> sorry, was afk
14:58:34 <ewoud> dcaro_: eedri any opinions on that?
14:59:05 <eedri> ewoud, sorry, i wasn't involved latetly with puppet too much
14:59:16 <eedri> ewoud, so i'm not sure i can pitch in
15:00:08 <dcaro_> ewoud: should be easy, but I really think that we should not maintain a parallel git repo just to run the hooks
15:00:13 <ewoud> eedri: in essence my proposal is that we create a jenkins job that monitors a gerrit repo and on every change, it just pushes all branches as a mirror
15:01:36 <ewoud> dcaro_: maybe we can simplify the plugin and just ensure that it's set up for the production environment (== production branch)
15:01:43 <ewoud> s/plugin/hook/
15:02:22 <dcaro_> ewoud: I don't mind the hook, the thing I don't like is the repo, it's a source of conflicts ;)
15:02:39 <ewoud> dcaro_: agreed
15:03:43 <ewoud> anyway, let's take that to the list then and end the meeting
15:05:05 <dcaro_> ewoud: sure
15:12:03 <eedri> ewoud, +!
15:12:04 <eedri> +1
15:13:27 <ewoud> #endmeeting