14:04:03 <ewoud> #startmeeting infra weekly meeting
14:04:03 <ovirtbot> Meeting started Mon Aug 26 14:04:03 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:04:03 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:04:08 <ewoud> #chair eedri_
14:04:08 <ovirtbot> Current chairs: eedri_ ewoud
14:04:15 <ewoud> ping obasan dcaroest Rydekull
14:04:20 <obasan> ewoud, hello!
14:04:22 * obasan here
14:04:25 <ewoud> #chair obasan
14:04:25 <ovirtbot> Current chairs: eedri_ ewoud obasan
14:04:51 * dcaroest is here too
14:04:55 <ewoud> #chair dcaroest
14:04:55 <ovirtbot> Current chairs: dcaroest eedri_ ewoud obasan
14:05:42 <ewoud> agenda: http://www.ovirt.org/Infrastructure_team_meetings#2013-08-26
14:05:53 <ewoud> minutes from last week: http://resources.ovirt.org/meetings/ovirt/2013/ovirt.2013-08-19-14.01.html
14:06:02 <eedri_> obasan, ?
14:06:07 <eedri_> ok
14:06:10 <obasan> eedri_, here
14:06:13 <ewoud> #topic items from last week
14:06:28 <ewoud> obasan update jenkins-ovirt.org to the latest lts
14:06:35 <Rydekull> o7
14:06:38 <ewoud> obasan: any progress on that?
14:06:40 <ewoud> #chair Rydekull
14:06:40 <ovirtbot> Current chairs: Rydekull dcaroest eedri_ ewoud obasan
14:06:44 <obasan> ewoud, done. of course
14:06:50 <ewoud> obasan: very good :)
14:06:58 <obasan> ewoud, Also another small update on my side
14:07:07 <obasan> ewoud, the coverity job should now be fixed once and for all
14:07:07 <Rydekull> (and today I dont have to leave, cannot guarantee I wont be disturbed though) :-)
14:07:16 <obasan> ewoud, since a patch was merged upstream to fix it
14:07:40 <obasan> ewoud, also update some crucial jenkins plugins
14:07:45 <obasan> ewoud, updated*
14:07:53 <ewoud> obasan: what was the coverity issue again?
14:08:12 <ewoud> Rydekull: I doubt anyone can guarantee they won't be disturbed :)
14:08:14 <obasan> ewoud, the problem was in some new java convention that coverity thought that was problematic
14:08:36 <obasan> ewoud, so coverity thought that the project failed compiling
14:08:41 <obasan> ewoud, while it actually did not.
14:08:49 <ewoud> obasan: ok, good that it's fixed upstream
14:09:07 * ewoud doesn't like keeping patches
14:09:11 <ewoud> knesenko install f18 + 2 centos 6.4 on rackspace2
14:09:20 <ewoud> anyone knows if knesenko had time for that?
14:09:42 <obasan> ewoud, I don't know.
14:09:45 <obasan> eedri_, do you know?
14:09:56 <eedri_> ewoud, i think centos vms are installed and running on jenkins
14:10:08 <eedri_> ewoud, but they might be missing rpms, so not sure we're using them yet
14:10:31 <eedri_> ewoud, i suggest migrating the current jobs that use rhell64 label to centos64
14:10:40 <eedri_> ewoud, and finding out what rpms are missing
14:10:49 <ewoud> eedri_: sounds good
14:11:01 <ewoud> #topic hosting
14:11:02 <eedri_> ewoud, don't think he installed f18 vms yet
14:11:20 <ewoud> I saw a mail about more storage for rackspace
14:13:11 <ewoud> anyone can describe to me how it's setup?
14:13:25 <eedri_> ewoud, i think david added the disks, not sure if he updated the fs (lvm?)
14:13:26 <eedri_> dcaroest, ?
14:13:38 <ewoud> in the minutes from last week I see '14:03:29 <knesenko> its a kind of external … :)'
14:14:13 <ewoud> does that mean each host now has another 2 TB disk in it?
14:14:35 <dcaroest> The storages were added phisycally, but not logically to the host (meaning they were not partitioned or mounted)
14:16:17 <ewoud> ok, would it be a candidate to try gluster?
14:17:00 <eedri_> ewoud, i think it should be probably better than local storage
14:17:11 <dcaroest> I'm not familiar with gluster setups of ovirt, but if it's possible it would be nice to have some ha machines there
14:18:01 <ewoud> especially if the self hosted ovirt feature is ready
14:18:08 <ewoud> anyone willing to pick that up?
14:18:53 <Rydekull> gluster is very easy to setup fyi
14:19:54 <Rydekull> and ovirt supports gluster, yes, however, there are some caveats, not sure how our datastores are configured today
14:20:06 <ewoud> Rydekull: do I hear a volunteer?
14:20:44 <Rydekull> I have no access to the new hosts, so someone will have to spend more time setting my accounts up then it takes to setup gluster :-)
14:20:57 <obasan> I have 0 experience with gluster
14:21:27 <ewoud> Rydekull: I think you having access to those hosts is a good thing by itself
14:21:38 <Rydekull> Im glad you think that ;-)
14:22:05 <Rydekull> But yeah, sure, I can partition disks and install gluster and configure that
14:22:15 <Rydekull> If I get access, that is
14:22:41 * ewoud doesn't know how the access is set up
14:22:50 <ewoud> but we'll manage that after the meeting
14:23:04 <ewoud> so all in favor of setting up gluster?
14:23:35 <ewoud> eedri_: obasan dcaroest Rydekull ?
14:23:42 <Rydekull> +1
14:23:54 <obasan> +1
14:24:53 <ewoud> #action Rydekull set up gluster on rackspace hosts
14:24:56 <eedri_> will it affect current data?
14:25:08 <ewoud> eedri_: I'd assume that since it's a new storage domain, it shouldn't
14:25:09 <eedri_> meaning all vms that are currently running (vm19)
14:25:14 <eedri_> ewoud, ok
14:25:21 <ewoud> Rydekull: can you confirm that?
14:25:33 <Rydekull> Confirmed, if it is like described above with the disks
14:25:39 <eedri_> ewoud, we also decided that once that storage is up and running we can migrate backups to it
14:25:46 <Rydekull> No
14:25:49 <Rydekull> No no no
14:26:11 <eedri_> Rydekull, we have 2 disks there
14:26:24 <eedri_> Rydekull, one should be raid 1 mirror - for backups
14:26:27 <Rydekull> No
14:26:33 <eedri_> Rydekull, the other one is for jenkins slaves
14:26:34 <Rydekull> That's not backup, that's diskredundancy
14:26:39 <Rydekull> Backups are stored somewhere else
14:26:47 <ewoud> Rydekull: but rackspace hosts are just jenkins slaves
14:26:51 <Rydekull> Hmm
14:26:55 <Rydekull> Right
14:26:55 <eedri_> Rydekull, it is somewhere lese
14:27:00 <eedri_> Rydekull, from alterway
14:27:01 <Rydekull> No vital data, ok
14:27:10 <Rydekull> I give, store backups :-)
14:27:32 <eedri_> Rydekull, yes.. storing backups
14:28:00 <eedri_> Rydekull, from other machines like ovirt.org/foreman/jenkins/gerrit
14:28:18 <eedri_> Rydekull, what lies mostly on linode1 now
14:29:48 <dcaroest> sorry, I was away for a moment
14:30:24 <Rydekull> so... anything else?
14:37:44 * Rydekull doodles
14:41:17 <dneary> Hi everyone
14:41:18 * ewoud was interrupted
14:41:34 <ewoud> dneary: hi
14:41:38 <ewoud> anything else on hosting?
14:42:07 <obasan> ewoud, not that I can think of
14:42:11 <Rydekull> dneary: hello
14:42:17 <dneary> Still catching up - I see the storage issue is in the process of being resolved?
14:42:36 <ewoud> dneary: yes
14:42:40 <dneary> yay!
14:42:43 <ewoud> #topic jenkins
14:43:12 <ewoud> #info updated to latest LTS by obasan, including some critical plugins
14:43:42 <ewoud> #info in the process of adding more f18 and centos6.4 slaves
14:44:32 <ewoud> anything else on jenkins?
14:44:45 <eedri_> ewoud, network function tests for vdsm was added
14:45:01 <eedri_> ewoud, need to convert it to per patch mode
14:45:18 <ewoud> eedri_: I saw there were concerns about it becoming unreachable
14:45:30 <ewoud> is that solved or something we should keep monitoring?
14:46:05 <eedri_> ewoud, not afaik, we need gvallarelli response
14:46:20 <eedri_> ewoud, if those tests might potentially take a jenkins slave off, it's a problem
14:46:40 <ewoud> #info network functional tests added, still some concern that may make jenkins slaves unreachable
14:47:29 <dcaroest> eedri_: ewoud , under normal circumstances they should not, but during the creating of the jobs, the slave got disconnected a couple of times, meaning that if the tests fail, they can potentially take a slave offline
14:47:29 <ewoud> eedri_: I agree, and some dedicated mini jenkins slaves would be nice
14:47:57 <dcaroest> I was looking at linux containers to see if they can be used for those tests easily
14:48:14 <ewoud> and I'd assume that vdsm needs a lot less memory for a slave
14:48:21 <dcaroest> I was not able to set one up yet (only spent a couple of hours looking)
14:48:47 <dcaroest> yes, has relatively low resource needs
14:48:57 <eedri_> ewoud, maybe now with more storage on rackspace we can create mini vms
14:48:57 <gvallarelli> ewoud: it should not happen.
14:48:58 <dcaroest> but it needs networking exclusive for itself
14:49:38 <eedri_> ewoud, we'll need to poc running it per network patch and see if jenkins can handle the load
14:49:44 <gvallarelli> it also depends on how the test are written.
14:49:48 <ewoud> gvallarelli: should not, but how likely is it that if we have per patch jobs that one will take the slave with itself?
14:50:07 <gvallarelli> that's why I could not give 100% guarantee that the slave can be disconnected.
14:50:30 <gvallarelli> *can't
14:50:36 <ewoud> gvallarelli: ok
14:51:00 <ewoud> +1 to a poc
14:51:11 <gvallarelli> ewoud: at the moment we're porting and fixing tests, we should be in an enough stable condition soon.
14:51:51 <gvallarelli> eedri_: I'm not aware of any decision taken in regards of how to trigger the job in an automatic way.
14:52:09 <ewoud> gvallarelli: jenkins can be triggered on any new gerrit patch set
14:52:19 <eedri_> gvallarelli, i think this should network developer responsibility
14:52:38 <eedri_> gvallarelli, once you provide infra a way to differ network commit, we'll implement it on jenkins
14:53:00 <ewoud> eedri_: can't we use gerrit labels?
14:53:01 <gvallarelli> eedri_: for us the simplest thing would be to prefix for example the commit title with 'networking'
14:53:09 <eedri_> gvallarelli, so for example if it means to check the commit header ("network..") than we'll check that
14:53:15 <gvallarelli> +1
14:53:26 <eedri_> ewoud, gerrit labels?
14:54:10 <ewoud> since gerrit 2.6 you can create custom labels and I think someone suggested that
14:54:46 <dcaroest> the problem with labels is that you'll need one per each test group, for example, one label called network, one called storage... etc
14:55:21 <dcaroest> right now you can't add combo boxes or custom values for the labes (only +1, 0 -2 ...)
14:55:34 <ewoud> dcaroest: but I think putting it in the commit message is also messy
14:56:04 <dcaroest> ewoud: we can oput it in the gerrit comment
14:56:15 <dcaroest> but that will force the developer to create a comment
14:56:59 <dcaroest> (not that it's bad, but it's not fully automated, labes solution also has the same problem)
14:57:56 <ewoud> dcaroest: true, but prefixing the commit message is also not automated
14:58:21 <dcaroest> ewoud: it only needs the developer to add that message and push
14:58:42 <dcaroest> the other require the developer to go to gerrit and add a comment/review
14:59:16 <ewoud> still, I'm unsure I'd prefix the subject since that line is already so short
14:59:17 <eedri_> ewoud, it is
14:59:20 <eedri_> ewoud, partically
14:59:30 <eedri_> ewoud, via the gerrit commit template
14:59:34 <eedri_> ewoud, in ovirt-engine today
14:59:43 <dcaroest> ewoud: I would not prefix the subject, I'll put it on a tag in the last lines
14:59:54 <dcaroest> as the Change-id and the Bug.-Url
15:00:09 <ewoud> dcaroest: I'd be fine with that
15:00:17 <ewoud> since that's convention on metadata
15:00:33 <eedri_> ewoud, but if we already have the prefix in place for commits?
15:00:36 <eedri_> ewoud, can't we use that?
15:01:00 <dcaroest> what's the reason for that prefix? (just curious)
15:01:26 <ewoud> eedri_: if it's already there, fine but I wouldn't want to force it on people
15:01:41 <ewoud> what if a patch touches multiple sections?
15:02:43 <dcaroest> the subject will grow too much xd
15:03:00 <ewoud> exactly, and it's already short
15:03:06 <ewoud> 50 chars or something IIRC
15:03:58 <eedri_> ewoud, it's a problematic issue and i don't think we'll solve it here and now
15:04:06 <ewoud> eedri_: agreed on that
15:04:10 <eedri_> ewoud, this is why for networking sake, i asked gvallarelli to provide info
15:04:17 <eedri_> ewoud, since we only have now networkking tests
15:04:26 <ewoud> ok, we've gone a bit offtopic
15:04:30 <ewoud> anything else on jenkins?
15:04:32 <eedri_> ewoud, it can only come from dev side
15:05:55 <gvallarelli> eedri_: on vdsm side I'm aware only of our efforts
15:06:06 <gvallarelli> to provide tests that are runnable in a ci fashion.
15:06:17 <gvallarelli> So for now you can assume that's only networking functional tests
15:06:38 <gvallarelli> anyway we're recently receving contributions for functional tests on the virt side.
15:06:54 <gvallarelli> but I dunno how much time it will require for the virt team.
15:07:30 <eedri_> ewoud, there is an open ticket on fixing setup + upgrade job
15:07:42 <eedri_> ewoud, i would like it running cause i think it's very important to the project
15:07:54 <eedri_> ewoud, but didn't saw any volunteers to pick it up :/
15:09:42 <ewoud> eedri_: can you link it?
15:09:51 <eedri_> ewoud, the ticket?
15:10:22 <eedri_> ewoud, https://fedorahosted.org/ovirt/ticket/73
15:12:01 <dcaroest> eedri_: I still have the puppet and spamassasin tickets pending
15:12:34 <Rydekull> Hmm
15:12:53 <Rydekull> Oh, btw
15:13:00 <Rydekull> the RH-ticket seem to have gone through
15:13:11 <Rydekull> we now have a spf-record on ovirt.org
15:13:13 <Rydekull> ovirt.org.              300     IN      TXT     "v=spf1 include:linode01.ovirt.org ~all"
15:13:25 <Rydekull> Which hopefully will help us in not getting blocket at gmail
15:13:30 <Rydekull> blocked*
15:13:35 * Rydekull will monitor it a bit
15:13:44 <ewoud> Rydekull: nice
15:14:03 <ewoud> eedri_: what should be done about that ticket #73?
15:15:09 <eedri_> ewoud, there is a job already, just need to fix the code for it - since setup was changed to otopi
15:15:16 <eedri_> ewoud, and it used to work on legacy setup
15:15:30 <ewoud> eedri_: looks like the answers.file was changed, so maybe we should ask someone who was involved with changing the installer to otopi?
15:15:41 <dcaroest> Rydekull: it fails the test from kitterman tools
15:15:43 <ewoud> eedri_: it could indicate a regression in the code itself
15:15:49 <dcaroest> http://www.kitterman.com/spf/validate.html
15:16:08 <eedri_> ewoud, i doubt it, since the whole infra was changed, including answer file
15:16:16 <eedri_> ewoud, so the job is not relevant anymore
15:16:28 <eedri_> ewoud, needs to be rewritten for new setup
15:16:57 <ewoud> eedri_: I wouldn't know how to test it
15:17:18 <ewoud> Rydekull: shouldn't it be a:linode01.ovirt.org or something like that?
15:21:44 <ewoud> ok, closing the meeting
15:21:46 <ewoud> #endmeeting