14:02:44 <ewoud> #startmeeting infra weekly
14:02:44 <ovirtbot> Meeting started Mon Jun 10 14:02:44 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:44 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:02:53 <ewoud> #chair knesenko
14:02:53 <ovirtbot> Current chairs: ewoud knesenko
14:03:15 <knesenko> dcaro: obasan
14:03:17 <knesenko> edong23:
14:03:19 <knesenko> eedri_:
14:03:20 * eedri_ here
14:03:22 <knesenko> edong23: ignore
14:03:26 * dcaro here
14:03:35 <ewoud> #chair eedri_ dcaro
14:03:35 <ovirtbot> Current chairs: dcaro eedri_ ewoud knesenko
14:03:45 <ewoud> Agenda from http://www.ovirt.org/Infrastructure_team_meetings#2013-06-10
14:03:50 <ewoud> Introductions
14:03:50 <ewoud> Review of action items
14:03:50 <ewoud> Hosting
14:03:50 <ewoud> Puppet
14:03:50 <ewoud> Jenkins
14:03:52 <ewoud> Other business?
14:03:55 <ewoud> Trac review
14:05:23 <ewoud> any additions?
14:05:50 <ewoud> #topic introductions
14:06:42 <ewoud> any introductions?
14:06:49 <knesenko> :\
14:07:10 * obasan here
14:07:21 <ewoud> #chair obasan
14:07:21 <ovirtbot> Current chairs: dcaro eedri_ ewoud knesenko obasan
14:07:55 <knesenko> ok next
14:07:57 <obasan> I don't think that there are introductions
14:08:06 <ewoud> #topic review of action items
14:08:09 <ewoud> from the minutes
14:08:20 <ewoud> * continue puppet repo push to foreman on infra list
14:08:20 <ewoud> * add an additonal administrator for gerrit.ovirt.org to add redundancy and backup for weekends
14:08:23 <ewoud> * re-discuss full backup of infra services once we have ext. storage
14:08:25 <ewoud> * reschdule infra meeting time after reviewing doodle results
14:08:28 <ewoud> * send request to add ovirt as project to linkedin
14:09:19 <ewoud> I did see a thread about the linkedin project, but that's not very active anymore
14:09:26 <knesenko> ewoud: yeah'
14:09:31 <obasan> ewoud, I have read your puppet suggestion
14:09:43 <ewoud> obasan: cool
14:09:44 <knesenko> for a registration I need an *.ovirt.org email
14:09:57 <knesenko> and we don't have one to register .
14:10:15 <ewoud> knesenko: I think quaid suggested a private mailing list, would that do?
14:10:16 <knesenko> quaid: hey , what about email account for linkedin ?
14:10:25 <knesenko> ewoud: I think so .
14:10:39 <ewoud> obasan: any comments?
14:10:56 <obasan> ewoud, I am not familiar with hiera. I do use setting params in foreman which I find very comfortable
14:11:11 <obasan> ewoud, and I recommend we will use that. unless someone here experieced with hiera and will disagree
14:12:30 <obasan> ewoud, regarding librarian puppet, I think it's worth a shot
14:13:03 <ewoud> obasan: ok, maybe we should summarize this in the puppet topic in the agenda
14:13:13 <obasan> ewoud, ok.
14:13:29 <ewoud> knesenko: I think you did fix something with gerrit
14:13:43 <ewoud> jenkins has an option to restart, but you did get SSH access?
14:13:59 <ewoud> I will follow up on the doodle results
14:14:12 <ewoud> #action ewoud reschedule infra meeting with the doodle results
14:17:01 <knesenko> hello
14:17:02 <knesenko> :) :)
14:17:12 <knesenko> what is the next item ?
14:17:24 <ewoud> knesenko: I was wondering how the gerrit item went
14:17:33 <ewoud> 16:08:18 < ewoud> * add an additonal administrator for gerrit.ovirt.org to add redundancy and backup for weekends
14:17:46 <knesenko> ewoud: this one for dcaro
14:17:52 <knesenko> ewoud: that's not me :)
14:17:54 <ewoud> ah ok
14:17:56 <eedri_> ewoud, i think itamar added david, but just ssh for now not sudo
14:18:24 <ewoud> let's move on to hosting then
14:18:26 <ewoud> #topic hosting
14:18:40 <dcaro> eedri_: eedri_ actually, I have sudo, but I'm not allowed to change nothing without notice
14:19:06 <ewoud> dcaro: ok
14:19:19 <ewoud> knesenko: you wrote a mail about the infra design
14:19:41 <knesenko> ewoud: and you replied
14:20:00 <ewoud> yes, I think it's a good idea in general
14:20:38 <knesenko> dcaro: eedri_ obasan we need you guys to review the plan as well
14:20:50 <knesenko> we need to push it and start working on the infra design asap
14:21:02 <knesenko> eedri_: what about the storage for us ? any news ?
14:21:07 <dcaro> knesenko: ok
14:21:28 <eedri_> knesenko, we need to send a query to whoever manage the rackspace account
14:21:31 <ewoud> knesenko: maybe a small drawing could help
14:21:32 <eedri_> quaid, ping
14:22:29 <knesenko> ewoud: I can create one
14:23:31 <ewoud> knesenko: how would you like to move forward
14:24:20 <knesenko> ewoud: first of all I would like to see another team members reviewing this design
14:24:22 <itamar> eedri - i gave david sudo to fix his findings. i asked him to not do changed before communicating them for now.
14:24:23 <eedri_> ewoud, i suggest giving a few days to ask for VM (engine) + external storage
14:24:34 <eedri_> itamar, thanks
14:24:40 <knesenko> ewoud: after a review, we will prepare some schema/drawing
14:25:04 <knesenko> and after will prepare a work plan
14:25:19 <knesenko> review the plan .... if it is ok so then go go :)\
14:25:22 <knesenko> sounds good ?
14:26:40 <ewoud> knesenko: yes
14:26:49 <knesenko> ok
14:26:52 <dcaro> knesenko: +1
14:27:01 <ewoud> knesenko: I'd prefer to leave the alterway design as we're now building it and start setting up rackspace properly
14:27:17 <ewoud> then when we have rackspace running, look at alterway
14:27:20 <eedri_> ewoud, +1
14:27:27 <obasan> knesenko, +1
14:27:53 <knesenko> what about storage ? gluster ? external ?
14:27:55 <eedri_> ewoud, knesenko so it means we're checking if we can get our hands on a VM for engine and storage for rackspace install
14:28:01 <knesenko> i am talking about rackspace
14:28:08 <knesenko> eedri_: ok
14:28:23 <eedri_> knesenko, if we can get external good, if not we'll go with gluster
14:28:34 <ewoud> eedri_: we could either use one of the rackspace hosts as engine or a separate VM
14:28:53 <eedri_> ewoud, no, we said using AIO is not optimal
14:29:06 <ewoud> also, what do we want to run on rackspace besides jenkins slaves?
14:29:08 <eedri_> ewoud, it's better to utilize the 2 hosts as hypervisors
14:29:10 <ewoud> maybe a master?
14:29:17 <ewoud> jenkins master*
14:29:30 <eedri_> ewoud, i wouldn't migrate the master for now
14:29:39 <ewoud> eedri_: I mean in the longer run
14:29:53 <eedri_> ewoud, those servers are intended for jenkins slaves + bare metal/nested for functional tests
14:30:08 <eedri_> ewoud, mostly hosts for functional/automation/system tests
14:30:16 <ewoud> I still think jenkins slaves don't need shared storage because it generally is slower
14:30:37 <ewoud> and it's only useful if you want HA, but for slaves that's not really needed
14:30:52 <ewoud> so I'd prefer to use local storage for jenkins slaves
14:31:06 <eedri_> ewoud, ok.. i have no problem with starting off without external storage, just concerned about migrating in the future
14:31:21 <eedri_> ewoud, will be able to migrate it to external storage afterwards?
14:31:41 <ewoud> I don't know, but installing jenkins slaves should be easy when we have puppet set up properly
14:32:10 <ewoud> eedri_: I'd really like to see jenkins slaves as throw away VMs in terms of HA/backup
14:32:19 <knesenko> ewoud: eedri_ so whats the decision ?
14:32:38 <ewoud> if they're down, I'd hope the master just schedules the task somewhere else
14:32:48 <eedri_> ewoud, i agree
14:32:53 <eedri_> ewoud, jenkins slaves are stateless
14:33:03 <ewoud> eedri_: exactly
14:33:23 <eedri_> ewoud, should be, as long as they are managed with puppet/foreman
14:33:45 <eedri_> knesenko, i think we'll query about external storage for rackspace, but not block on it
14:33:49 <eedri_> knesenko, we still need a VM for engine
14:34:14 <knesenko> eedri_: ok ... so lets take it to the mailing list
14:34:39 <eedri_> #action eedri to send email requesting external storage & addtional vm on rackspace
14:35:29 <eedri_> #action knesenko to prepare a diagram for new infra layout of ovirt-engine
14:36:12 <ewoud> knesenko: can you also add the network design for rackspace hosts in your diagram?
14:36:41 <knesenko> ewoud: ok
14:37:06 <ewoud> any more on hosting?
14:38:03 <ewoud> #topic puppet
14:38:17 <eedri_> ewoud, wanted to ask about migrating vms to alterway02
14:38:22 <ewoud> oh yes
14:38:24 <ewoud> #topic hosting
14:38:44 <eedri_> ewoud, but maybe we want to wait with more migrating stuff until we'll improve alterway deployment?
14:38:58 <ewoud> I noticed that when I use foreman, I sometimes lose my network to it
14:39:03 <eedri_> ewoud, i.e - move ovirt-engine to a VM
14:39:18 <eedri_> ewoud, i thought that issue was fixed
14:39:48 <ewoud> eedri_: the continuous packet loss was fixed, but sometimes I fully lose it
14:39:58 <ewoud> haven't investigated it properly, but I'll mail it
14:40:01 <eedri_> ewoud, can you send a ping to alterway in it?
14:40:07 <ewoud> yes
14:40:35 <ewoud> eedri_: but I think we should continue using stuff on alterway02 more if it's stable
14:40:57 <eedri_> ewoud, my only concern is to relay more and more on an AIO installation with limited storage
14:41:17 <eedri_> ewoud, rather then halting migration until we can improve the layout there
14:41:30 <eedri_> ewoud, i.e moving the engine to a vm and re-adding the hosts as hypervisor
14:41:59 <eedri_> ewoud, what do you think?
14:42:16 <ewoud> eedri_: I was thinking that we could continue deploying, then move jenkins to a VM, reinstall alterway01 to a host, then migrate the VMs
14:42:32 <ewoud> it may be a bit more work and downtime involved
14:43:24 <eedri_> ewoud, ok, what about extending current storage there, is that an issue? knesenko ?
14:43:25 <ewoud> linode01 is currently stable I think, so maybe we should just continue building the current VMs on it, but don't migrate new ones
14:44:25 <knesenko> eedri_: ewoud ok I talked to fsimonce and he told me that extending the LV and resizing the filesystem should do the work
14:44:45 <knesenko> it wont affect the localstorage DC
14:45:50 <eedri_> knesenko, ok. can you do that then on alterway2?
14:45:59 <knesenko> let me rephrase it: it should not affect the local DC
14:46:02 <eedri_> knesenko, resize it to the whole free space
14:46:07 <knesenko> eedri_: yeah I can do it
14:46:15 <eedri_> knesenko, then we can continue on the migration/adding new vms
14:46:26 <eedri_> knesenko, thanks
14:46:39 <ewoud> btw, what about storage at alterway?
14:47:06 <ewoud> I feel we shouldn't ask for too much since it's sponsored so I'm thinking about gluster
14:47:54 <knesenko> ewoud: but we can try :)
14:48:09 <ewoud> knesenko: true
14:49:10 <ewoud> eedri_: does that answer your question?
14:49:34 <ewoud> and any other hosting items?
14:49:48 <eedri_> ewoud, yes
14:50:01 <ewoud> #topic puppet
14:50:04 <eedri_> ewoud, i mean yes, answeed my questions, no more items on hosting
14:50:11 <ewoud> eedri_: I assumed as much :)
14:50:18 <eedri_> #action knesenko extend current LV on alterway02 for ovirt-engine
14:50:35 <ewoud> ok, obasan already gave a few comments on puppet proposal
14:50:59 <ewoud> 6:10:55 < obasan> ewoud, I am not familiar with hiera. I do use setting params in foreman which I find very comfortable
14:51:02 <ewoud> 16:11:09 < obasan> ewoud, and I recommend we will use that. unless someone here experieced with hiera and will disagree
14:51:05 <ewoud> 16:12:29 < obasan> ewoud, regarding librarian puppet, I think it's worth a shot
14:52:08 <ewoud> obasan: I agree with getting started with params in foreman
14:52:14 <obasan> ewoud, I am sure dcaro will have more insights
14:53:22 <obasan> ewoud, maybe hiera is more comfortable. I don't know. I use foreman
14:53:51 <ewoud> and regarding librarian I don't intend to work on it just yet because submodules works for now
14:54:32 <ewoud> obasan: one advantage of hiera is that you can also add defines, so it can replace the ENC
14:54:42 <dcaro> there's one aspect that I don't like of foreman, and it's that it does not allow to undo changes
14:55:00 <dcaro> it does store an audit log with what was changed
14:55:01 <ewoud> dcaro: undo changes as in parameters?
14:55:42 <dcaro> for example, if you want to rollback a parameter change to something you had 2 weeks ago
14:56:57 <ewoud> dcaro: with hiera if you store it in a git backend, you can revert it
14:58:45 <dcaro> yep, that was my point, but hiera does not have a nice frontend :), and it's not integrated with foreman (you can set up foreman as a hiera backend though, but it has the same problem of versioning)
14:59:33 <ewoud> dcaro: do you have experience using hiera?
14:59:53 <dcaro> just the few tests I did by myself, no real experience
15:00:17 <ewoud> ok, I haven't used it
15:00:53 <ewoud> anyway, are you guys OK with me pushing the repo to gerrit and then just go through normal code review?
15:01:42 <knesenko> yes
15:01:43 <knesenko> ok for me
15:01:50 <ewoud> ok
15:01:53 <eedri_> ewoud, +1
15:02:35 <obasan> ewoud, +1
15:03:32 <ewoud> obasan: I'd also like to start hacking on icinga/nrpe patches for our puppet env using exported resources
15:03:40 <ewoud> do you have experience with it?
15:04:09 <obasan> ewoud, I have some experience with nagios and nrpe
15:04:16 <obasan> ewoud, but not with exported resources
15:04:20 <ewoud> ok
15:04:43 <dcaro> ewoud: +1
15:05:03 <ewoud> once it's in gerrit I'll just add you guys as reviewers
15:05:04 <dcaro> ewoud: I have some experience with exported resources
15:05:18 <ewoud> I only have experience with munin exporting
15:07:34 <ewoud> ok, anything about jenkins?
15:07:38 <ewoud> #topic jenkins
15:07:39 <eedri_> ewoud, yes
15:08:09 <eedri_> ewoud, findbugs gerrit job re-enabled after dcaro fixed some issues on gerrit + gerrit was upgraded to latest
15:08:15 <knesenko> finished the engine upgrade job
15:08:23 <eedri_> ewoud, need to monitor and see if jobs stil fail on cloning
15:09:28 <eedri_> also, engine-upgrade job gerrit was added to run on per patch
15:09:47 <eedri_> on patches relevant (packaging and db-scripts)
15:10:03 <ewoud> ok
15:10:07 <eedri_> also need to monitor if jenkins can handle current queue of jobs
15:10:20 <eedri_> ewoud, this is why we need to push forward rackspace asap
15:10:27 <ewoud> eedri_: I agree
15:13:37 <eedri_> ewoud, ok
15:13:42 <eedri_> ewoud, any more items on jenkins?
15:13:49 <knesenko> no
15:13:56 <eedri_> ewoud, actually we attended jenkins user conf laste week
15:14:04 <eedri_> ewoud, some interesting highligts
15:14:11 <knesenko> obasan: what about the ovirt lise iso job ?
15:14:14 <knesenko> live
15:14:19 <obasan> knesenko, I need to fix it
15:14:25 <knesenko> obasan: please do
15:14:32 <eedri_> #action obasan to fix ovirt-live job
15:14:47 <eedri_> #action eedri_ to send info from jenkins conf to infra list
15:17:24 <knesenko> ewoud: seems like we are done
15:21:03 <eedri_> next topic
15:22:31 <ewoud> sorry, had to help a colleague
15:22:49 <ewoud> #topic other items
15:22:59 <ewoud> as said earlier, I will follow up on the new meeting time
15:23:06 <ewoud> anything else?
15:23:40 <knesenko> ewoud: nope
15:23:48 <eedri_> ewoud, trac tickets
15:25:13 <ewoud> eedri_: yes, we should review them but I think we already ran out of time :S
15:25:21 <eedri_> ewoud, yes.
15:25:27 <eedri_> ewoud, we should change the meeting time
15:25:42 <eedri_> ewoud, maybe split it to 2 meeting
15:28:16 <ewoud> eedri_: maybe we should
15:28:37 <ewoud> eedri_: I'll take that into consideration when suggesting a new meeting time
15:28:42 <ewoud> then I'll end it now
15:28:59 <ewoud> #endmeeting