14:02:21 <ewoud> #startmeeting infra weekly
14:02:22 <ovirtbot> Meeting started Mon Jul 22 14:02:21 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:22 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:02:24 <mskrivanek> leon80: it is feature-wise. Mostly. This console connectivity is an exception.
14:02:36 <ewoud> #chair Rydekull dcaro eedri_
14:02:36 <ovirtbot> Current chairs: Rydekull dcaro eedri_ ewoud
14:02:54 <mskrivanek> leon80: and then couple of hundred bugfixes more:)
14:03:29 <ewoud> so Kiril is away for 2 weeks
14:03:44 <ewoud> copy paste from http://www.ovirt.org/Infrastructure_team_meetings#2013-07-22
14:03:48 <ewoud> Introductions
14:03:48 <ewoud> Review of action items
14:03:48 <ewoud> Hosting
14:03:48 <ewoud> Puppet
14:03:48 <ewoud> Jenkins
14:03:51 <ewoud> Other business?
14:03:53 <ewoud> Trac review
14:04:21 <ewoud> I don't think we have new (potential) infra members, so I'd like to skip introductions this week
14:04:56 <ewoud> looking at the minutes from http://resources.ovirt.org/meetings/ovirt/2013/ovirt.2013-07-08-13.59.html I only see one action item
14:04:57 <leon80> mskrivanek: Thank you for the great explanation! :)
14:05:01 <ewoud> knesenko add linkedin profile link to the main page ovirt.org
14:05:38 * obasan here
14:05:43 <ewoud> #chair obasan
14:05:43 <ovirtbot> Current chairs: Rydekull dcaro eedri_ ewoud obasan
14:05:51 <ewoud> you didn't miss much, just the agenda
14:05:52 <obasan> ewoud, eedri will be joining in a few minutes
14:06:04 <obasan> ewoud, knesenko is away for 2-3 weeks :(
14:06:20 <ewoud> obasan: I know, he told me
14:06:51 <ewoud> #topic review of action items
14:06:58 <ewoud> knesenko add linkedin profile link to the main page ovirt.org
14:07:07 <ewoud> obasan: do you know if that was done?
14:07:22 <obasan> ewoud, I don't know. let's see
14:07:24 <ewoud> I do see the link in the about, but not sure that's it
14:07:24 <Rydekull> doesnt look like it
14:07:30 <obasan> ewoud, was not done
14:08:06 <ewoud> ok
14:08:11 <obasan> ewoud, what else is on the agenda?
14:08:21 <Rydekull> http://www.ovirt.org/Infrastructure_team_meetings#2013-07-22
14:08:21 <obasan> ewoud, I finished the openshift quota monitoring.
14:08:21 <ewoud> obasan: the usual I think
14:08:24 <Rydekull> same agenda
14:08:39 <ewoud> I'm leaving the action item then
14:08:46 <ewoud> #action knesenko add linkedin profile link to the main page ovirt.org
14:08:49 <ewoud> #topic hosting
14:09:03 <ewoud> #info obasan Added monitoring to the openshift quota
14:09:51 <ewoud> obasan: I think we should also add disk monitoring and set up some system so more people monitor the messages
14:10:12 <Rydekull> Backup, can we move it from linode01?
14:10:34 <ewoud> Rydekull: but where?
14:10:50 <obasan> ewoud, I thought about adding the alerts to the infra mailing list
14:10:53 <obasan> ewoud, what do you think?
14:10:55 <Rydekull> Like I offered, I can create a offsite-location for it reachable via sftp/ssh
14:11:07 <obasan> ewoud, which disks would you like to monitor?
14:11:23 <ewoud> obasan: all of them? :)
14:11:37 <ewoud> Rydekull: that's something I meant to reply to
14:11:39 <dcaro> Do we really need that much ammount of backups?
14:11:45 <obasan> ewoud, the jenkins slaves? :)
14:11:55 <ewoud> Rydekull: I think it can be good to have an offsite location
14:11:58 <Rydekull> dcaro: today it builds up to roughly 30 GB
14:12:06 <obasan> ewoud, that can be done once all the slaves are up
14:12:08 <Rydekull> dcaro: and I dont think we back everything up, honestly
14:12:31 <ewoud> obasan: oh, I mostly meant everything from linode but I guess we could monitor all
14:12:31 <dcaro> Rydekull: What does it include (and what do you think is missing)
14:12:44 <obasan> ewoud, what exactly is this linode?
14:12:53 <Rydekull> obasan: a machine, called linode01
14:12:59 <Rydekull> obasan: hosted on linode
14:13:04 <obasan> Rydekull, ewoud no problem monitoring it
14:13:06 <Rydekull> obasan: www.linode.com
14:13:07 <ewoud> obasan: it's a VM hosted at linode and initially everything was hosted there
14:13:20 <Rydekull> dcaro: Mostly gerrit, and some db-stuff
14:13:27 <ewoud> obasan: so website, mailing list, resources (RPMs)
14:13:39 <ewoud> gerrit is on another VM
14:13:45 <Rydekull> dcaro: and I feel that there probably is some new services, jenkins etc that atleast want configuration backed up
14:13:49 <ewoud> the website was moved to openshift
14:14:14 <Rydekull> dcaro: to be able to rebuild it quickly and easy if need be
14:14:21 * eedri here
14:14:30 <ewoud> #chair eedri
14:14:30 <ovirtbot> Current chairs: Rydekull dcaro eedri eedri_ ewoud obasan
14:14:46 <ewoud> obasan: the goal is to move everything from linode01 to other machines
14:14:46 <dcaro> Rydekull: I see that the whole hard drive for gerrit is 28Gb...
14:15:23 <ewoud> so what do you guys think about moving backups to Rydekulls space?
14:15:35 <obasan> ewoud, do you think that until everything is migrated we should monitor it?
14:15:56 <eedri> as for backups
14:15:57 <ewoud> obasan: yes, because we've had trouble with it filling up and the mailing lists are still on it
14:16:06 <ewoud> so users can experience problems because of it
14:16:10 <eedri> i suggest to 1st remove all f17 rpms from linode + stop building it on jenkins
14:16:19 <obasan> ewoud, ok
14:16:42 <eedri> 2nd - i can try and request external storage from rackspace
14:16:57 <ewoud> +1 to removing f17 RPMs
14:17:18 <ewoud> Rydekull: I'm wondering how secure your backup space is
14:17:40 <eedri> 3nd i sent to review the cleanup scripts on linode a while ago, i think there is something wrong there and we're keeping stuff we dont need
14:18:04 <ewoud> eedri: I think I made some inline comments that were not yet adressed
14:18:05 <eedri> ewoud, Rydekull i think it's worth trying to get hosted service for backups with high availability
14:18:37 <eedri> ewoud, could be, it's worth picking it up again
14:18:41 <Rydekull> ewoud: Define secure, or 'security'. Its redundant storage and only reachable via ssh. So in either term, i'd say its secure
14:19:27 <ewoud> ok, small vote on moving backups to Rydekull?
14:19:28 <Rydekull> Im all for 'A place to store backups'
14:19:31 <dcaro> Rydekull: 527MB of the 781 MB that the review_site dir occupies in gerrit are logs
14:19:35 <ewoud> +1 from me
14:20:24 <Rydekull> dcaro: I'd say logs are important to backup aswell. Depending on what they contains, naturally
14:20:28 <dcaro> Rydekull: and onlu 2mb are the logs for today xd
14:20:29 <eedri> ewoud, i propose raising it on the list
14:20:39 <Rydekull> eedri: I thought I did :-)
14:20:59 <eedri> Rydekull, i must have missed it then :/, i was on PTO the whole last week
14:21:06 <ewoud> eedri: you even responded to it I think
14:21:06 <Rydekull> as for removing f17 RPMs, I dont think it should be done without http://fedoraproject.org/wiki/End_of_life
14:21:06 <dcaro> Rydekull: we are not deleting old logs, and we are inluding them on the backup each time
14:21:09 <ewoud> http://lists.ovirt.org/pipermail/infra/2013-July/003633.html
14:21:17 <eedri> ewoud, i responded to the fact that we need backups
14:21:24 <eedri> ewoud, didn't see a vote i think
14:21:30 <ewoud> eedri: ah like that
14:21:42 <ewoud> ok, let's promote the thread to a vote then :)
14:22:15 <eedri> ewoud, do you want me to try and get hosted service via rackspace 1st?
14:22:21 <Rydekull> We might decrease how often we build them, like removing nightly, but still doing a weekly build or something
14:22:28 <Rydekull> but removing f17 before it going EoL seems bad
14:22:30 <Rydekull> to me
14:22:39 <ewoud> Rydekull: I think f17 is already EoL
14:22:44 <ewoud> since F19 is out
14:22:59 <eedri> Rydekull, you think that anyone uses it?
14:23:11 <ewoud> eedri: will non RH-employees have access to the hosted service via rackspace?
14:23:14 * eedri thinks we should really add download statistics on ovirt rpms
14:23:23 <ewoud> eedri: +1
14:23:34 <eedri> ewoud, we can ask to open anything we want
14:23:38 <eedri> ewoud, it's not hosted on RH
14:23:52 <eedri> ewoud, i think rackspace servers are open to all infra memebers currently
14:23:58 <ewoud> eedri: afaik that was an issue with the physical machines
14:24:17 <eedri> dcaro, you added all public keys to rackspace servers right?
14:24:34 <eedri> ewoud, i think we just added 'jenkins.ovirt.org as a gating server
14:24:36 <ewoud> eedri: oh yes, through SSH you can if you use alterway as jump host, right?
14:24:42 <eedri> ewoud, otherwise you need vpn access
14:24:43 <dcaro> eedri: yep, and now are bing managed by puppet, so they should be already updated
14:24:56 <Rydekull> eedri: what if alterway is the nodes that are down then? :-)
14:25:01 <eedri> ewoud, exactly
14:25:11 <eedri> Rydekull, then we can use direct access via VPN
14:25:18 <Rydekull> eedri: RH-guys, right
14:25:27 <eedri> Rydekull, no nessasarily
14:25:31 <Rydekull> Anyway, that really needs to be documented
14:25:35 <eedri> Rydekull, if it is needed, other people can be added
14:25:43 <Rydekull> And, my time is running short here
14:25:54 <ewoud> yes, shall we move this backups discussion to the ML?
14:25:59 <Rydekull> Yes
14:26:00 <eedri> ewoud, +1
14:26:09 <ewoud> #info discussion about the backups should move to the ML
14:26:19 <dcaro> ewoud: +1
14:26:26 <eedri> ewoud, i think at imeediate action we should stop building nigthlies for f17
14:26:26 <ewoud> obasan: would you mind adding monitoring for disk space to all hosts?
14:26:36 <eedri> ewoud, i don't mind keep stable versions for f17
14:27:07 <eedri> ewoud, surely not keeping 3 nigthlies back
14:27:07 <ewoud> already +1'ed http://lists.ovirt.org/pipermail/infra/2013-July/003654.html
14:27:08 <obasan> ewoud, I can do that
14:27:18 <ewoud> #action obasan add disk space monitoring to all hosts
14:27:21 <ewoud> obasan: cool
14:27:32 <eedri> ewoud, you think we should give it time or just go a head and remove it?
14:27:41 <Rydekull> Hrm
14:27:42 <Rydekull> Fedora 18 will be maintained until 1 month after the release of Fedora 20.
14:27:42 <Rydekull> Fedora 19 will be maintained until 1 month after the release of Fedora 21
14:27:53 <Rydekull> so, f17 is passed one month after the release of f19
14:27:54 <eedri> ewoud, i think we already got approval in ovirt meeting
14:27:56 <Rydekull> so we should be safe
14:28:04 <ewoud> eedri: running nightlies on an EoL platform sounds … contradictory
14:28:06 <eedri> Rydekull, and we're talking about niglies
14:28:11 <eedri> Rydekull, not the stable releases...
14:28:21 <ewoud> eedri: so imho just remove nightlies for F17
14:28:25 <eedri> ewoud, exactly
14:28:40 <Rydekull> eedri: yeah, like I said, nightlies are fine to remove for me, just think a bit more about the other stuff :-)
14:28:41 <eedri> #action eedri to remove nighlies for f17 on jenkins.ovirt.org
14:28:48 <Rydekull> we should have an archive either way of older copies imho
14:29:09 <eedri> Rydekull, for nigthlies ?
14:29:11 <Rydekull> anyway, I bid you guys a good eve. Im heading out
14:29:13 <Rydekull> eedri: no
14:29:26 <eedri> Rydekull, for official releases, i agree
14:29:46 <ewoud> #agreed no more nightlies for F17
14:30:16 <ewoud> Rydekull: thanks for your time and I'll ping you when I know more about a small vacation to stockholm
14:30:23 <Rydekull> :-)
14:30:25 <Rydekull> &
14:30:32 <ewoud> any other items on hosting?
14:30:50 <dcaro> Rydekull: bye!
14:31:58 <ewoud> there's another agenda item I'd like to add and before we forget it, I'm adding it now
14:32:03 <ewoud> #topic organisation
14:32:19 <ewoud> we had a discussion about quaids mail
14:32:47 <ewoud> the one about the project coordiatinator
14:33:08 <dcaro> I though kiril was the one that got that role
14:33:38 <ewoud> and another from dneary about openstack infra bootcamp that got less attention
14:35:32 <ewoud> as you noticed earlier, we still haven't drawn a good map of what we want to do, how we're doing it now and what we'd like to change about that
14:36:18 <ewoud> http://lists.ovirt.org/pipermail/infra/2013-July/003450.html is the mail in question
14:36:34 <eedri> ewoud, how about we'll try to do a hangout meeting with video chat?
14:36:57 <eedri> ewoud, and try to map all the issues that's missing
14:37:12 <obasan> eedri, +1
14:37:30 <ewoud> dcaro: I don't have experience using those, but it may be a solution
14:37:37 <ewoud> euh, eedri that is
14:37:44 <eedri> ewoud, it's quite simple.. doing with google hangout
14:37:54 <eedri> ewoud, can be either with voice/video
14:38:06 <ewoud> eedri: I just said I had no experience with it, so I don't know how effective something like that is
14:38:23 <dcaro> ewoud: but you'll need a google account for that
14:38:42 <eedri> ewoud, it gives you opertunity to share documentes while in the meeting
14:38:57 <eedri> ewoud, and maybe share diagrams ..
14:39:24 <eedri> ewoud, but not mandatory... we can do it on a pad as well
14:39:32 <eedri> ewoud, just write everything down
14:39:46 <eedri> ewoud, i think we had that pad in the past, we can just update it
14:39:54 <eedri> ewoud, with the existing services
14:39:56 <ewoud> would it be wise to plan this somewhere when Kiril is back?
14:40:07 <ewoud> eedri: yes, we could use that as a basis for what we have now
14:40:09 <eedri> yes
14:40:20 <eedri> ewoud, i would wait for kiril to come back
14:41:00 <ewoud> eedri: I think that's a good idea because I think IRC is a bit too asynchronous to effectively plan bigger things such as the architecture
14:41:25 <dcaro> ewoud: agree
14:41:34 <eedri> ewoud, exactly
14:42:10 <ewoud> #info a google hangout is suggested as a more synchronous tool to plan our infrastructure
14:42:30 <ewoud> I'll suggest it in the thread
14:42:43 <ewoud> #action ewoud suggest a google hangout session on the ML
14:42:53 <ewoud> ok, on to puppet
14:42:57 <ewoud> #topic puppet
14:43:24 <ewoud> we still have a discussion open on how we want to use modules
14:43:47 <ewoud> it's mostly in http://gerrit.ovirt.org/16907
14:45:59 <ewoud> dcaro: any opinions?
14:46:08 <ewoud> or others? questions?
14:46:54 * obasan looking
14:47:01 <dcaro> ewoud: never used git submodules, but I have been advised against them a few times (by people that used them, maybe the wrong way though)
14:47:45 <obasan> eedri, the ntp module seems pretty flexible and well written
14:48:01 <ewoud> dcaro: they may not be optimal (as I said), but git subtree isn't that easy either
14:48:35 <obasan> ewoud, what about this puppet librarian?
14:48:38 <ewoud> that's why a Puppetfile based solution such as r10k may be nicer, but that's not integrated in our deployment
14:48:49 <ewoud> so that would require some work
14:49:08 <eedri> obasan, i'm fine with either ntp module, seems pretty basic need
14:49:42 <ewoud> obasan: I've seen more from r10k than puppet-librarian, but I think they're pretty similar
14:51:03 <obasan> ewoud, I have no experience with neither :(
14:51:13 <obasan> ewoud, and nor with git submodules.
14:52:31 <ewoud> so do we want to continue the discussion in gerrit? ML? vote now?
14:53:43 <dcaro> mmm, librarian has a lot more followers and commiters but a lot less activity (committs last year)
14:54:23 <obasan> dcaro, I would always go with the bigger community
14:54:35 <obasan> dcaro, activity is top priority
14:57:18 <dcaro> xd, librarian is made by a github staff member, r10k by a puyppetlabs programmer
14:59:02 <obasan> dcaro, tough choices.
15:00:22 <dcaro> obasan: I also see that the developer under r10k has been much more active in github (not only r10k) than the librarian one
15:00:52 <obasan> dcaro, do r10k it is?
15:01:16 <ewoud> I'm leaning to git submodules in the very short term (because the deployment is already there) while we investigate other tools
15:01:26 <dcaro> obasan: rephrase please?
15:01:47 <ewoud> btw, there's also a fork of librarian-puppet at https://rubygems.org/gems/librarian-puppet-maestrodev
15:02:41 <dcaro> ewoud: they were starting to use maestrodev in my last company, did not have the chance to try it though
15:11:39 <dcaro> I think we should take it to the ml (I'd go for any of them, both use puppetfiles so it should be easy to switch between them in any case)
15:15:30 <ewoud> dcaro: ok
15:15:55 <ewoud> I think we can close the meeting now
15:16:52 <ewoud> #endmeeting