14:02:21 #startmeeting infra weekly 14:02:22 Meeting started Mon Jul 22 14:02:21 2013 UTC. The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:02:22 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:02:24 leon80: it is feature-wise. Mostly. This console connectivity is an exception. 14:02:36 #chair Rydekull dcaro eedri_ 14:02:36 Current chairs: Rydekull dcaro eedri_ ewoud 14:02:54 leon80: and then couple of hundred bugfixes more:) 14:03:29 so Kiril is away for 2 weeks 14:03:44 copy paste from http://www.ovirt.org/Infrastructure_team_meetings#2013-07-22 14:03:48 Introductions 14:03:48 Review of action items 14:03:48 Hosting 14:03:48 Puppet 14:03:48 Jenkins 14:03:51 Other business? 14:03:53 Trac review 14:04:21 I don't think we have new (potential) infra members, so I'd like to skip introductions this week 14:04:56 looking at the minutes from http://resources.ovirt.org/meetings/ovirt/2013/ovirt.2013-07-08-13.59.html I only see one action item 14:04:57 mskrivanek: Thank you for the great explanation! :) 14:05:01 knesenko add linkedin profile link to the main page ovirt.org 14:05:38 * obasan here 14:05:43 #chair obasan 14:05:43 Current chairs: Rydekull dcaro eedri_ ewoud obasan 14:05:51 you didn't miss much, just the agenda 14:05:52 ewoud, eedri will be joining in a few minutes 14:06:04 ewoud, knesenko is away for 2-3 weeks :( 14:06:20 obasan: I know, he told me 14:06:51 #topic review of action items 14:06:58 knesenko add linkedin profile link to the main page ovirt.org 14:07:07 obasan: do you know if that was done? 14:07:22 ewoud, I don't know. let's see 14:07:24 I do see the link in the about, but not sure that's it 14:07:24 doesnt look like it 14:07:30 ewoud, was not done 14:08:06 ok 14:08:11 ewoud, what else is on the agenda? 14:08:21 http://www.ovirt.org/Infrastructure_team_meetings#2013-07-22 14:08:21 ewoud, I finished the openshift quota monitoring. 14:08:21 obasan: the usual I think 14:08:24 same agenda 14:08:39 I'm leaving the action item then 14:08:46 #action knesenko add linkedin profile link to the main page ovirt.org 14:08:49 #topic hosting 14:09:03 #info obasan Added monitoring to the openshift quota 14:09:51 obasan: I think we should also add disk monitoring and set up some system so more people monitor the messages 14:10:12 Backup, can we move it from linode01? 14:10:34 Rydekull: but where? 14:10:50 ewoud, I thought about adding the alerts to the infra mailing list 14:10:53 ewoud, what do you think? 14:10:55 Like I offered, I can create a offsite-location for it reachable via sftp/ssh 14:11:07 ewoud, which disks would you like to monitor? 14:11:23 obasan: all of them? :) 14:11:37 Rydekull: that's something I meant to reply to 14:11:39 Do we really need that much ammount of backups? 14:11:45 ewoud, the jenkins slaves? :) 14:11:55 Rydekull: I think it can be good to have an offsite location 14:11:58 dcaro: today it builds up to roughly 30 GB 14:12:06 ewoud, that can be done once all the slaves are up 14:12:08 dcaro: and I dont think we back everything up, honestly 14:12:31 obasan: oh, I mostly meant everything from linode but I guess we could monitor all 14:12:31 Rydekull: What does it include (and what do you think is missing) 14:12:44 ewoud, what exactly is this linode? 14:12:53 obasan: a machine, called linode01 14:12:59 obasan: hosted on linode 14:13:04 Rydekull, ewoud no problem monitoring it 14:13:06 obasan: www.linode.com 14:13:07 obasan: it's a VM hosted at linode and initially everything was hosted there 14:13:20 dcaro: Mostly gerrit, and some db-stuff 14:13:27 obasan: so website, mailing list, resources (RPMs) 14:13:39 gerrit is on another VM 14:13:45 dcaro: and I feel that there probably is some new services, jenkins etc that atleast want configuration backed up 14:13:49 the website was moved to openshift 14:14:14 dcaro: to be able to rebuild it quickly and easy if need be 14:14:21 * eedri here 14:14:30 #chair eedri 14:14:30 Current chairs: Rydekull dcaro eedri eedri_ ewoud obasan 14:14:46 obasan: the goal is to move everything from linode01 to other machines 14:14:46 Rydekull: I see that the whole hard drive for gerrit is 28Gb... 14:15:23 so what do you guys think about moving backups to Rydekulls space? 14:15:35 ewoud, do you think that until everything is migrated we should monitor it? 14:15:56 as for backups 14:15:57 obasan: yes, because we've had trouble with it filling up and the mailing lists are still on it 14:16:06 so users can experience problems because of it 14:16:10 i suggest to 1st remove all f17 rpms from linode + stop building it on jenkins 14:16:19 ewoud, ok 14:16:42 2nd - i can try and request external storage from rackspace 14:16:57 +1 to removing f17 RPMs 14:17:18 Rydekull: I'm wondering how secure your backup space is 14:17:40 3nd i sent to review the cleanup scripts on linode a while ago, i think there is something wrong there and we're keeping stuff we dont need 14:18:04 eedri: I think I made some inline comments that were not yet adressed 14:18:05 ewoud, Rydekull i think it's worth trying to get hosted service for backups with high availability 14:18:37 ewoud, could be, it's worth picking it up again 14:18:41 ewoud: Define secure, or 'security'. Its redundant storage and only reachable via ssh. So in either term, i'd say its secure 14:19:27 ok, small vote on moving backups to Rydekull? 14:19:28 Im all for 'A place to store backups' 14:19:31 Rydekull: 527MB of the 781 MB that the review_site dir occupies in gerrit are logs 14:19:35 +1 from me 14:20:24 dcaro: I'd say logs are important to backup aswell. Depending on what they contains, naturally 14:20:28 Rydekull: and onlu 2mb are the logs for today xd 14:20:29 ewoud, i propose raising it on the list 14:20:39 eedri: I thought I did :-) 14:20:59 Rydekull, i must have missed it then :/, i was on PTO the whole last week 14:21:06 eedri: you even responded to it I think 14:21:06 as for removing f17 RPMs, I dont think it should be done without http://fedoraproject.org/wiki/End_of_life 14:21:06 Rydekull: we are not deleting old logs, and we are inluding them on the backup each time 14:21:09 http://lists.ovirt.org/pipermail/infra/2013-July/003633.html 14:21:17 ewoud, i responded to the fact that we need backups 14:21:24 ewoud, didn't see a vote i think 14:21:30 eedri: ah like that 14:21:42 ok, let's promote the thread to a vote then :) 14:22:15 ewoud, do you want me to try and get hosted service via rackspace 1st? 14:22:21 We might decrease how often we build them, like removing nightly, but still doing a weekly build or something 14:22:28 but removing f17 before it going EoL seems bad 14:22:30 to me 14:22:39 Rydekull: I think f17 is already EoL 14:22:44 since F19 is out 14:22:59 Rydekull, you think that anyone uses it? 14:23:11 eedri: will non RH-employees have access to the hosted service via rackspace? 14:23:14 * eedri thinks we should really add download statistics on ovirt rpms 14:23:23 eedri: +1 14:23:34 ewoud, we can ask to open anything we want 14:23:38 ewoud, it's not hosted on RH 14:23:52 ewoud, i think rackspace servers are open to all infra memebers currently 14:23:58 eedri: afaik that was an issue with the physical machines 14:24:17 dcaro, you added all public keys to rackspace servers right? 14:24:34 ewoud, i think we just added 'jenkins.ovirt.org as a gating server 14:24:36 eedri: oh yes, through SSH you can if you use alterway as jump host, right? 14:24:42 ewoud, otherwise you need vpn access 14:24:43 eedri: yep, and now are bing managed by puppet, so they should be already updated 14:24:56 eedri: what if alterway is the nodes that are down then? :-) 14:25:01 ewoud, exactly 14:25:11 Rydekull, then we can use direct access via VPN 14:25:18 eedri: RH-guys, right 14:25:27 Rydekull, no nessasarily 14:25:31 Anyway, that really needs to be documented 14:25:35 Rydekull, if it is needed, other people can be added 14:25:43 And, my time is running short here 14:25:54 yes, shall we move this backups discussion to the ML? 14:25:59 Yes 14:26:00 ewoud, +1 14:26:09 #info discussion about the backups should move to the ML 14:26:19 ewoud: +1 14:26:26 ewoud, i think at imeediate action we should stop building nigthlies for f17 14:26:26 obasan: would you mind adding monitoring for disk space to all hosts? 14:26:36 ewoud, i don't mind keep stable versions for f17 14:27:07 ewoud, surely not keeping 3 nigthlies back 14:27:07 already +1'ed http://lists.ovirt.org/pipermail/infra/2013-July/003654.html 14:27:08 ewoud, I can do that 14:27:18 #action obasan add disk space monitoring to all hosts 14:27:21 obasan: cool 14:27:32 ewoud, you think we should give it time or just go a head and remove it? 14:27:41 Hrm 14:27:42 Fedora 18 will be maintained until 1 month after the release of Fedora 20. 14:27:42 Fedora 19 will be maintained until 1 month after the release of Fedora 21 14:27:53 so, f17 is passed one month after the release of f19 14:27:54 ewoud, i think we already got approval in ovirt meeting 14:27:56 so we should be safe 14:28:04 eedri: running nightlies on an EoL platform sounds … contradictory 14:28:06 Rydekull, and we're talking about niglies 14:28:11 Rydekull, not the stable releases... 14:28:21 eedri: so imho just remove nightlies for F17 14:28:25 ewoud, exactly 14:28:40 eedri: yeah, like I said, nightlies are fine to remove for me, just think a bit more about the other stuff :-) 14:28:41 #action eedri to remove nighlies for f17 on jenkins.ovirt.org 14:28:48 we should have an archive either way of older copies imho 14:29:09 Rydekull, for nigthlies ? 14:29:11 anyway, I bid you guys a good eve. Im heading out 14:29:13 eedri: no 14:29:26 Rydekull, for official releases, i agree 14:29:46 #agreed no more nightlies for F17 14:30:16 Rydekull: thanks for your time and I'll ping you when I know more about a small vacation to stockholm 14:30:23 :-) 14:30:25 & 14:30:32 any other items on hosting? 14:30:50 Rydekull: bye! 14:31:58 there's another agenda item I'd like to add and before we forget it, I'm adding it now 14:32:03 #topic organisation 14:32:19 we had a discussion about quaids mail 14:32:47 the one about the project coordiatinator 14:33:08 I though kiril was the one that got that role 14:33:38 and another from dneary about openstack infra bootcamp that got less attention 14:35:32 as you noticed earlier, we still haven't drawn a good map of what we want to do, how we're doing it now and what we'd like to change about that 14:36:18 http://lists.ovirt.org/pipermail/infra/2013-July/003450.html is the mail in question 14:36:34 ewoud, how about we'll try to do a hangout meeting with video chat? 14:36:57 ewoud, and try to map all the issues that's missing 14:37:12 eedri, +1 14:37:30 dcaro: I don't have experience using those, but it may be a solution 14:37:37 euh, eedri that is 14:37:44 ewoud, it's quite simple.. doing with google hangout 14:37:54 ewoud, can be either with voice/video 14:38:06 eedri: I just said I had no experience with it, so I don't know how effective something like that is 14:38:23 ewoud: but you'll need a google account for that 14:38:42 ewoud, it gives you opertunity to share documentes while in the meeting 14:38:57 ewoud, and maybe share diagrams .. 14:39:24 ewoud, but not mandatory... we can do it on a pad as well 14:39:32 ewoud, just write everything down 14:39:46 ewoud, i think we had that pad in the past, we can just update it 14:39:54 ewoud, with the existing services 14:39:56 would it be wise to plan this somewhere when Kiril is back? 14:40:07 eedri: yes, we could use that as a basis for what we have now 14:40:09 yes 14:40:20 ewoud, i would wait for kiril to come back 14:41:00 eedri: I think that's a good idea because I think IRC is a bit too asynchronous to effectively plan bigger things such as the architecture 14:41:25 ewoud: agree 14:41:34 ewoud, exactly 14:42:10 #info a google hangout is suggested as a more synchronous tool to plan our infrastructure 14:42:30 I'll suggest it in the thread 14:42:43 #action ewoud suggest a google hangout session on the ML 14:42:53 ok, on to puppet 14:42:57 #topic puppet 14:43:24 we still have a discussion open on how we want to use modules 14:43:47 it's mostly in http://gerrit.ovirt.org/16907 14:45:59 dcaro: any opinions? 14:46:08 or others? questions? 14:46:54 * obasan looking 14:47:01 ewoud: never used git submodules, but I have been advised against them a few times (by people that used them, maybe the wrong way though) 14:47:45 eedri, the ntp module seems pretty flexible and well written 14:48:01 dcaro: they may not be optimal (as I said), but git subtree isn't that easy either 14:48:35 ewoud, what about this puppet librarian? 14:48:38 that's why a Puppetfile based solution such as r10k may be nicer, but that's not integrated in our deployment 14:48:49 so that would require some work 14:49:08 obasan, i'm fine with either ntp module, seems pretty basic need 14:49:42 obasan: I've seen more from r10k than puppet-librarian, but I think they're pretty similar 14:51:03 ewoud, I have no experience with neither :( 14:51:13 ewoud, and nor with git submodules. 14:52:31 so do we want to continue the discussion in gerrit? ML? vote now? 14:53:43 mmm, librarian has a lot more followers and commiters but a lot less activity (committs last year) 14:54:23 dcaro, I would always go with the bigger community 14:54:35 dcaro, activity is top priority 14:57:18 xd, librarian is made by a github staff member, r10k by a puyppetlabs programmer 14:59:02 dcaro, tough choices. 15:00:22 obasan: I also see that the developer under r10k has been much more active in github (not only r10k) than the librarian one 15:00:52 dcaro, do r10k it is? 15:01:16 I'm leaning to git submodules in the very short term (because the deployment is already there) while we investigate other tools 15:01:26 obasan: rephrase please? 15:01:47 btw, there's also a fork of librarian-puppet at https://rubygems.org/gems/librarian-puppet-maestrodev 15:02:41 ewoud: they were starting to use maestrodev in my last company, did not have the chance to try it though 15:11:39 I think we should take it to the ml (I'd go for any of them, both use puppetfiles so it should be easy to switch between them in any case) 15:15:30 dcaro: ok 15:15:55 I think we can close the meeting now 15:16:52 #endmeeting