14:00:50 #startmeeting infra weekly meeting 14:00:50 Meeting started Mon Apr 22 14:00:50 2013 UTC. The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:50 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:01:02 #chair Rydekull dcaro dneary eedri_ quaid 14:01:02 Current chairs: Rydekull dcaro dneary eedri_ ewoud quaid 14:01:10 * eedri_ here 14:01:19 ewoud: yeah, I saw, it was excellent, however, I need to run now 14:01:23 * Rydekull reads the backlog later 14:01:47 Rydekull: ok 14:02:00 Introductions 14:02:00 Review of action items 14:02:00 Rydekull add the jenkins slaves to a list on the wiki 14:02:00 quaid to setup docs sprint 14:02:00 eedri to send email about ip space on rackspace servers 14:02:02 ewoud mail to infra@ to make sure we don't have a SPOF on people with access 14:02:06 ewoud restart the puppet ML thread 14:02:08 Hosting 14:02:11 rackspace01 installed? 14:02:13 Puppet 14:02:16 Jenkins 14:02:18 Other business? 14:02:21 Trac review 14:02:23 other items people would like to add? 14:03:16 #topic introductions 14:03:30 do we have any new people? 14:04:47 #topic Review of action items 14:05:16 #info Rydekull is working on a list, but it's still an offline list 14:05:26 I think quaid is on vacation 14:05:41 eedri_: did you get any info from rackspace about the IP space? 14:05:43 * quaid is back 14:05:52 ewoud, as for jenkins slaves list, it's temporarily till we'll have ovirt-engine running on rackspace 14:06:05 ewoud, no, should i? 14:06:10 ewoud, don't you mean alterway? 14:06:35 14[[07Features/Design/DetailedExternalTasks14]]4 !10 02http://www.ovirt.org/index.php?diff=8439&oldid=8434&rcid=8651 5* 03Emesika 5* (+150) 10/* Post */  14:06:47 eedri_: during our last meeting we talked about IP space for alterway, but we were also wondering about rackspace 14:06:56 since we want to virtualize there too 14:07:05 ewoud, i had the feeling we're waiting for the server to be ready 1st 14:07:30 ewoud, maybe quaid can shed some light on it 14:07:49 we did the same thing with alterway and now we're waiting for that, so I'd like to get the info asap 14:08:13 ewoud, who is our contacts for rackspace? 14:08:22 ewoud, do we have wiki on it? emails? 14:08:30 maybe dneary knows? 14:08:46 or someone else at RH IT who quaid had contact with 14:09:26 I'm here 14:09:35 catching up 14:12:12 #info ewoud started http://lists.ovirt.org/pipermail/infra/2013-April/002625.html to make a list of services and SPOFs on people 14:12:43 let me switch consoles 14:13:17 going to keep open action items in the minutes 14:13:20 #action ewoud restart the puppet ML thread 14:13:29 #action Rydekull add the jenkins slaves on a list on the wiki 14:14:27 sorry - was on a call that went over 14:14:57 re RAX, I'm afraid I do not have the information 14:15:00 no problem 14:16:06 * quaid on a real keyboard now 14:16:34 ewoud: do you want to tackle the rackspace topic, or is this an action item review & I should save it for a new topic? 14:17:08 quaid: right now it was mostly an action item, we'll go in detail on the hosting topic 14:17:18 quaid: but any time to work on the docs sprint? 14:17:47 ewoud: yes, esp. if I can hand off the rackspace problem 14:17:59 quaid: and do you by any chance know about the IP space at rackspace? 14:18:24 ewoud: not yet 14:18:44 we need public IPs, yes? 14:19:06 not sure for jenkins slaves, but I'd like to know the details anyway 14:19:31 I'll just keep the docs sprint on it the action items and move on 14:19:39 #action quaid work on a docs sprint 14:19:42 #topic hosting 14:19:49 right, rackspace 14:20:18 eedri_: I don't think we need public IPs for jenkins slaves or do we? 14:21:08 ewoud, i think we might 14:21:13 I think I should hand off the rackspace install situation, if someone such as eedri_ or theron can get involved 14:21:22 ewoud, since jenkins master need to communicate with them 14:21:44 #info may need public IPs for RackSpace 14:22:25 anyway i think getting public ips from rackspace might be easier then alterway 14:22:32 quaid: if you can at least spread the credentials to someone else we don't have to rely on you 14:22:34 #info quaid looking to handoff installation as he has run out of time blocks 14:22:35 since they have hosting infra already 14:22:52 quaid, i'm sure that between me & david we can handle the installation 14:23:16 ewoud: I think it might be currently limited to @redhat.com for something, but I'm not sure - the whole vpn + ssh + iDRAC thing still confuses me a bit about who can be on the aCL 14:23:48 quaid, since me & dcaro are from @redhat, shouldn't be an issue i suppose 14:23:59 eedri_: ok, good - I think it's just a time and persistence thing, in that I can't get the terminal console to work for me, but perhaps if you use a RHEL 6 based Firefox or another OS with a Java that works better for the iDRAC 14:24:01 quaid: I don't mind not getting access, I just don't want to rely on one person 14:24:07 ewoud: +1 14:24:18 eedri_: ok, I'll work on adding you and dcaro so you can takeover the installation 14:24:27 quaid, +1 and thanks 14:24:29 quaid: hence my email from this morning 14:24:43 +1 14:24:44 ewoud: didn't see it yet :) 14:25:08 #action quaid to bring dcaro & eedri_ in to the ACLs for RackSpace so they can take it over 14:25:28 yeah, my project is really heated up & taking all my time through to the summer 14:26:35 ok, let's see if we can get rackspace hosts up to speed fast since it's becoming a blocker 14:27:05 about alterway IP space 14:27:06 mhh 14:27:25 is there a meeting atm ? 14:27:29 YamaKasY1: yes 14:27:37 ewoud: okay thanks 14:27:45 I pinged about it and I think we need at least 3 public IPs at rackspace for VMs 14:28:25 http://lists.ovirt.org/pipermail/infra/2013-April/002622.html 14:28:28 did I miss anything? 14:29:33 ewoud, i think we have more servers 14:29:48 ewoud, i thought you were asking only on linode server 14:29:55 ewoud, what about gerrit? 14:29:57 quaid, Do you think we could organise a hand-over during the next week or two to relieve you of being SPOF on some of this stuff? 14:30:10 eedri_: good point 14:30:15 ewoud, foreman/puppet 14:30:21 eedri_: I mentioned that 14:30:27 ewoud, we have a pad with a list of servers 14:30:55 * eedri_ can't remember the url.. 14:31:03 * ewoud is searching in his logs 14:32:36 dneary: yes, I think that's what I just agreed to :) 14:32:56 quaid, My apologies, it appears I am behind the times 14:33:03 http://etherpad.ovirt.org/p/new_hosting_design_Jan_2013 is it I think 14:33:09 (also doing ~6 things at the same time, which doesn't help) 14:33:34 * quaid just sent the email asking for ACL for eedri_ & dcaro 14:34:19 eedri_: I see an artifactory.ovirt.org but I think that could be part of resources.ovirt.org 14:35:13 not sure about backup.ovirt.org; ideally that would not be on alterway02 since then we can't use it to backup gerrit which is on the same host 14:37:02 ewoud, yea, artifactory is nice to have, so far i'm not seeing any failures on maven repos 14:37:08 ewoud, but it can be on resources for sure 14:38:48 sorry, some connection issue 14:39:05 redirects to ovirt.org could also be on resources 14:39:30 eedri_: so all in all we need 4 IPs for VMs 14:40:05 ewoud, and if we're missing, we can abuse resources.ovirt.org with proxy redirect 14:40:20 eedri_: only if we have to 14:40:55 #info we need 4 public IPs at alterway: gerrit, resources, lists and foreman 14:41:13 anything else on hosting? 14:42:14 ewoud, do we need other infra? 14:42:20 ewoud, other than ips... ? 14:42:32 ewoud, storage servers i assume we'll use local 14:42:40 eedri_: I think so too 14:42:40 ewoud, or use one server as nfs server 14:42:58 ewoud, dns/dhcp? 14:43:04 ewoud, what about dhcp ? 14:43:21 eedri_: I think that should be part of the foreman install since that's also a smartproxy 14:43:42 ewoud, and that dhcp will be able to manage the public ips? 14:43:59 eedri_: yes, we do that at $dayjob with foreman as well 14:44:04 ewoud, ok 14:44:23 * eedri_ doesn't manage dhcp with foreman since it's not allowed in $company 14:45:09 we have manage a total 3 /24 blocks of public IP space, last one is split into multiple smaller blocks 14:46:13 btw, I was thinking about a mirror system for resources.ovirt.org 14:46:48 anyone has experience setting that up? 14:48:06 or if there's even need for that? 14:48:18 ewoud, jenkins job that runs rsync? 14:48:38 ewoud, but there might be better tools for that 14:49:38 eedri_: maybe there's no need for it, but I thought I'd bring it up 14:49:54 dcaro: any idea? 14:50:26 maybe we should first install some monitoring / trending 14:50:58 I have good experiences with munin to monitor the load 14:51:21 ewoud: some simple monitoring will give us an idea if we need mirroring for load 14:51:23 maybe that should be a separate machine on alterway 14:52:14 we have a nice setup at $dayjob that uses puppet exported resources to auto build the list of hosts 14:53:12 #info a monitoring VM would be nice too 14:54:29 ewoud, there is gangalia project we can use 14:55:36 eedri_: I have no experience with it 14:55:48 but munin does not offer notifications right? 14:56:09 it's just performance analisys? 14:56:28 yes it can :) 14:57:20 http://munin-monitoring.org/wiki/HowToContact 14:57:41 (misleading url though) 14:59:07 one alternative is zabbix, but I like munin because it's very easy to set up fast 14:59:31 so we could start with it and later look at a more permanent solution 15:00:17 well.. you have monit - nagios/icinga - zabbix - munin - graphite - ganglia (and some more) 15:00:31 from monitoring/alerts -> performance 15:00:47 yes, and all have their upsides and downsides 15:01:23 a colleague is looking into something better for us at $dayjob, but in the mean time nagios + munin isn't that bad 15:01:44 what do you use? 15:02:23 right now we are using nagios+ganglia and a little of graphite (has way better ui for custom graphs) 15:03:10 how easy it that to get going? 15:03:20 graphite? 15:03:34 ganglia? 15:03:34 anything 15:04:05 I'd like to get at least trending going and I know that with munin I can set that up in 5 minutes for linode01 15:04:29 and I know it scales decently 15:04:38 You will not with any of the others xd (well, monit yes) 15:05:23 I'll raise the subject on the ML 15:05:35 #action ewoud start a thread on monitoring/trending 15:05:53 especially since we had an issue with a full disk too often 15:06:27 looking at the time I think we can close this meeting 15:06:53 ok, let's continue on the ml 15:07:24 going once 15:07:36 going twice 15:07:45 #endmeeting