14:00:50 <ewoud> #startmeeting infra weekly meeting
14:00:50 <ovirtbot> Meeting started Mon Apr 22 14:00:50 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:50 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:01:02 <ewoud> #chair Rydekull dcaro dneary eedri_ quaid
14:01:02 <ovirtbot> Current chairs: Rydekull dcaro dneary eedri_ ewoud quaid
14:01:10 * eedri_ here
14:01:19 <Rydekull> ewoud: yeah, I saw, it was excellent, however, I need to run now
14:01:23 * Rydekull reads the backlog later
14:01:47 <ewoud> Rydekull: ok
14:02:00 <ewoud> Introductions
14:02:00 <ewoud> Review of action items
14:02:00 <ewoud> Rydekull add the jenkins slaves to a list on the wiki
14:02:00 <ewoud> quaid to setup docs sprint
14:02:00 <ewoud> eedri to send email about ip space on rackspace servers
14:02:02 <ewoud> ewoud mail to infra@ to make sure we don't have a SPOF on people with access
14:02:06 <ewoud> ewoud restart the puppet ML thread
14:02:08 <ewoud> Hosting
14:02:11 <ewoud> rackspace01 installed?
14:02:13 <ewoud> Puppet
14:02:16 <ewoud> Jenkins
14:02:18 <ewoud> Other business?
14:02:21 <ewoud> Trac review
14:02:23 <ewoud> other items people would like to add?
14:03:16 <ewoud> #topic introductions
14:03:30 <ewoud> do we have any new people?
14:04:47 <ewoud> #topic Review of action items
14:05:16 <ewoud> #info Rydekull is working on a list, but it's still an offline list
14:05:26 <ewoud> I think quaid is on vacation
14:05:41 <ewoud> eedri_: did you get any info from rackspace about the IP space?
14:05:43 * quaid is back
14:05:52 <eedri_> ewoud, as for jenkins slaves list, it's temporarily till we'll have ovirt-engine running on rackspace
14:06:05 <eedri_> ewoud, no, should i?
14:06:10 <eedri_> ewoud, don't you mean alterway?
14:06:35 <ovirtbot> 14[[07Features/Design/DetailedExternalTasks14]]4 !10 02http://www.ovirt.org/index.php?diff=8439&oldid=8434&rcid=8651 5* 03Emesika 5* (+150) 10/* Post */ 
14:06:47 <ewoud> eedri_: during our last meeting we talked about IP space for alterway, but we were also wondering about rackspace
14:06:56 <ewoud> since we want to virtualize there too
14:07:05 <eedri_> ewoud, i had the feeling we're waiting for the server to be ready 1st
14:07:30 <eedri_> ewoud, maybe quaid can shed some light on it
14:07:49 <ewoud> we did the same thing with alterway and now we're waiting for that, so I'd like to get the info asap
14:08:13 <eedri_> ewoud, who is our contacts for rackspace?
14:08:22 <eedri_> ewoud, do we have wiki on it? emails?
14:08:30 <ewoud> maybe dneary knows?
14:08:46 <ewoud> or someone else at RH IT who quaid had contact with
14:09:26 <dneary> I'm here
14:09:35 <dneary> catching up
14:12:12 <ewoud> #info ewoud started http://lists.ovirt.org/pipermail/infra/2013-April/002625.html to make a list of services and SPOFs on people
14:12:43 <quaid> let me switch consoles
14:13:17 <ewoud> going to keep open action items in the minutes
14:13:20 <ewoud> #action ewoud restart the puppet ML thread
14:13:29 <ewoud> #action Rydekull add the jenkins slaves on a list on the wiki
14:14:27 <dneary> sorry - was on a call that went over
14:14:57 <dneary> re RAX, I'm afraid I do not have the information
14:15:00 <ewoud> no problem
14:16:06 * quaid on a real keyboard now
14:16:34 <quaid> ewoud: do you want to tackle the rackspace topic, or is this an action item review & I should save it for a new topic?
14:17:08 <ewoud> quaid: right now it was mostly an action item, we'll go in detail on the hosting topic
14:17:18 <ewoud> quaid: but any time to work on the docs sprint?
14:17:47 <quaid> ewoud: yes, esp. if I can hand off the rackspace problem
14:17:59 <ewoud> quaid: and do you by any chance know about the IP space at rackspace?
14:18:24 <quaid> ewoud: not yet
14:18:44 <quaid> we need public IPs, yes?
14:19:06 <ewoud> not sure for jenkins slaves, but I'd like to know the details anyway
14:19:31 <ewoud> I'll just keep the docs sprint on it the action items and move on
14:19:39 <ewoud> #action quaid work on a docs sprint
14:19:42 <ewoud> #topic hosting
14:19:49 <ewoud> right, rackspace
14:20:18 <ewoud> eedri_: I don't think we need public IPs for jenkins slaves or do we?
14:21:08 <eedri_> ewoud, i think we might
14:21:13 <quaid> I think I should hand off the rackspace install situation, if someone such as eedri_ or theron can get involved
14:21:22 <eedri_> ewoud, since jenkins master need to communicate with them
14:21:44 <quaid> #info may need public IPs for RackSpace
14:22:25 <eedri_> anyway i think getting public ips from rackspace might be easier then alterway
14:22:32 <ewoud> quaid: if you can at least spread the credentials to someone else we don't have to rely on you
14:22:34 <quaid> #info quaid looking to handoff installation as he has run out of time blocks
14:22:35 <eedri_> since they have hosting infra already
14:22:52 <eedri_> quaid, i'm sure that between me & david we can handle the installation
14:23:16 <quaid> ewoud: I think it might be currently limited to @redhat.com for something, but I'm not sure - the whole vpn + ssh + iDRAC thing still confuses me a bit about who can be on the aCL
14:23:48 <eedri_> quaid, since me & dcaro are from @redhat, shouldn't be an issue i suppose
14:23:59 <quaid> eedri_: ok, good - I think it's just a time and persistence thing, in that I can't get the terminal console to work for me, but perhaps if you use a RHEL 6 based Firefox or another OS with a Java that works better for the iDRAC
14:24:01 <ewoud> quaid: I don't mind not getting access, I just don't want to rely on one person
14:24:07 <quaid> ewoud: +1
14:24:18 <quaid> eedri_: ok, I'll work on adding you and dcaro so you can takeover the installation
14:24:27 <eedri_> quaid, +1 and thanks
14:24:29 <ewoud> quaid: hence my email from this morning
14:24:43 <dcaro> +1
14:24:44 <quaid> ewoud: didn't see it yet :)
14:25:08 <quaid> #action quaid to bring dcaro & eedri_ in to the ACLs for RackSpace so they can take it over
14:25:28 <quaid> yeah, my project is really heated up & taking all my time through  to the summer
14:26:35 <ewoud> ok, let's see if we can get rackspace hosts up to speed fast since it's becoming a blocker
14:27:05 <ewoud> about alterway IP space
14:27:06 <YamaKasY1> mhh
14:27:25 <YamaKasY1> is there a meeting atm ?
14:27:29 <ewoud> YamaKasY1: yes
14:27:37 <YamaKasY1> ewoud: okay thanks
14:27:45 <ewoud> I pinged about it and I think we need at least 3 public IPs at rackspace for VMs
14:28:25 <ewoud> http://lists.ovirt.org/pipermail/infra/2013-April/002622.html
14:28:28 <ewoud> did I miss anything?
14:29:33 <eedri_> ewoud, i think we have more servers
14:29:48 <eedri_> ewoud, i thought you were asking only on linode server
14:29:55 <eedri_> ewoud, what about gerrit?
14:29:57 <dneary> quaid, Do you think we could organise a hand-over during the next week or two to relieve you of being SPOF on some of this stuff?
14:30:10 <ewoud> eedri_: good point
14:30:15 <eedri_> ewoud, foreman/puppet
14:30:21 <ewoud> eedri_: I mentioned that
14:30:27 <eedri_> ewoud, we have a pad with a list of servers
14:30:55 * eedri_ can't remember the url..
14:31:03 * ewoud is searching in his logs
14:32:36 <quaid> dneary: yes, I think that's what I just agreed to :)
14:32:56 <dneary> quaid, My apologies, it appears I am behind the times
14:33:03 <ewoud> http://etherpad.ovirt.org/p/new_hosting_design_Jan_2013 is it I think
14:33:09 <dneary> (also doing ~6 things at the same time, which doesn't help)
14:33:34 * quaid just sent the email asking for ACL for eedri_ & dcaro
14:34:19 <ewoud> eedri_: I see an artifactory.ovirt.org but I think that could be part of resources.ovirt.org
14:35:13 <ewoud> not sure about backup.ovirt.org; ideally that would not be on alterway02 since then we can't use it to backup gerrit which is on the same host
14:37:02 <eedri_> ewoud, yea, artifactory is nice to have, so far i'm not seeing any failures on maven repos
14:37:08 <eedri_> ewoud, but it can be on resources for sure
14:38:48 <ewoud> sorry, some connection issue
14:39:05 <ewoud> redirects to ovirt.org could also be on resources
14:39:30 <ewoud> eedri_: so all in all we need 4 IPs for VMs
14:40:05 <eedri_> ewoud, and if we're missing, we can abuse resources.ovirt.org with proxy redirect
14:40:20 <ewoud> eedri_: only if we have to
14:40:55 <ewoud> #info we need 4 public IPs at alterway: gerrit, resources, lists and foreman
14:41:13 <ewoud> anything else on hosting?
14:42:14 <eedri_> ewoud, do we need other infra?
14:42:20 <eedri_> ewoud, other than ips... ?
14:42:32 <eedri_> ewoud, storage servers i assume we'll use local
14:42:40 <ewoud> eedri_: I think so too
14:42:40 <eedri_> ewoud, or use one server as nfs server
14:42:58 <eedri_> ewoud, dns/dhcp?
14:43:04 <eedri_> ewoud, what about dhcp ?
14:43:21 <ewoud> eedri_: I think that should be part of the foreman install since that's also a smartproxy
14:43:42 <eedri_> ewoud, and that dhcp will be able to manage the public ips?
14:43:59 <ewoud> eedri_: yes, we do that at $dayjob with foreman as well
14:44:04 <eedri_> ewoud, ok
14:44:23 * eedri_ doesn't manage dhcp with foreman since it's not allowed in $company
14:45:09 <ewoud> we have manage a total 3 /24 blocks of public IP space, last one is split into multiple smaller blocks
14:46:13 <ewoud> btw, I was thinking about a mirror system for resources.ovirt.org
14:46:48 <ewoud> anyone has experience setting that up?
14:48:06 <ewoud> or if there's even need for that?
14:48:18 <eedri_> ewoud, jenkins job that runs rsync?
14:48:38 <eedri_> ewoud, but there might be better tools for that
14:49:38 <ewoud> eedri_: maybe there's no need for it, but I thought I'd bring it up
14:49:54 <ewoud> dcaro: any idea?
14:50:26 <ewoud> maybe we should first install some monitoring / trending
14:50:58 <ewoud> I have good experiences with munin to monitor the load
14:51:21 <dcaro> ewoud: some simple monitoring will give us an idea if we need mirroring for load
14:51:23 <ewoud> maybe that should be a separate machine on alterway
14:52:14 <ewoud> we have a nice setup at $dayjob that uses puppet exported resources to auto build the list of hosts
14:53:12 <ewoud> #info a monitoring VM would be nice too
14:54:29 <eedri_> ewoud, there is gangalia project we can use
14:55:36 <ewoud> eedri_: I have no experience with it
14:55:48 <dcaro> but munin does not offer notifications right?
14:56:09 <dcaro> it's just performance analisys?
14:56:28 <dcaro> yes it can :)
14:57:20 <dcaro> http://munin-monitoring.org/wiki/HowToContact
14:57:41 <dcaro> (misleading url though)
14:59:07 <ewoud> one alternative is zabbix, but I like munin because it's very easy to set up fast
14:59:31 <ewoud> so we could start with it and later look at a more permanent solution
15:00:17 <dcaro> well.. you have monit - nagios/icinga - zabbix - munin - graphite - ganglia (and some more)
15:00:31 <dcaro> from monitoring/alerts -> performance
15:00:47 <ewoud> yes, and all have their upsides and downsides
15:01:23 <ewoud> a colleague is looking into something better for us at $dayjob, but in the mean time nagios + munin isn't that bad
15:01:44 <ewoud> what do you use?
15:02:23 <dcaro> right now we are using nagios+ganglia and a little of graphite (has way better ui for custom graphs)
15:03:10 <ewoud> how easy it that to get going?
15:03:20 <dcaro> graphite?
15:03:34 <dcaro> ganglia?
15:03:34 <ewoud> anything
15:04:05 <ewoud> I'd like to get at least trending going and I know that with munin I can set that up in 5 minutes for linode01
15:04:29 <ewoud> and I know it scales decently
15:04:38 <dcaro> You will not with any of the others xd (well, monit yes)
15:05:23 <ewoud> I'll raise the subject on the ML
15:05:35 <ewoud> #action ewoud start a thread on monitoring/trending
15:05:53 <ewoud> especially since we had an issue with a full disk too often
15:06:27 <ewoud> looking at the time I think we can close this meeting
15:06:53 <dcaro> ok, let's continue on the ml
15:07:24 <ewoud> going once
15:07:36 <ewoud> going twice
15:07:45 <ewoud> #endmeeting