14:06:43 <ewoud> #startmeeting infra weekly
14:06:43 <ovirtbot> Meeting started Mon May 27 14:06:43 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:06:43 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:06:52 <ewoud> #chair obasan eedri
14:06:52 <ovirtbot> Current chairs: eedri ewoud obasan
14:06:53 * obasan here
14:07:05 <ewoud> #chair dcaro
14:07:05 <ovirtbot> Current chairs: dcaro eedri ewoud obasan
14:07:21 * eedri here, sharing attention with other tasks..
14:07:35 <obasan> eedri, you're a natural born multi tasker
14:07:42 * ewoud is also sort of multitasking
14:07:57 <ewoud> and this is the third week in a row I forgot to update the agenda on the wiki
14:08:55 <dcaro> obasan: don't be fooled, he's only a really good context switcher
14:09:08 <obasan> dcaro, jejejejeje :)
14:09:43 <obasan> anyways, regarding icinga
14:09:52 <obasan> I have set up the service and it's working.
14:09:58 <ewoud> #topic icinga
14:10:08 <ewoud> #info obasan set up icinga on monitoring.ovirt.org
14:10:09 <obasan> in order to connect machines to icinga
14:10:38 <obasan> I need nrpe installed on all machines that should be monitored
14:10:44 <obasan> and all the nagios plugins
14:10:49 <obasan> so I can begin adding services.
14:10:55 <ewoud> does nrpe differ between nagios and icinga?
14:11:03 <obasan> As far as I understood no
14:11:08 <obasan> it's 100% compatible
14:11:18 <obasan> besides icinga and much cooler and has a mobile app and other neat stuff
14:12:07 <ewoud> ok, so we could roll out nrpe clients using puppet?
14:12:30 <obasan> This is the best solution in my opinion. installing nrpe. installing plugins and keeping the nrpe service alive and running
14:14:14 <dcaro> obasan: in the past I've used nfs to provide the custom nagios plugins, but we can use puppet for that
14:14:21 <dcaro> *too
14:14:42 <ewoud> ok
14:16:02 <ewoud> now we mention puppet, I finally took the time to write a proposal on our repository layout
14:16:22 <ewoud> http://lists.ovirt.org/pipermail/infra/2013-May/003132.html
14:16:28 <ewoud> please have a look and reply
14:16:48 <dcaro> ewoud: I must
14:16:53 <dcaro> and I will
14:16:58 <obasan> will do
14:17:33 <ewoud> I don't want a definitive layout, just something to get us started which we can improve over time
14:18:01 <ewoud> #info puppet repository proposal was sent
14:18:30 <ewoud> obasan: how much experience do you have with puppet?
14:19:06 <obasan> ewoud, I am ok with the basics.
14:19:25 <ewoud> obasan: could you have a look at installing nrpe on the clients using puppet?
14:19:36 <obasan> ewoud, no problem
14:19:39 <ewoud> obasan: cool
14:19:42 <obasan> ewoud, a class that does it is simple.
14:19:51 <obasan> ewoud, but we need a puppet master and repo
14:20:02 <ewoud> #action obasan look at installing nrpe using puppet
14:20:11 <ewoud> obasan: we have a master and an empty repo
14:20:20 <ewoud> just one problem: the master is puppet 2.6
14:20:26 <ewoud> and fedora clients have 2.7
14:20:34 <obasan> ewoud, I suggest updating them both to 3.1
14:20:39 <ewoud> obasan: I agree
14:20:51 <obasan> ewoud, are there major syntax changes between 2.7 and 3.1?
14:20:52 <ewoud> though puppetlabs already has 3.2, but that should be no problem
14:21:13 <obasan> ewoud, so 3.2. even better.
14:21:14 <ewoud> obasan: syntax wise not, but the scope has some differences
14:21:24 <obasan> ewoud, I see. ok
14:21:58 <ewoud> at $employer we've migrated partly to 3.x and I must say I like it better than 2.x
14:22:10 <ewoud> and given we have an empty repo now, upgrading should be easy
14:22:25 <ewoud> I will install the puppetlabs repo and update all clients
14:22:37 <ewoud> #action ewoud update puppet to 3.x using puppetlabs repository
14:23:12 <obasan> great :)
14:23:27 <eedri> ewoud, we should take into account we'll have centos/el clients
14:23:39 <eedri> ewoud, and those won't run puppet 3.x until rhel 7 afaikl
14:23:41 <ewoud> eedri: puppetlabs provides EL6 packages
14:23:51 <ewoud> we do it in production and works well
14:24:00 <eedri> ewoud, ok
14:24:35 <ewoud> not sure about EL5, but I don't think we have to care about that
14:24:43 <ewoud> also, it's most important that the master is newer than clients
14:25:24 <ewoud> anything else on icinga?
14:25:33 <obasan> I think that this is it
14:25:33 <obasan> for now of course
14:25:39 <ewoud> how are we going to manage users there? every ovirt member one account?
14:26:00 <obasan> ewoud, yes. I think
14:26:05 <obasan> ewoud, we don't have ldap
14:26:10 <obasan> ewoud, right?
14:26:11 <ewoud> obasan: how are they managed now? htpaswd?
14:26:12 <ewoud> obasan: no
14:26:37 <obasan> ewoud, once we have ldap we can use it. until them it's htpasswd
14:26:51 <ewoud> ok
14:27:07 <ewoud> #topic hosting
14:27:16 <ewoud> dcaro: eedri any progress on rackspace hosts?
14:27:35 <dneary> Hi all
14:27:53 <eedri> ewoud, dcaro did you try installing with the cdrom?
14:28:27 <eedri> ewoud, dcaro tried installing it with multiple os, but we keep hitting the DRAC problems no matter what
14:28:30 <ewoud> eedri: I don't have access to the machines
14:28:49 <eedri> ewoud, sorry, i meant to ping dcaro only :)
14:28:53 <ewoud> eedri: is it a DRAC firmware bug? maybe that needs to be upgraded?
14:29:10 <eedri> ewoud, quaid suggested we contact rackspace to put a cd in the machine and it might work
14:29:50 <eedri> ewoud, there is an open bug on it, and a workaround as well, but it also didn't help
14:29:57 <ewoud> eedri: I don't know about the VPN, but can you DHCP + TFTP over it?
14:30:12 <eedri> ewoud, the vpn issue is resolved
14:31:08 <eedri> ewoud, i would think that letting them install the os would be optimal, but that's not part of the support
14:31:16 <dcaro> eedri: I'm trying to install using kickstart from foreman, but I don't seem to be lucky, I'll ask them to put in a cd
14:31:50 <dcaro> ewoud: ^
14:31:50 <eedri> dcaro, yea.. let's try it simple
14:31:55 * ewoud has to brb
14:34:19 <ewoud> ok, keep us informend
14:34:24 <ewoud> informed*
14:34:27 <dcaro> at least we have now some installation media working on foreman :)
14:35:30 <ewoud> I just did think of something
14:35:38 <ewoud> some RH employee has to request monitoring.ovirt.org
14:35:43 <ewoud> in the DNS
14:35:56 <eedri> dneary, ^^ ?
14:36:17 <dneary> ewoud, eedri: Specifically the domain name?
14:36:44 <ewoud> dneary: it should point to 31.15.26.10
14:36:56 <dneary> ewoud, eedri: Can one of you open a ticket in Trac, and I will ensure it is looked at? We should initially assign it to Theron
14:37:07 <dcaro> can we do that any subdomain will point there also? it comes very handy when doing tests and similar things
14:37:15 <dneary> ewoud, That's a piwik install?
14:37:40 <ewoud> dneary: no, icinga and I hope to add something like munin for trending as well
14:37:49 <dneary> ewoud, Or onitoring in the nagios sense?
14:38:30 <ewoud> dneary: so in the nagios sense
14:38:38 <dneary> ok
14:38:43 <dneary> We definitely need that
14:39:27 <ewoud> we shouldn't have issues with full disks etc, rather sad we had downtime in this time
14:40:07 <eedri> ewoud, i understood that the ovirt-instance on alterway2 is lacking of space?
14:40:26 <eedri> ewoud, can't we add another lvm to it ?
14:40:30 <ewoud> eedri: the LV was rather small I think
14:41:29 <eedri> ewoud, i understood that it might be a problem to extend it on a running system?
14:41:49 <ewoud> eedri: I think Kiril was looking into it
14:42:18 <eedri> ewoud, natually i would think that we can just extend it
14:42:32 <ewoud> eedri: do you know where Kiril is?
14:43:02 <eedri> ewoud, he's ooo
14:43:13 <eedri> ewoud, i think there shouldn't be a problem with extending the vg/lv
14:43:24 <eedri> ewoud, we might want to check that on a test vm with ovirt-engien
14:43:35 <eedri> ewoud, before doing it on alterway2
14:43:43 <ewoud> eedri: I don't know if you can expand a storage volume in ovirt
14:44:33 <eedri> ewoud, maybe asking on users/devel list?
14:44:59 <ewoud> eedri: Kiril was checking with someone from storage I think, but couldn't hurt to ask on users/devel
14:46:05 <dcaro> dneary: can we ask for globbed subdomains? *.monitoring.redhat.com for example?
14:46:23 <ewoud> dcaro: what do you want that for?
14:46:49 <ewoud> or did you mean *.monitoring.ovirt.org?
14:46:56 <dneary> dcaro, I don't know - we can try.
14:47:02 <dneary> dcaro, What is the use-case?
14:47:04 <dcaro> ewoud: yep,  it comes very handy when doing upgrades, installing new services...
14:47:33 <dcaro> f.e. munin.monitoring.ovirt.org will point to that, we can have a vhost for munin
14:47:57 <dcaro> or if we want to try other services/version we can have them in parallel without having to ask for new domain entries
14:48:25 <dneary> dcaro, I'm no DNS expert - how easy is something like that in BIND?
14:48:29 <ewoud> dcaro: I'd like to just have control over the domain so maybe we should look into something
14:48:33 <dcaro> it's just one entry
14:49:54 <dcaro> ewoud: that'd be the best option, sure
14:50:49 <ewoud> I have no experience with domain providers other than some two Dutch providers
14:51:02 <ewoud> one of which being my employer
14:51:55 <dcaro> dcaro: I usually just install bind on my own and manage it from there :S (always had redundant machines for that)
14:52:11 <dcaro> but I've used also a couple of spanish and french providers
14:52:49 <ewoud> maybe we should just ignore it for now, plenty of other tasks
14:53:13 <ewoud> #topic jenkins
14:53:15 <ewoud> anything new?
14:54:26 <ewoud> eedri: dcaro ?
14:54:54 <dcaro> ewoud: not afaik
14:54:57 <ewoud> ok
14:55:06 <ewoud> #topic other business
14:55:22 <ewoud> I think we should move the meeting to another time, so I'll propose something on the ML
14:57:22 <eedri> ewoud, yes
14:57:27 <eedri> ewoud, there is one issue on #jenkins
14:57:44 <ewoud> eedri: backups?
14:57:46 <eedri> ewoud, kiril sent an email on backing up jenkins
14:57:48 <eedri> ewoud, yes
14:57:53 <eedri> ewoud, no one replied
14:58:07 <eedri> ewoud, this is very important issue, since we currently don't have backups
14:59:06 <dneary> I have one other piece of business
14:59:21 <ewoud> eedri: I'll have a look
14:59:31 <dneary> We have some updates that need to be applied to the theme to defend against potential XSS
14:59:55 <eedri> dneary, which theme?
15:00:06 <ewoud> speaking of updates, we should also update the machines at alterway with the latest yum updates
15:00:09 <dneary> The website themeI need a volunteer with time to apply the fix and check - and we should review the theme to ensure that
15:00:33 <dneary> ... to ensure that the same problem isn't affecting the site elsewhere
15:00:33 <ovirtbot> dneary: Error: ".." is not a valid command.
15:00:51 <dneary> ovirtbot, I didn't kniw . did anything special
15:00:51 <ovirtbot> dneary: Error: "I" is not a valid command.
15:01:04 <dneary> ovirtbot, thanks
15:01:04 <ovirtbot> dneary: Error: "thanks" is not a valid command.
15:01:21 * dneary stops messing and gets serious again
15:01:42 <ewoud> dneary: I'd help, but not sure I have the time
15:05:47 <ewoud> eedri: can you update jenkins.ovirt.org?
15:06:16 <eedri> ewoud, yum update?
15:06:20 <ewoud> eedri: yes
15:06:26 <eedri> ewoud, yes
15:06:28 <eedri> ewoud, but
15:06:33 <eedri> ewoud, wouldn't we want to run backup 1st?
15:06:44 <ewoud> eedri: true
15:07:07 <dcaro> yep!
15:07:17 <ewoud> same could be said for alterway02 btw
15:07:35 <ewoud> dneary: maybe you can mail the ML for help?
15:10:22 <ewoud> anyway, I think we should close the meeting
15:10:48 <ewoud> #action eedri update jenkins.ovirt.org after a backup was made
15:10:51 <ewoud> going once
15:12:01 <ewoud> #endmeeting