15:01:09 <knesenko> #startmeeting oVirt Infra
15:01:10 <ovirtbot> Meeting started Mon Nov  4 15:01:09 2013 UTC.  The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:10 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:01:19 <knesenko> #chair ewoud obasan
15:01:19 <ovirtbot> Current chairs: ewoud knesenko obasan
15:01:29 <obasan> dcaro, ping
15:01:47 * dcaro is here
15:01:50 <dcaro> obasan: pong
15:01:51 <obasan> #chair dcaro
15:01:51 <ovirtbot> Current chairs: dcaro ewoud knesenko obasan
15:02:28 <knesenko> #chair eedri
15:02:28 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan
15:03:00 <knesenko> #topic Hosting
15:03:22 <knesenko> dcaro: have you rebooted the rackspace03 server ?
15:04:35 <dcaro> knesenko: I've tried connecting to the webadmin with no luck, and I haven't found a way of doing it from the ui, I'll open a ticket for it
15:04:46 <knesenko> dcaro: thanks
15:05:02 <knesenko> #action dcaro open a ticket to rackspace to reboot ovirt03
15:05:14 <knesenko> so i am still blocked on installation
15:05:43 <knesenko> any other news/issues on hosting ?
15:06:29 <knesenko> I have some thoughts about gluster way we choose
15:06:58 <dcaro> knesenko: please share
15:07:13 <knesenko> because we are going to use a local disks, and those disks will be used by jenkins slaves I think it will create high IO load on those disks
15:07:46 <knesenko> because currently jenkins slaves are in high load
15:08:02 <knesenko> that's what I think ...
15:08:10 <knesenko> more VMs we will add, more IO load
15:08:47 <knesenko> thoughts ?
15:09:02 <ewoud> can't ovirt use local disks and bind a VM to a specific hypervisor inside a cluster?
15:09:56 <ewoud> we could create a /var/lib/jenkins mount point and pass that through
15:09:59 <obasan> ewoud, I think that it's possible to bind a vm to a specific hypervisor
15:10:23 <knesenko> ewoud: its possible
15:10:38 <dcaro> knesenko: are you sure that the current bottleneck is the disk io?
15:10:59 <knesenko> dcaro: I am not 100% sure ... but that's what Sasha told ...
15:11:30 <dcaro> knesenko: can we check with iotop or something?
15:11:50 <knesenko> dcaro: i think we should
15:12:08 <ewoud> often it's a fair guess
15:12:26 <ewoud> btw, using tuned could also help
15:12:32 <dcaro> let's make sure we are not fixing a problem we don't have though
15:12:43 <ewoud> the default scheduler isn't that well suited for virtualisation which also schedules
15:15:15 <knesenko> dcaro: its just a guess
15:16:31 <knesenko> so what we decide to do here ?
15:16:41 <knesenko> monitor IO a little bit on those servers ?
15:16:53 <dcaro> knesenko: that's ok, guesses are ok as a starter point, now let's continue the idea and make sure, I vote for gathering some info on current usage before taking any paliative action
15:17:06 <knesenko> dcaro: +1
15:17:19 <knesenko> obasan: can we monitor disks IO on incignia ?
15:17:19 <ewoud> sounds like we're back to monitoring and trending
15:17:35 <obasan> knesenko, sure
15:18:04 <dcaro> yep :/, I've installed a graphite instance on stats.ovirt.org, I'd like to send there some metrics, maybe we can use it also for that
15:18:06 <knesenko> #action ad rackspace* servers to incingia to monitor IO, CPIU, mem etc .ASAP
15:18:17 <knesenko> dcaro: +1
15:18:20 <knesenko> can you do that ?
15:18:33 <knesenko> if  u have time for that
15:18:51 <dcaro> we will have to figure out a way of securing the channel (graphite does not handle any authentication/ssl)
15:19:51 <knesenko> dcaro: do we care ?
15:19:57 <knesenko> dcaro: its just monitoring info ...
15:20:06 <knesenko> dcaro: and without anycredentials ...
15:20:21 <dcaro> knesenko: at least to avoid anyone from sending us fake stats
15:20:38 <knesenko> dcaro: i assume rackspace* servers will send info to stats.ovirt.org - right ?
15:20:51 <dcaro> yep
15:21:16 <knesenko> dcaro: you can create iptables rules on stats.ovirt.org to receive info only from rackspace* hosts
15:21:20 <knesenko> can we ?
15:21:25 <dcaro> sure
15:21:38 <knesenko> dcaro: i think it will e enough for now ...
15:21:42 <knesenko> objections ?
15:22:12 <ewoud> not right now
15:22:42 <ewoud> I don't know graphite though
15:24:00 <knesenko> dcaro: obasan ?
15:24:06 <dcaro> +1
15:24:13 <obasan> obasan, I don't know either :)
15:24:42 <knesenko> #action dcaro setup grafite to collect info from rackspace servers + iptables rules
15:24:45 <knesenko> ok
15:24:54 <knesenko> #topic Foreman and Puppet
15:25:01 <knesenko> dcaro: ewoud news guys
15:25:02 <knesenko> ?=
15:25:20 <dcaro> I did nothing... :/
15:25:28 <ewoud> no, but I do intend to finish the r10k deployment patch today
15:25:34 <dcaro> nice!
15:25:35 <obasan> how about merging that http://gerrit.ovirt.org/#/c/20622/ ?
15:26:05 <ewoud> dcaro: I did describe the general workflow a while back so if you can take a look at that
15:27:43 <ewoud> obasan: I gave my +1, but I'd like dcaro to give it at least +1 as well
15:28:01 <obasan> ewoud, great !
15:28:05 <dcaro> looks good to me :)
15:29:21 <obasan> I also think that we need a testing environment. it's a problem verifying puppet patches like this
15:29:29 <knesenko> dcaro: ewoud do we have F20 hostgroup ?
15:30:00 <ewoud> knesenko: no, but I think everything we do for F19 should work on F20 as well
15:30:06 <knesenko> nm we have only slaves hostgroup
15:30:14 <dcaro> yep
15:31:01 <knesenko> ok
15:31:05 <knesenko> anything else on puppet ?
15:31:08 <knesenko> foreman ?
15:31:22 <ewoud> I'd like to upgrade to 1.3
15:31:36 <obasan> I also think that we need a testing environment. it's a problem verifying puppet patches like this
15:31:44 <obasan> I think its important
15:31:49 <obasan> and prevent future breakage
15:31:53 <ewoud> obasan: agreed, but sometimes you sort of need production load to test
15:32:03 <ewoud> such as tmpwatch that only does something after 12 hours
15:32:29 <obasan> ewoud, correct. but I still think that we need some kind of a simple way to test functionality
15:33:09 <ewoud> ideally we'd write tests with rspec-puppet and rspec-system to do CI builds before pushing
15:33:15 <obasan> ewoud, in the tmpwatch case I would like to see it installs and adds the cron entry
15:33:37 <obasan> ewoud, that sounds interesting. I don't know these tools
15:33:59 <obasan> ewoud, but maybe we should add that to jenkins if it tests puppet manifests functioning
15:34:01 <ewoud> obasan: http://rspec-puppet.com/tutorial/ you can at least let puppet compile the catalog and verify the package entry is there
15:34:31 <obasan> it looks good. I recommend a jenkins gerrit job
15:34:39 <obasan> that will use this tool to auto verify puppet patches
15:35:00 <ewoud> actually, that's http://gerrit.ovirt.org/20367
15:35:13 <ewoud> obasan: that should allow at least basic testing
15:35:26 <obasan> ewoud, great. why is it not merged yet? problems?
15:35:39 <ewoud> obasan: no reviews
15:35:39 <knesenko> #action obasan add job in jenkins to use http://gerrit.ovirt.org/20367 and verify puppet patches
15:36:00 <obasan> ewoud, I will review it
15:36:15 <ewoud> obasan: if you need help writing a jenkins job for it, do say so
15:36:30 <obasan> ewoud, thank you . let's get it merged first
15:37:34 <knesenko> ewoud: obasan +1
15:37:52 <ewoud> for foreman 1.3, should I send an announcement at least a few hours in advance?
15:38:06 <obasan> ewoud, +1
15:38:37 <ewoud> #info foreman upgrade to 1.3 will be announced on the ML
15:39:09 <knesenko> ok ... anything else in this topic ?
15:39:18 <ewoud> not from me
15:39:32 <knesenko> #topic jenkins
15:39:46 <knesenko> obasan: you upgraded jenkins last week right ?
15:40:09 <obasan> knesenko, nope. we're on the latest LTS
15:40:23 <knesenko> obasan: ok :\
15:40:31 <knesenko> where is eedri to update with the status ?
15:40:56 <knesenko> status looks ok on jenkins for me
15:40:58 <knesenko> at least
15:42:27 <knesenko> ok ...
15:42:47 <knesenko> #topic other business
15:42:58 <knesenko> any other topics we want to discuss ?
15:45:28 <knesenko> #endmeeting