15:01:09 <knesenko> #startmeeting oVirt Infra 15:01:10 <ovirtbot> Meeting started Mon Nov 4 15:01:09 2013 UTC. The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:10 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic. 15:01:19 <knesenko> #chair ewoud obasan 15:01:19 <ovirtbot> Current chairs: ewoud knesenko obasan 15:01:29 <obasan> dcaro, ping 15:01:47 * dcaro is here 15:01:50 <dcaro> obasan: pong 15:01:51 <obasan> #chair dcaro 15:01:51 <ovirtbot> Current chairs: dcaro ewoud knesenko obasan 15:02:28 <knesenko> #chair eedri 15:02:28 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan 15:03:00 <knesenko> #topic Hosting 15:03:22 <knesenko> dcaro: have you rebooted the rackspace03 server ? 15:04:35 <dcaro> knesenko: I've tried connecting to the webadmin with no luck, and I haven't found a way of doing it from the ui, I'll open a ticket for it 15:04:46 <knesenko> dcaro: thanks 15:05:02 <knesenko> #action dcaro open a ticket to rackspace to reboot ovirt03 15:05:14 <knesenko> so i am still blocked on installation 15:05:43 <knesenko> any other news/issues on hosting ? 15:06:29 <knesenko> I have some thoughts about gluster way we choose 15:06:58 <dcaro> knesenko: please share 15:07:13 <knesenko> because we are going to use a local disks, and those disks will be used by jenkins slaves I think it will create high IO load on those disks 15:07:46 <knesenko> because currently jenkins slaves are in high load 15:08:02 <knesenko> that's what I think ... 15:08:10 <knesenko> more VMs we will add, more IO load 15:08:47 <knesenko> thoughts ? 15:09:02 <ewoud> can't ovirt use local disks and bind a VM to a specific hypervisor inside a cluster? 15:09:56 <ewoud> we could create a /var/lib/jenkins mount point and pass that through 15:09:59 <obasan> ewoud, I think that it's possible to bind a vm to a specific hypervisor 15:10:23 <knesenko> ewoud: its possible 15:10:38 <dcaro> knesenko: are you sure that the current bottleneck is the disk io? 15:10:59 <knesenko> dcaro: I am not 100% sure ... but that's what Sasha told ... 15:11:30 <dcaro> knesenko: can we check with iotop or something? 15:11:50 <knesenko> dcaro: i think we should 15:12:08 <ewoud> often it's a fair guess 15:12:26 <ewoud> btw, using tuned could also help 15:12:32 <dcaro> let's make sure we are not fixing a problem we don't have though 15:12:43 <ewoud> the default scheduler isn't that well suited for virtualisation which also schedules 15:15:15 <knesenko> dcaro: its just a guess 15:16:31 <knesenko> so what we decide to do here ? 15:16:41 <knesenko> monitor IO a little bit on those servers ? 15:16:53 <dcaro> knesenko: that's ok, guesses are ok as a starter point, now let's continue the idea and make sure, I vote for gathering some info on current usage before taking any paliative action 15:17:06 <knesenko> dcaro: +1 15:17:19 <knesenko> obasan: can we monitor disks IO on incignia ? 15:17:19 <ewoud> sounds like we're back to monitoring and trending 15:17:35 <obasan> knesenko, sure 15:18:04 <dcaro> yep :/, I've installed a graphite instance on stats.ovirt.org, I'd like to send there some metrics, maybe we can use it also for that 15:18:06 <knesenko> #action ad rackspace* servers to incingia to monitor IO, CPIU, mem etc .ASAP 15:18:17 <knesenko> dcaro: +1 15:18:20 <knesenko> can you do that ? 15:18:33 <knesenko> if u have time for that 15:18:51 <dcaro> we will have to figure out a way of securing the channel (graphite does not handle any authentication/ssl) 15:19:51 <knesenko> dcaro: do we care ? 15:19:57 <knesenko> dcaro: its just monitoring info ... 15:20:06 <knesenko> dcaro: and without anycredentials ... 15:20:21 <dcaro> knesenko: at least to avoid anyone from sending us fake stats 15:20:38 <knesenko> dcaro: i assume rackspace* servers will send info to stats.ovirt.org - right ? 15:20:51 <dcaro> yep 15:21:16 <knesenko> dcaro: you can create iptables rules on stats.ovirt.org to receive info only from rackspace* hosts 15:21:20 <knesenko> can we ? 15:21:25 <dcaro> sure 15:21:38 <knesenko> dcaro: i think it will e enough for now ... 15:21:42 <knesenko> objections ? 15:22:12 <ewoud> not right now 15:22:42 <ewoud> I don't know graphite though 15:24:00 <knesenko> dcaro: obasan ? 15:24:06 <dcaro> +1 15:24:13 <obasan> obasan, I don't know either :) 15:24:42 <knesenko> #action dcaro setup grafite to collect info from rackspace servers + iptables rules 15:24:45 <knesenko> ok 15:24:54 <knesenko> #topic Foreman and Puppet 15:25:01 <knesenko> dcaro: ewoud news guys 15:25:02 <knesenko> ?= 15:25:20 <dcaro> I did nothing... :/ 15:25:28 <ewoud> no, but I do intend to finish the r10k deployment patch today 15:25:34 <dcaro> nice! 15:25:35 <obasan> how about merging that http://gerrit.ovirt.org/#/c/20622/ ? 15:26:05 <ewoud> dcaro: I did describe the general workflow a while back so if you can take a look at that 15:27:43 <ewoud> obasan: I gave my +1, but I'd like dcaro to give it at least +1 as well 15:28:01 <obasan> ewoud, great ! 15:28:05 <dcaro> looks good to me :) 15:29:21 <obasan> I also think that we need a testing environment. it's a problem verifying puppet patches like this 15:29:29 <knesenko> dcaro: ewoud do we have F20 hostgroup ? 15:30:00 <ewoud> knesenko: no, but I think everything we do for F19 should work on F20 as well 15:30:06 <knesenko> nm we have only slaves hostgroup 15:30:14 <dcaro> yep 15:31:01 <knesenko> ok 15:31:05 <knesenko> anything else on puppet ? 15:31:08 <knesenko> foreman ? 15:31:22 <ewoud> I'd like to upgrade to 1.3 15:31:36 <obasan> I also think that we need a testing environment. it's a problem verifying puppet patches like this 15:31:44 <obasan> I think its important 15:31:49 <obasan> and prevent future breakage 15:31:53 <ewoud> obasan: agreed, but sometimes you sort of need production load to test 15:32:03 <ewoud> such as tmpwatch that only does something after 12 hours 15:32:29 <obasan> ewoud, correct. but I still think that we need some kind of a simple way to test functionality 15:33:09 <ewoud> ideally we'd write tests with rspec-puppet and rspec-system to do CI builds before pushing 15:33:15 <obasan> ewoud, in the tmpwatch case I would like to see it installs and adds the cron entry 15:33:37 <obasan> ewoud, that sounds interesting. I don't know these tools 15:33:59 <obasan> ewoud, but maybe we should add that to jenkins if it tests puppet manifests functioning 15:34:01 <ewoud> obasan: http://rspec-puppet.com/tutorial/ you can at least let puppet compile the catalog and verify the package entry is there 15:34:31 <obasan> it looks good. I recommend a jenkins gerrit job 15:34:39 <obasan> that will use this tool to auto verify puppet patches 15:35:00 <ewoud> actually, that's http://gerrit.ovirt.org/20367 15:35:13 <ewoud> obasan: that should allow at least basic testing 15:35:26 <obasan> ewoud, great. why is it not merged yet? problems? 15:35:39 <ewoud> obasan: no reviews 15:35:39 <knesenko> #action obasan add job in jenkins to use http://gerrit.ovirt.org/20367 and verify puppet patches 15:36:00 <obasan> ewoud, I will review it 15:36:15 <ewoud> obasan: if you need help writing a jenkins job for it, do say so 15:36:30 <obasan> ewoud, thank you . let's get it merged first 15:37:34 <knesenko> ewoud: obasan +1 15:37:52 <ewoud> for foreman 1.3, should I send an announcement at least a few hours in advance? 15:38:06 <obasan> ewoud, +1 15:38:37 <ewoud> #info foreman upgrade to 1.3 will be announced on the ML 15:39:09 <knesenko> ok ... anything else in this topic ? 15:39:18 <ewoud> not from me 15:39:32 <knesenko> #topic jenkins 15:39:46 <knesenko> obasan: you upgraded jenkins last week right ? 15:40:09 <obasan> knesenko, nope. we're on the latest LTS 15:40:23 <knesenko> obasan: ok :\ 15:40:31 <knesenko> where is eedri to update with the status ? 15:40:56 <knesenko> status looks ok on jenkins for me 15:40:58 <knesenko> at least 15:42:27 <knesenko> ok ... 15:42:47 <knesenko> #topic other business 15:42:58 <knesenko> any other topics we want to discuss ? 15:45:28 <knesenko> #endmeeting