14:22:51 #startmeeting oVirt Infra 14:22:51 Meeting started Mon Jun 16 14:22:51 2014 UTC. The chair is dcaro. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:22:51 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:23:07 #chair ewoud eedri_ dcaro 14:23:07 Current chairs: dcaro eedri_ ewoud 14:23:28 anyine else around? 14:23:32 *anyone 14:24:00 yep 14:24:23 rbarry, here? 14:24:24 #chair misc 14:24:24 Current chairs: dcaro eedri_ ewoud misc 14:24:37 rbarry, care to join the infra meeting as well ? 14:27:29 dcaro, shall we start? 14:28:13 dcaro: do you just want to review tickets or do you have an agenda? 14:29:44 ewoud: well, there's no clear agenda, but we have some points to discuss 14:29:51 #topic Hosting 14:30:36 #info new lab in phx is almost here, found some issues installing base os 14:30:53 woow 14:31:05 hopefully it will be ready pretty soon, I'm (right now) talking with the guy to get all the issues fixed 14:31:15 misc, care to update on your efforts on the alterway vms? 14:31:26 misc, you've updated some vms right? 14:32:35 eedri_: let me dig what I did, as i was quite busy on others projects 14:32:49 In fact, i just did update the VM where I had access 14:33:06 stats.ovirt.org 14:33:06 foreman.ovirt.org 14:33:06 monitoring.ovirt.org 14:33:10 lists.ovirt.org 14:33:57 eedri_: but we do not really have a list of hosts, or I didn't found a complete one 14:35:31 misc, i think they should be managed all via foreman.ovirt.org 14:35:38 misc, so maybe we can extract a list from there? 14:35:45 eedri_: misc I think that there are a couple that are not 14:35:50 gerrit for example 14:36:06 (that does not mean that they shouldn't) 14:36:18 well, ideally, everything should be there 14:36:33 yep 14:36:41 but right now, foreman say 100% out of sync :( 14:36:49 misc, maybe this can help http://monitoring.ovirt.org/ovirt.apps/ 14:36:55 we can set a task to add them at least (even if notihng runs, just to inventory) 14:36:56 misc, we can update it 14:37:43 eedri_: yeah, that's one information source 14:37:52 I also did look at the password file 14:37:54 and the DNS 14:38:41 misc, yea, i think we should do a survey and document all the services/server on the wiki page 14:38:55 misc, including which backups we ahve 14:39:01 misc, or dont :( 14:39:17 eedri_: yep, let's do that 14:39:26 make sure that they are in puppet/foreman 14:39:28 mmm, I don't like having that info in a wiki (too static) 14:39:32 in monitoring 14:39:38 dcaro, what do you suggest? 14:39:47 dcaro, maybe using jenkins/graphite? 14:40:03 dcaro, to autodiscover 14:40:04 adding the hosts to foreman/nagios for inventory and alerts 14:40:10 dcaro: yep 14:40:15 that's the end goal 14:40:27 for now, we need a list, then make sure everything is in foreman 14:40:32 and maybe (highly recommended) add a check for backups 14:40:44 yes, the list is ok as a checklist 14:40:46 dcaro, so once all is managed by foreman - we can add a job that creates weekly reports? 14:40:58 dcaro, misc we can use etherpad maybe for a checklist 14:41:02 dcaro, i think we had one 14:41:03 +1 14:41:16 either way is fine for me 14:41:19 that is also easier to modify and add comments 14:41:21 as long as we all use the same :) 14:41:43 dcaro, how can we look for existing etherpads on etherpad.ovirt.org? 14:41:43 #action create a pad with the current list of hosts to add to foreman 14:42:13 eedri_: no idea... who's the admin 14:42:14 ? 14:42:38 that's on openshift 14:43:01 etherpad-ovirtapps.rhcloud.com , so the person who own the domain is the admin 14:43:20 misc, http://etherpad.ovirt.org/ 14:43:24 misc, this one also? 14:44:25 yes, they are the same 14:44:27 eedri_: yep 14:44:34 ahadas: ping 14:45:10 misc: eedri_ wehre can we get that info? 14:45:21 dcaro, maybe bkp knows 14:45:23 dcaro, or dneary 14:45:26 yeah 14:45:31 dcaro, or the file on resources.ovirt.org 14:45:32 if not, I can ask to openshift people 14:45:45 Info on OpenShift? 14:45:55 bkp: yep, there is 2 account used by ovirt 14:46:02 ovirtapps and ovirt 14:46:13 ovirt as shared by quaid and ou and dneary (and me) have access 14:46:17 ovirtapps, no idea 14:46:34 Yeah, I do not know about ovirtapps 14:46:44 I go in through ovirt. 14:46:50 jvandewege: ping using ldap ? 14:47:31 ok, let's say that I am gonna find out for next meeting ? 14:48:15 misc, +1 14:48:33 dcaro, let's review important tasks? 14:48:47 #action misc find out who own ovirtapps account on openshift 14:48:58 eedri_: ok 14:49:08 #topic Tasks 14:49:27 #action write a list of servers/services on etherpad, to have them in foreman/monitoring/backup 14:50:06 https://fedorahosted.org/ovirt/ticket/151 14:50:37 gerrit load was caused by the garbage on the git repos, running 'gerrit gc' on the repos leveraged a lot the load 14:51:05 I've created a task to add a cron/job to run it periodically 14:51:52 that does not mean that the mirrors are not welcome though, specially for jenkins jobs 14:52:12 for now some of the jobs are using github as mirror to clone and just fetch from gerrit 14:52:56 any ideas? 14:53:21 Using the python-sdk I can create disks/add them to a VM, but I can't figure out how to "activate" them.  Can someone point me in the right direction? (I did see an 'activate' action, but I don't know how to use it for this) 14:54:21 dmachi: a irc meeting is going on, we will answer likely later :) 14:54:54 ok, next task: https://fedorahosted.org/ovirt/ticket/17 14:55:03 that is related to the action item se set up earlier 14:55:04 misc: thanks, sorry that I interrupted, didn't realize that was happening. 14:55:18 (the etherpad with the inventory) 14:56:23 misc: do you mind creating a trac for the etherpad list and add it as a dep here? 14:57:09 dcaro: yep, will do 14:57:18 thanks :) 14:57:56 dcaro, ideas for scaling gerrit? 14:57:56 dcaro, well, the mirrors has be public? 14:58:10 ooops 14:58:19 * eedri got disconnected 14:58:22 eedri: internal mirror on the new lab would be enough, I think 14:58:45 eedri: about the mirrors, I see no need for them to be private 14:59:58 https://fedorahosted.org/ovirt/ticket/136 - kiril is not around for this 15:00:47 https://fedorahosted.org/ovirt/ticket/111 <- this is mine, it's not that critical, but it has to be done, I'll change the priority though 15:01:21 https://fedorahosted.org/ovirt/ticket/131 <- this is blocked 15:01:30 (that was phx3 foreman proxy) 15:02:26 ok, now we have 3 jenkins jobs failures 15:03:46 we are having a few issues with packaging and dependencies lately, and we have started migrating to mock some of the jobs, anyone has a better alternative? (the main issue with mock is that it's rpm centric and only meant to build rpms, not run jobs) 15:04:30 * misc can you explain a bit more ? 15:04:42 sure 15:05:08 when running a job for a project, it usually needs some specific packages 15:05:27 right now as we mostly have packaginf jobs, it's usually requirements to build the projec 15:06:09 it might change by project version and by target distribution 15:06:25 dcaro: ansible 15:06:39 dmachi: how can ansible help there? 15:07:32 (we already use puppet btw, and afaik, they are quite similar) 15:07:45 well i don't know many details about what you are trying to do obviously, but since you can easily test for os/packages/etc its easy to customize a job that is being run and that job cn be running physically anywhere you can ssh to 15:08:03 yes similar to chef/puppet, but with no central server tying it all together. 15:08:28 so the idea would be to write a playbook ( ie, equivalent of a script ) for the job 15:08:38 yep 15:08:44 and make them run on any server, or even create a server for them to run 15:08:49 well, the issue is that we want to use the current resources as much as possible, and doing that binds a slave to a job, so it can't run anything else 15:08:51 and then remove the server or reuse it ? 15:09:16 yeah, well using the ovirt module, you can include launching/setting up the node, run a job, shutdown node. 15:09:37 (which is in fact what I'm doing with our cluster) :) 15:10:27 that's interesting (as we tought of doing that also at some point) 15:11:14 so for that we'll need all the machines we have to be ovirt hosts or single executor slaves 15:11:33 to manage a big set of server, you need something like puppet (config management) and remote execution (fabric, ansible), for now, we only have puppet 15:13:12 dmachi: then the issue is how long does it take to provision a host on ovirt 15:13:13 misc: I'd never use fabric to manage a big set of servers, but ansible is certainly able to handle it (and if you need a centralized server, there are those available too), but anyway, it was just a lurker's observation :) 15:13:44 * misc also use ansible for his own server and for others projects 15:14:34 misc: my nodes start (from a template) pretty quickly, couple minutes. I have a bunch of other config for networks and such , so it an take longer for some, but generally its pretty quick. Not to say its perfect for this case, as I just butted into a conversation I don't know the details of :P 15:15:25 dmachi: how do you manage remote phisical servers? (we have quite a few servers not on the same networks/locations ) 15:16:30 just occurs via ssh, my physical servers I define as "static" inventory, and then i have a dynamic inventory script that talks to ovirt api for dynamic inventory. Everything occurs over ssh, and hte whole playbook setup is in git. 15:16:56 You can also tell ansible to proxy ssh connections via another host if you need to get behind a firewall or something. 15:17:08 yep, it use the ssh binary, so you can be quite creative 15:17:38 I meant, you only run one job at the same time on each host? 15:18:11 you would, yes 15:18:21 most of our hosts are big machines on remote places, running ony one job at a time seems like waiting resources 15:18:23 *wasting 15:18:25 well, depend on how you do it 15:18:27 well if you ran two ansible playbooks in two different shells simultaneously you could probably acocomplish more than one, but by default its one at a time. 15:18:48 the fact is what if 1 job requires to install 1 rpm and the 2nd requires to remove it :) 15:18:55 exactly 15:19:50 that's the issue (running a versioned manifest without master can be done with puppet too, that's not new for ansible) 15:20:29 sure, you'd certainly have to plan, but is there any case where if you have two competing scripts like that on the same host you aren't going to have an issue? 15:21:03 if you use mock, for example, it creates a chroot for each process with it's own package layout and binary versions 15:21:16 what about docker ? 15:21:37 so you can run many parallel rpm builds with different distribution layouts and arches (well, only i686 can be mocked in x86_64 machines) 15:22:14 was gonna say docker too. So using ansible (or puppet presumably) you can automate that too, doesn't have to be ovirt hosts. 15:22:41 we started checking docker, but it's quite new and it does not support systemd, needed for most of the non-package tests, so no big advantage over mock (well, yes, that it can build non-rpm packages) 15:22:42 but docker has no selinux support, so you're not going to be able to test everything 15:22:57 also yes 15:23:38 do all the tests require selinux support or just a subset of them? 15:24:14 I think unit tests generally don't care about them 15:24:28 but as long as you have 1, then you need a 2nd solution 15:24:29 but integration tests would provide a more realistic result 15:26:39 i just ask because you could use docker for the more frequent unit tests, and then vms for the integration tests which would perhaps take a little bit longer to setup and be isolated to a single job/vm butbe more realistic 15:27:37 I'd expect you'd end up with something similar yes 15:27:39 * misc has another meeting 15:27:42 mmm, that will be a lot nicer if we use get the phx hosts 15:28:07 okok, we can go closing, I'll start a thread on the list to discuss this 15:28:30 #action dcaro to start a thread on build env alternatives (mock, docker, ovirt spawnn vms) 15:28:47 but I guess that at least the vm part will have to wait to phx 15:29:15 anyone wants to rais an issue before finishing the meeting? 15:29:31 dmachi: thanks for the ideas! :) 15:29:54 dcaro: np 15:31:47 dcaro: I'm making an additional ansible module for managing some of the other object types (they only do vms right now). But I can't figure out how to activate a disk that has been attached to a vm with the python sdk. Can you point me in the right direction? 15:32:52 dmachi: not really, haven't played too much with it, you can try asking on devel list (or vdsm list) 15:33:27 ewoud: eedri misc ok, so I'll close the meeting for today 15:33:42 dcaro: ok, thanks 15:33:46 dcaro, ok 15:33:49 #endmeeting