14:22:51 <dcaro> #startmeeting oVirt Infra
14:22:51 <ovirtbot> Meeting started Mon Jun 16 14:22:51 2014 UTC.  The chair is dcaro. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:22:51 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:23:07 <dcaro> #chair ewoud eedri_ dcaro
14:23:07 <ovirtbot> Current chairs: dcaro eedri_ ewoud
14:23:28 <dcaro> anyine else around?
14:23:32 <dcaro> *anyone
14:24:00 <misc> yep
14:24:23 <eedri_> rbarry, here?
14:24:24 <dcaro> #chair misc
14:24:24 <ovirtbot> Current chairs: dcaro eedri_ ewoud misc
14:24:37 <eedri_> rbarry, care to join the infra meeting as well ?
14:27:29 <eedri_> dcaro, shall we start?
14:28:13 <ewoud> dcaro: do you just want to review tickets or do you have an agenda?
14:29:44 <dcaro> ewoud: well, there's no clear agenda, but we have some points to discuss
14:29:51 <dcaro> #topic Hosting
14:30:36 <dcaro> #info new lab in phx is almost here, found some issues installing base os
14:30:53 <misc> woow
14:31:05 <dcaro> hopefully it will be ready pretty soon, I'm (right now) talking with the guy to get all the issues fixed
14:31:15 <eedri_> misc, care to update on your efforts on the alterway vms?
14:31:26 <eedri_> misc, you've updated some vms right?
14:32:35 <misc> eedri_: let me dig what I did, as i was quite busy on others projects
14:32:49 <misc> In fact, i just did update the VM where I had access
14:33:06 <misc> stats.ovirt.org
14:33:06 <misc> foreman.ovirt.org
14:33:06 <misc> monitoring.ovirt.org
14:33:10 <misc> lists.ovirt.org
14:33:57 <misc> eedri_: but we do not really have a list of hosts, or I didn't found a complete one
14:35:31 <eedri_> misc, i think they should be managed all via foreman.ovirt.org
14:35:38 <eedri_> misc, so maybe we can extract a list from there?
14:35:45 <dcaro> eedri_: misc I think that there are a couple that are not
14:35:50 <dcaro> gerrit for example
14:36:06 <dcaro> (that does not mean that they shouldn't)
14:36:18 <misc> well, ideally, everything should be there
14:36:33 <dcaro> yep
14:36:41 <misc> but right now, foreman say 100% out of sync :(
14:36:49 <eedri_> misc, maybe this can help http://monitoring.ovirt.org/ovirt.apps/
14:36:55 <dcaro> we can set a task to add them at least (even if notihng runs, just to inventory)
14:36:56 <eedri_> misc, we can update it
14:37:43 <misc> eedri_: yeah, that's one information source
14:37:52 <misc> I also did look at the password file
14:37:54 <misc> and the DNS
14:38:41 <eedri_> misc, yea, i think we should do a survey and document all the services/server on the wiki page
14:38:55 <eedri_> misc, including which backups we ahve
14:39:01 <eedri_> misc, or dont :(
14:39:17 <misc> eedri_: yep, let's do that
14:39:26 <misc> make sure that they are in puppet/foreman
14:39:28 <dcaro> mmm, I don't like having that info in a wiki (too static)
14:39:32 <misc> in monitoring
14:39:38 <eedri_> dcaro, what do you suggest?
14:39:47 <eedri_> dcaro, maybe using jenkins/graphite?
14:40:03 <eedri_> dcaro, to autodiscover
14:40:04 <dcaro> adding the hosts to foreman/nagios for inventory and alerts
14:40:10 <misc> dcaro: yep
14:40:15 <misc> that's the end goal
14:40:27 <misc> for now, we need a list, then make sure everything is in foreman
14:40:32 <dcaro> and maybe (highly recommended) add a check for backups
14:40:44 <dcaro> yes, the list is ok as a checklist
14:40:46 <eedri_> dcaro, so once all is managed by foreman - we can add a job that creates weekly reports?
14:40:58 <eedri_> dcaro, misc we can use etherpad maybe for a checklist
14:41:02 <eedri_> dcaro, i think we had one
14:41:03 <dcaro> +1
14:41:16 <misc> either way is fine for me
14:41:19 <dcaro> that is also easier to modify and add comments
14:41:21 <misc> as long as we all use the same :)
14:41:43 <eedri_> dcaro, how can we look for existing etherpads on etherpad.ovirt.org?
14:41:43 <dcaro> #action create a pad with the current list of hosts to add to foreman
14:42:13 <dcaro> eedri_: no idea... who's the admin
14:42:14 <dcaro> ?
14:42:38 <misc> that's on openshift
14:43:01 <misc> etherpad-ovirtapps.rhcloud.com , so the person who own the domain is the admin
14:43:20 <eedri_> misc, http://etherpad.ovirt.org/
14:43:24 <eedri_> misc, this one also?
14:44:25 <dcaro> yes, they are the same
14:44:27 <misc> eedri_: yep
14:44:34 <yzaslavs> ahadas: ping
14:45:10 <dcaro> misc: eedri_ wehre can we get that info?
14:45:21 <eedri_> dcaro, maybe bkp knows
14:45:23 <eedri_> dcaro, or dneary
14:45:26 <misc> yeah
14:45:31 <eedri_> dcaro, or the file on resources.ovirt.org
14:45:32 <misc> if not, I can ask to openshift people
14:45:45 <bkp> Info on OpenShift?
14:45:55 <misc> bkp: yep, there is 2 account used by ovirt
14:46:02 <misc> ovirtapps and ovirt
14:46:13 <misc> ovirt as shared by quaid and ou and dneary (and me) have access
14:46:17 <misc> ovirtapps, no idea
14:46:34 <bkp> Yeah, I do not know about ovirtapps
14:46:44 <bkp> I go in through ovirt.
14:46:50 <YamakasY> jvandewege: ping using ldap ?
14:47:31 <misc> ok, let's say that I am gonna find out for next meeting ?
14:48:15 <eedri_> misc, +1
14:48:33 <eedri_> dcaro, let's review important tasks?
14:48:47 <misc> #action misc find out who own ovirtapps account on openshift
14:48:58 <dcaro> eedri_: ok
14:49:08 <dcaro> #topic Tasks
14:49:27 <misc> #action write a list of servers/services on etherpad, to have them in foreman/monitoring/backup
14:50:06 <dcaro> https://fedorahosted.org/ovirt/ticket/151
14:50:37 <dcaro> gerrit load was caused by the garbage on the git repos, running 'gerrit gc' on the repos leveraged a lot the load
14:51:05 <dcaro> I've created a task to add a cron/job to run it periodically
14:51:52 <dcaro> that does not mean that the mirrors are not welcome though, specially for jenkins jobs
14:52:12 <dcaro> for now some of the jobs are using github as mirror to clone and just fetch from gerrit
14:52:56 <dcaro> any ideas?
14:53:21 <dmachi> Using the python-sdk I can create disks/add them to a VM, but I can't figure out how to "activate" them.  Can someone point me in the right direction? (I did see an 'activate' action, but I don't know how to use it for this)
14:54:21 <misc> dmachi: a irc meeting is going on, we will answer likely later :)
14:54:54 <dcaro> ok, next task: https://fedorahosted.org/ovirt/ticket/17
14:55:03 <dcaro> that is related to the action item se set up earlier
14:55:04 <dmachi> misc: thanks, sorry that I interrupted, didn't realize that was happening.
14:55:18 <dcaro> (the etherpad with the inventory)
14:56:23 <dcaro> misc: do you mind creating a trac for the etherpad list and add it as a dep here?
14:57:09 <misc> dcaro: yep, will do
14:57:18 <dcaro> thanks :)
14:57:56 <eedri_> dcaro, ideas for scaling gerrit?
14:57:56 <eedri_> dcaro, well, the mirrors has be public?
14:58:10 <dcaro> ooops
14:58:19 * eedri got disconnected
14:58:22 <misc> eedri: internal mirror on the new lab would be enough, I think
14:58:45 <dcaro> eedri: about the mirrors, I see no need for them to be private
14:59:58 <dcaro> https://fedorahosted.org/ovirt/ticket/136 - kiril is not around for this
15:00:47 <dcaro> https://fedorahosted.org/ovirt/ticket/111 <- this is mine, it's not that critical, but it has to be done, I'll change the priority though
15:01:21 <dcaro> https://fedorahosted.org/ovirt/ticket/131 <- this is blocked
15:01:30 <dcaro> (that was phx3 foreman proxy)
15:02:26 <dcaro> ok, now we have 3 jenkins jobs failures
15:03:46 <dcaro> we are having a few issues with packaging and dependencies lately, and we have started migrating to mock some of the jobs, anyone has a better alternative? (the main issue with mock is that it's rpm centric and only meant to build rpms, not run jobs)
15:04:30 * misc can you explain a bit more ?
15:04:42 <dcaro> sure
15:05:08 <dcaro> when running a job for a project, it usually needs some specific packages
15:05:27 <dcaro> right now as we mostly have packaginf jobs, it's usually requirements to build the projec
15:06:09 <dcaro> it might change by project version and by target distribution
15:06:25 <dmachi> dcaro: ansible
15:06:39 <dcaro> dmachi: how can ansible help there?
15:07:32 <dcaro> (we already use puppet btw, and afaik, they are quite similar)
15:07:45 <dmachi> well i don't know many details about what you are trying to do obviously, but since you can easily test for os/packages/etc its easy to customize a job that is being run and that job cn be running physically anywhere you can ssh to
15:08:03 <dmachi> yes similar to chef/puppet, but with no central server tying it all together.
15:08:28 <misc> so the idea would be to write a playbook ( ie, equivalent of a script ) for the job
15:08:38 <dmachi> yep
15:08:44 <misc> and make them run on any server, or even create a server for them to run
15:08:49 <dcaro> well, the issue is that we want to use the current resources as much as possible, and doing that binds a slave to a job, so it can't run anything else
15:08:51 <misc> and then remove the server or reuse it ?
15:09:16 <dmachi> yeah, well using the ovirt module,  you can include launching/setting up the node, run a job, shutdown node.
15:09:37 <dmachi> (which is in fact what I'm doing with our cluster) :)
15:10:27 <dcaro> that's interesting (as we tought of doing that also at some point)
15:11:14 <dcaro> so for that we'll need all the machines we have to be ovirt hosts or single executor slaves
15:11:33 <misc> to manage a big set of server, you need something like puppet (config management) and remote execution (fabric, ansible), for now, we only have puppet
15:13:12 <misc> dmachi: then the issue is how long does it take to provision a host on ovirt
15:13:13 <dmachi> misc: I'd never use fabric to manage a big set of servers, but ansible is certainly able to handle it (and if you need a centralized server, there are those available too), but anyway, it was just a lurker's observation :)
15:13:44 * misc also use ansible for his own server and for others projects
15:14:34 <dmachi> misc: my nodes start (from a template) pretty quickly, couple minutes.  I have a bunch of other config for networks and such , so it an take longer for some, but generally its pretty quick.  Not to say its perfect for this case, as I just butted into a conversation I don't know the details of :P
15:15:25 <dcaro> dmachi: how do you manage remote phisical servers? (we have quite a few servers not on the same networks/locations )
15:16:30 <dmachi> just occurs via ssh, my physical servers I define as "static" inventory, and then i have a dynamic inventory script that talks to ovirt api for dynamic inventory.  Everything occurs over ssh, and hte whole playbook setup is in git.
15:16:56 <dmachi> You can also tell ansible to proxy ssh connections via another host if you need to get behind a  firewall or something.
15:17:08 <misc> yep, it use the ssh binary, so you can be quite creative
15:17:38 <dcaro> I meant, you only run one job at the same time on each host?
15:18:11 <misc> you would, yes
15:18:21 <dcaro> most of our hosts are big machines on remote places, running ony one job at a time seems like waiting resources
15:18:23 <dcaro> *wasting
15:18:25 <misc> well, depend on how you do it
15:18:27 <dmachi> well if you ran two ansible playbooks in two different shells simultaneously you could probably acocomplish more than one, but by default its one at a time.
15:18:48 <misc> the fact is what if 1 job requires to install 1 rpm and the 2nd requires to remove it :)
15:18:55 <dcaro> exactly
15:19:50 <dcaro> that's the issue (running a versioned manifest without master can be done with puppet too, that's not new for ansible)
15:20:29 <dmachi> sure, you'd certainly have to plan, but is there any case where if you have two competing scripts like that on the same host you aren't going to have an issue?
15:21:03 <dcaro> if you use mock, for example, it creates a chroot for each process with it's own package layout and binary versions
15:21:16 <misc> what about docker ?
15:21:37 <dcaro> so you can run many parallel rpm builds with different distribution layouts and arches (well, only i686 can be mocked in x86_64 machines)
15:22:14 <dmachi> was gonna say docker too.  So using ansible (or puppet presumably) you can automate that too, doesn't have to be ovirt hosts.
15:22:41 <dcaro> we started checking docker, but it's quite new and it does not support systemd, needed for most of the non-package tests, so no big advantage over mock (well, yes, that it can build non-rpm packages)
15:22:42 <ewoud> but docker has no selinux support, so you're not going to be able to test everything
15:22:57 <dcaro> also yes
15:23:38 <dmachi> do all the tests require selinux support or just a subset of them?
15:24:14 <ewoud> I think unit tests generally don't care about them
15:24:28 <misc> but as long as you have 1, then you need a 2nd solution
15:24:29 <ewoud> but integration tests would provide a more realistic result
15:26:39 <dmachi> i just ask because you could use docker for the more frequent unit tests, and then vms for the integration tests which would perhaps take a little bit longer to setup and be isolated to a single job/vm butbe more realistic
15:27:37 <ewoud> I'd expect you'd end up with something similar yes
15:27:39 * misc has another meeting
15:27:42 <dcaro> mmm, that will be a lot nicer if we use get the phx hosts
15:28:07 <dcaro> okok, we can go closing, I'll start a thread on the list to discuss this
15:28:30 <dcaro> #action dcaro to start a thread on build env alternatives (mock, docker, ovirt spawnn vms)
15:28:47 <dcaro> but I guess that at least the vm part will have to wait to phx
15:29:15 <dcaro> anyone wants to rais an issue before finishing the meeting?
15:29:31 <dcaro> dmachi: thanks for the ideas! :)
15:29:54 <dmachi> dcaro: np
15:31:47 <dmachi> dcaro: I'm making an additional ansible module for managing some of the other object types (they only do vms right now).  But I can't figure out how to activate a disk that has been attached to a vm with the python sdk.  Can you point me in the right direction?
15:32:52 <dcaro> dmachi: not really, haven't played too much with it, you can try asking on devel list (or vdsm list)
15:33:27 <dcaro> ewoud: eedri misc ok, so I'll close the meeting for today
15:33:42 <dmachi> dcaro: ok, thanks
15:33:46 <eedri> dcaro, ok
15:33:49 <dcaro> #endmeeting