14:24:51 <ewoud> #startmeeting infra weekly
14:24:51 <ovirtbot> Meeting started Mon Jun  3 14:24:51 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:24:51 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:25:01 <ewoud> #chair eedri knesenko dcaro obasan
14:25:01 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan
14:25:23 <ewoud> a bit later than planned because I had to help a colleague
14:25:43 <knesenko> ewoud: ok
14:26:55 <ewoud> #topic hosting
14:27:35 <ewoud> #info rackspace01.ovirt.org installed
14:27:38 <dcaro> I'm around!
14:27:43 <eedri> dcaro, hey
14:27:44 <ewoud> dcaro: rackspace02 as well?
14:27:48 <dcaro> yep
14:27:59 <dcaro> they are not connected to foreman yet, not have puppet
14:28:13 <dcaro> but the base os is installed and all the admins have access
14:28:25 <dcaro> (to keep setting it up)
14:28:26 <ewoud> #undo
14:28:26 <ovirtbot> Removing item from minutes: <MeetBot.items.Info object at 0xa3b8cac>
14:28:45 <ewoud> #info rackspace0{1,2}.ovirt.org installed, not connected to foreman/puppet yet
14:29:08 <ewoud> dcaro: any specific problem with puppet or just lack of time?
14:29:56 <dcaro> ewoud: some problems with the ruby version
14:30:13 <dcaro> (and lack of time to solve it)
14:30:34 <ewoud> dcaro: ok
14:31:08 <ewoud> dcaro: is ovirt installed on them?
14:31:40 <dcaro> not yet asfaik, knesenko: ^
14:31:46 <eedri> ewoud, not yet, we want to rethink the infra deployment before installing it
14:31:55 <ewoud> eedri: ok
14:32:02 <eedri> ewoud, might not be optimal to do it as we planned, knesenko can elaborate
14:32:45 <ewoud> knesenko: I heard you could elaborate why we want to rethink the infra deployment
14:32:48 <eedri> knesenko, can you share some insight on your plan for the infra of ovort-engine
14:33:06 <eedri> knesenko has previous experiene with ovirt-engine installations
14:33:11 <knesenko> yes ... It make no sense to install alinone setups
14:33:40 <knesenko> so we need to define one engine that will manage our hosts ... for now we have 4 bare metal .
14:34:05 <knesenko> so what we need is - one amazon VM - for engine and it will manage our hosts.
14:34:20 <knesenko> this is the correct architecture ...
14:34:37 <knesenko> but for this we will need storage ... some external storage ... NFS or iscsi
14:34:46 <eedri> knesenko, we'll need to check what is the overhead (if exists) to manage multiple hosts access WAN/internet
14:35:28 <knesenko> the only problem that we need a storage domains that will be located near the hosts ...
14:36:07 <eedri> dneary, is it possible to get storage service from rackspace?
14:36:32 <dneary> eedri, I hate to say "I don't know", I haven't dealt with RAX at all
14:36:41 <eedri> dneary, will make our mgmt of infra much simpler and resilient than using one host as storage server
14:37:08 <eedri> dneary, ok. we'll have to query karsten then
14:37:22 <dneary> eedri, Would Akamai be an option?
14:37:46 <eedri> dneary, i think any storage service that can provide iscsi/nfs and open to the hosts will do
14:37:50 <ewoud> brb
14:38:17 <eedri> knesenko, can you send a desc of the proposed infra layout to the list? including request for the storage service
14:38:50 <knesenko> eedri: of course ... need to prepare it first
14:38:54 <eedri> knesenko, np
14:39:16 <eedri> #info knesenko to send desc of ovirt-engine layout to infra list
14:39:26 <BlairQ> hi, im having dependency issues with some packages in centos 6,
14:39:37 <BlairQ> vdsm-gluster-4.10.3-10.el6.noarch seems to be outdated
14:39:51 <eedri> BlairQ, we're in the middle of ovirt meeting, can you wait till it's over please?
14:40:04 <BlairQ> off course, im sorry
14:40:11 <eedri> BlairQ, np
14:40:15 <knesenko> BlairQ: its ok :)
14:40:15 <eedri> ewoud, continue?
14:40:41 <knesenko> he is not here ...
14:40:44 <knesenko> lets go ...
14:40:51 <eedri> ok, what else do we have on hosting
14:41:19 <knesenko> obasan: incinga ?
14:41:34 <obasan> knesenko, I have a puppet class to manage nrpe
14:41:46 <obasan> knesenko, but as dcaro mentioned it does require tweaking
14:41:53 <obasan> knesenko, has to be authenticated
14:42:08 <obasan> https://github.com/Ohadbasan/puppet-nrpe
14:42:32 <knesenko> obasan: authenticated ?
14:42:43 * ewoud is back
14:42:49 <obasan> knesenko, ssl between the icinga server and the monitored hosts
14:42:57 <obasan> nrpe by default is not secured
14:42:59 <eedri> ewoud, i think we're done with hosting, maybe we should change topic to puppet?
14:43:19 <ewoud> eedri: feel free to do it yourself next time when I'm not around :)
14:43:22 <ewoud> #topic puppet
14:43:52 <eedri> ewoud, i was about too :)
14:44:21 <ewoud> obasan: I did add the hosts manually to icinga to get a good feeling with it
14:44:35 <ewoud> but I'd really like to automate it
14:44:57 <obasan> ewoud, https://github.com/Ohadbasan/puppet-nrpe
14:45:29 <obasan> ewoud, auto adding of hosts can also be done
14:45:37 <ewoud> obasan: I was looking at it, will do a pull request with what I think is some beautification
14:45:51 <ewoud> also, I don't like ensure => latest in puppet
14:46:08 <obasan> ewoud, feel free to commit changes if you wish :)
14:46:21 <ewoud> obasan: more parameters would be better :)
14:46:28 <xoritor> is anyone else having a problem doing an update today using ovirt-stable?
14:46:49 <xoritor> i am getting i686 packages where I had none before (and they are causing a failure)
14:46:56 <obasan> ewoud, nrpe.cfg is pretty static...
14:47:28 <xoritor> Error: Package: vdsm-4.10.3-10.fc18.i686 (ovirt-stable)
14:47:34 <xoritor> yes .... again...  vdsm
14:48:04 <xoritor> but it seems the x86_64 version is trying to (for some reason) force the install of i686 multilib stuff
14:48:23 <xoritor> i don't have any i686 stuff installed
14:49:04 <xoritor> and --skip-broken does not fix it
14:49:20 <xoritor> nor does --setopt=protected_multilib=false
14:49:30 <xoritor> Protected multilib versions: vdsm-python-4.10.3-16.fc18.x86_64 != vdsm-python-4.10.3-10.fc18.x86_64
14:50:25 <xoritor> again... ihave nothing i686 installed and it is trying to install i686 stuff
14:51:00 <eedri> xoritor, can you please post these questions in 30 min? we're currently in an ovirt meeing
14:51:10 <xoritor> sure
14:51:40 <eedri> ewoud, we're still waiting for approval for puppet repo on gerrit.ovirt.org right?
14:51:42 <knesenko> BlairQ: xoritor I will try to help you both after the meeting
14:52:12 * quaid doesn't know off-hand of storage from RackSpace
14:52:12 <xoritor> thanks... sorry didn't read the /topic ... sorry
14:52:20 <ewoud> eedri: yes
14:52:27 <ewoud> obasan: at $employer we use https://github.com/ekohl/nrpe_basic
14:52:41 <ewoud> which is rather basic as well, but does a bit more
14:53:25 <eedri> quaid, can we open a ticket in rackspace to get info on that?
14:53:36 <obasan> ewoud, seems to be ok
14:54:02 <quaid> eedri: sure, any of us can ask, but it's probably ultimately something to talk with Jim Strong about
14:54:06 <obasan> ewoud, it just supports all sorts of linux platforms.
14:54:56 <eedri> quaid, ok, let's wait till knesenko will send his proposal for the ovirt-engine infra layout and then send our query to jim (with all the options/alternatives)
14:55:27 <ewoud> sorry, I really wish I had more time for the meeting today
14:55:32 <ewoud> #chair quaid
14:55:32 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan quaid
14:55:45 <knesenko> ewoud: np
14:55:52 <ewoud> #info puppet repo on gerrit requested
14:56:17 <ewoud> how do we want to push to foreman? using gerrit replication or a jenkins job?
14:56:48 <eedri> dcaro, i think you have experience with that, right?
14:57:14 <ewoud> does gerrit have proper logging of the replication?
14:58:06 <dcaro> right now we are doing it with a gerrit plugin @redhat, but a jenkins job should work also ok
14:58:07 <ewoud> I know jenkins can log easily
14:58:41 <ewoud> but I'd be fine with gerrit as well if we can get debug output when it fails
14:58:43 <ewoud> or a notification
14:58:47 <dcaro> *gerrit hok sorry
15:00:04 <dcaro> maybe we can link it to an icinga passive check, and update it from the hook
15:00:47 <eedri> ok, let's move on the topics and continue this on the list?
15:00:47 <obasan> why is a jenkins job with gerrit trigger a bad idea?
15:00:51 <obasan> it seems like the easiest solution
15:01:00 <eedri> we're out of time for the meeting and still got a lot to cover
15:01:05 <obasan> once you commit to puppet repo > jenkins job runs :)
15:01:10 <obasan> eedri, ok
15:01:49 <eedri> #action continue puppet repo push to foreman on infra list
15:02:06 <eedri> #topic jenkins
15:02:13 <ewoud> I've had a bit of a monologue on the puppet repository layout, I'd appreciate if you could reply to it btw
15:02:20 <eedri> ewoud, +1
15:02:42 <eedri> ewoud, sorry i didn't have time to go over it, been very busy in $employer lately
15:02:55 <dcaro> shame on me, I have it pending since you sent it :S, sorry+
15:03:19 <eedri> i added a new job in jenkins for running find_bugs per patch on ovirt-engine
15:03:41 <obasan> the tab is still open on my browser :)
15:03:48 <eedri> i know this is not optimal in current infra and we wanted to wait till new vms are alive
15:03:50 <obasan> will go over it asap
15:03:54 <knesenko> obasan: so reply to the email !
15:04:09 <eedri> but lately there has been too many failures on the jenkins on compilation and findbugs..
15:04:30 <eedri> it seems jenkins is coping with the load for now, but it might be it's overloading the gerrit instance and causing it to fail occasionaly
15:04:53 <eedri> itamar is planning to upgrade the gerrit instance to 2.5, hopefully it will solve the issue
15:05:22 <eedri> i will track the status of jenkins for the next few days, if this erratic behavior continues, i'll disable the job for now
15:05:51 * eedri finished
15:06:35 <dcaro> I propose adding someone more to be able to manage gerrit (with admin rights to the instance), so Itamar does not have to be on it fridays too
15:06:38 <eedri> another jenkins jobs in progress is testing engine-setup + engine-upgrade, which is failing on real issues
15:07:01 <ewoud> dcaro: +1
15:07:01 <eedri> dcaro, +1
15:07:05 <obasan> dcaro, +1
15:07:33 <knesenko> dcaro: +1
15:08:17 <eedri> #action add an additonal administrator for gerrit.ovirt.org to add redundancy and backup for weekends
15:08:47 <eedri> any other jenkins related stuff?
15:09:10 <eedri> quaid, i hope you're around cause other than changing topic i don't know how to manage the bot :)
15:09:53 <quaid> eedri: I'm here
15:09:59 <knesenko> finished backup for jenkins
15:10:04 <eedri> right
15:10:07 <eedri> thanks knesenko
15:10:27 <quaid> #agreed Add an additional admin for gerrit
15:10:29 <quaid> :)
15:10:35 * quaid tries out some other things
15:10:46 <eedri> #info jenkins configuration files are now backed up to alterway02 server
15:11:05 <eedri> nightly backus, kept for 30 days
15:11:25 <eedri> we should consider a full backup of jenkins homedir, around once a month...
15:11:35 <eedri> prefferably to a remote location (rackspace?)
15:11:56 <knesenko> maybe we will have some ext. storage
15:12:03 <knesenko> lets wait with a full backup for now
15:12:06 <eedri> not just jenkins, other ciritical servers as well
15:13:00 <eedri> #action re-discuss full backup of infra services once we have ext. storage
15:13:21 <eedri> we can now yum upgrade alterway01 (jenkins.ovirt.org)
15:14:13 <eedri> #info added new job to run quick rpm make for ovirt-engine on fedora and el (to verify packging isn't broken)
15:14:18 <eedri> ok, moving on
15:14:51 <eedri> do we have time to review trac tickets
15:14:52 <eedri> ?
15:15:16 <knesenko> eedri: hmmm ... have no idea .
15:15:24 <eedri> ewoud, quaid dneary ?
15:16:04 <eedri> i think we're overdue the meeting time anyhow, we'll do it next time
15:16:05 <knesenko> seems like 50%+- are assigned to a specific users
15:16:52 <eedri> ewoud, ping
15:17:01 <eedri> ewoud, do you want to add action to reschdule meeting time?
15:18:00 <knesenko> Guys , I would like to add an oVirt company proifile to linkedin same as Fedora - http://www.linkedin.com/company/32178?trk=vsrp_companies_res_name&trkInfo=VSRPsearchId%3A528744561370272644773%2CVSRPtargetId%3A32178%2CVSRPcmpt%3Aprimary
15:18:32 <knesenko> but for this purpose I need .*@ovirt.org email account . is it possible to open an account for linkedin or something ?
15:18:35 <knesenko> quaid: ^^
15:18:39 <eedri> knesenko, i suggest bringing it up the list
15:18:55 <knesenko> eedri: I will
15:19:07 <obasan> knesenko, good idea
15:19:32 <eedri> #action reschdule infra meeting time after reviewing doodle results
15:19:45 <eedri> please respond to ewoud email on changing meeting time
15:20:05 <knesenko> eedri: ok
15:20:11 <knesenko> end meeting ?
15:20:19 <eedri> i think we're ready to close the meeting unless someone has something to say?
15:20:35 <eedri> going once
15:20:41 <eedri> twice...
15:20:50 <dcaro> sold!
15:20:51 <eedri> quaid, .. can you end the meeting?
15:20:54 <obasan> I object!
15:20:55 <obasan> :)
15:21:32 <eedri> #action send request to add ovirt as project to linkedin
15:21:32 <quaid> knesenko: we can create an alias or a marketing list or something, too
15:21:45 * quaid trie the meeting end trick
15:21:46 <obasan> quaid, +1
15:21:53 <quaid> #endmeeting