14:08:09 <dcaro> #startmeeting oVirt Infra
14:08:09 <ovirtbot> Meeting started Mon Jul  7 14:08:09 2014 UTC.  The chair is dcaro. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:08:09 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:08:11 <dneary> eedri, !
14:08:22 <dcaro> #chair eedri obasan knesenko ewoud
14:08:22 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan
14:08:29 <dneary> eedri, bkp is also about I think
14:08:42 <dcaro> #chair dneary
14:08:42 <ovirtbot> Current chairs: dcaro dneary eedri ewoud knesenko obasan
14:08:55 * dneary will be lurking
14:09:01 * rbarry also lurking
14:09:12 <eedri> YamakasY, maybe also want to listen ;)
14:09:14 <eedri> dcaro, ok..
14:09:43 <dcaro> #chair rbarry
14:09:43 <ovirtbot> Current chairs: dcaro dneary eedri ewoud knesenko obasan rbarry
14:09:55 <dcaro> #topic Hosting
14:10:25 <dcaro> I have good news there, finally the storage servers are functional
14:10:38 <dcaro> I'll install one of the hosts and the engine this week to start the tests
14:10:55 <dcaro> #info storage server at PHX functional
14:11:28 <dcaro> #action install one host with hosted engine and test thephx lab setup
14:11:36 <eedri> dcaro, +1.. great
14:11:45 <eedri> dcaro, so we're expecting something like 13TB mirrored right?
14:11:56 <dcaro> eedri: yep
14:12:12 <eedri> dcaro, did we manage to get them to bond the network between storage servers?
14:12:53 <dcaro> eedri: they have a floating ip that pacemaker+cman/crm manage if that's what you mean
14:13:27 <dcaro> eedri: but right now they are using only one interface, so no bonding can be done yet (lacking the physical cables)
14:13:51 <eedri> dcaro, do we have some sort of tracking/tickets we can use?
14:14:02 <eedri> dcaro, for stuff we need from therm
14:14:23 <dcaro> eedri: just the email thread and the internal docs
14:14:38 <eedri> dcaro, ok, worth checking if we can open tickets somehow, will be easier to follow
14:14:47 <dcaro> eedri: ok
14:15:15 <dcaro> eedri: misc you upgraded the alterway02 host, ming saying a few words about hte issues/solutions?
14:15:28 <misc> well, yes
14:15:42 <misc> first, the upgrade didn't exactly wrok fine from a rpm point of view
14:15:48 <misc> I had to use yum-shell
14:15:48 <eedri> misc, might be one of the only one who ever upgraded all in one i think from 3.2 to 3.4 :)
14:16:06 <eedri> misc, but maybe i'm wrong..
14:16:17 <misc> didn't took note on the problem, and since ewoud told we were using non official RPM, and likely non supported jump, I didn't looked more
14:16:21 <misc> ( and just fixed )
14:16:36 <misc> then ewoud made the upgrade, using engine-setup
14:16:53 <misc> and now, I am trying to figure why the iso domain is down ( or rather, how to make it up )
14:16:58 <eedri> misc, yea, that's why we're aiming at more formal release and installation for phx2 - hosted-engine rather than all in one
14:17:26 <misc> eedri: I tought we were using hosted-engine already ?
14:17:33 <misc> I guess I need to RTFM a bit more :)
14:18:04 <misc> anyway, so it seems to work, besides that issue with storage domain, and I would appreciate that someone who have more than 3 days of experience with ovirt could take a look :)
14:19:15 <ewoud> misc: no, currently it's all-in-one installs
14:19:22 <dcaro> #info upgraded alterway02, but failing ot get iso domain up
14:19:23 <eedri> misc, no, for alterway and rackspace it's all-in-one
14:19:33 <eedri> misc, hosted-engine was only introduced in 3.4 formally
14:19:36 <dcaro> #action brin up the iso domain for alterway02
14:20:03 <ewoud> I must admit fixing it was also a bit hacky by manually changing the database settings file
14:20:03 <dcaro> misc: everything else is working fine?¿
14:20:12 <misc> dcaro: servers are running
14:20:23 <dcaro> misc: that's not what I asked ;)
14:20:25 <ewoud> we still need to schedule a reboot
14:20:27 <mlipchuk> ojorge: Hi, I was disconnected for a few minutes, how is the upgrade going?
14:20:28 <eedri> ewoud, we should take notice of that for next upgrade
14:20:40 <ewoud> it is possible that after a reboot servers won't come up
14:20:51 <eedri> ewoud, plan is to migrate all alterway eventually to a production DC on phx2
14:20:57 <eedri> ewoud, and use alterways as hypervisors
14:21:03 <misc> dcaro: well, I think the regular domain is also down, whcih is weird, i think stuff are in a incosnistant state, but I do not know how much
14:21:18 <ewoud> IMHO it's best practice to reboot after a big upgrade and it's needed for a new kernel anyway
14:21:23 <misc> yeah
14:21:32 <misc> but I would first make sure it work
14:21:40 <ewoud> agreed
14:21:46 <YamakasY> eedri: wazzup ?
14:21:53 <eedri> YamakasY, hey, we're in a meeting
14:21:53 <ewoud> how scared are we that we can't reach it at all after a reboot?
14:21:54 <dcaro> misc: ok, will have to take a deep look to see if we can avoid losses at reboot
14:22:05 <ewoud> i.e. do we need to have someone from alterway on standby?
14:22:06 <YamakasY> eedri: mhh... I'm not :D
14:22:07 <YamakasY> :P
14:22:11 <eedri> ewoud, best to sync it with alteray contact (kevin?)
14:22:22 <eedri> ewoud, so we'll have local people there to help
14:22:35 <ewoud> dneary: ^
14:22:40 <eedri> ewoud, i don't think we have console access there
14:22:49 <ewoud> eedri: I don't think so either
14:23:15 <dneary> ewoud, eedri, misc: I think Kévin left AlterWay
14:23:28 <eedri> dneary, who is our contact there now?
14:23:38 <dneary> ewoud, eedri, misc: Hervé Leclerc is the best contact person there now - he will tell us who else can help us if needed
14:23:52 <eedri> dneary, you have his email?
14:24:04 <eedri> dneary, best to update our file with it
14:24:09 <dneary> eedri, Yes - I believe he's also on infra@
14:24:49 <dneary> eedri, sent directly just to avoid it being in the public log
14:26:41 <eedri> dneary, thanks
14:28:13 <dcaro> dneary: thanks!
14:28:26 <dneary> np!
14:28:56 <dcaro> ok, so the action course should be -> Take a deep look -> contact hlecrerc -> reboot and cross fingers
14:28:58 <dcaro> ?
14:29:48 <eedri> dcaro, yea..
14:30:07 <eedri> dcaro, misc ewoud i updated the file with hlecrerc email
14:30:22 <eedri> misc, so if you're planning a reboot, might worth shooting him an email
14:30:33 <dcaro> #action After checking alterway02, program reboot with alterway assistance
14:31:50 <dcaro> ok, anythin else about hosting?
14:33:31 <dcaro> moving then
14:33:35 <dcaro> #topic Jenkins
14:33:55 <dcaro> #info We have ppc slaves available!
14:34:10 <dcaro> Still need to set them up and configure jobs to use them though
14:34:26 <dcaro> (set them up meaning install puppet if able and all the slave deps)
14:34:49 <dcaro> knesenko: want to talk a bit about the copr jobs?
14:35:46 <eedri> dcaro, +1
14:36:01 <eedri> ewoud, you know if we should expect any issues adding puppet + foreman to it?
14:36:08 <eedri> ewoud, i.e power pc 64 slave
14:36:43 <ewoud> eedri: as long as packages are available with the same name, I wouldn't expect any
14:37:24 <knesenko> dcaro: no prblem
14:37:36 <knesenko> dcaro: just let me know when
14:37:38 <knesenko> :)
14:37:49 <kobi> anyone familiar with this error: ***L:ERROR Internal error: type object 'Stages' has no attribute 'DB_CONNECTION_SETUP'
14:37:59 <kobi> and how to work around it?
14:38:23 <eedri> kobi, we're in the middle of infra meeting
14:39:06 <eedri> dcaro, also, work has started on 3.5 rpm jobs, via yaml
14:39:31 <dcaro> yep, will try to get generic easy to use templates so it0s easy to add new jobs
14:39:48 <dcaro> and create a howto for it
14:43:11 <eedri> dcaro, +1
14:44:06 <eedri> dcaro, anything else on jenkins?
14:44:16 <eedri> dcaro, i think there is also el7 slave right?
14:44:24 <dcaro> true!
14:44:56 <dcaro> #info new el7 physical host jenkins-slave-host06.ovirt.org
14:45:12 <dcaro> we can start migrating and testing jobs there
14:45:42 <eedri> dcaro, yea
14:46:03 <eedri> knesenko, want to talk about copr ?
14:46:22 <knesenko> eedri: sure
14:46:35 <knesenko> eedri: so we have a new jobs for building pkgs on copr
14:46:41 <knesenko> what is copr ?
14:46:58 <knesenko> copr is open source build system related to Fedora folks
14:47:21 <knesenko> here we have a list of projects that we can build on copr build system
14:47:22 <knesenko> http://copr.fedoraproject.org/coprs/ovirt/
14:47:51 <knesenko> here we have jobs to trigger the build on copr build system - http://jenkins.ovirt.org/view/copr/
14:48:11 <knesenko> this will reduce the load on our slaves when we are planning to build for official releases ...
14:49:08 <knesenko> next step is to add more jobs and implement a script to collect all builds from all projects and compose a single repo
14:49:16 <knesenko> eedri: dcaro that's all from my side I think
14:49:23 <ewoud> they can also serve as mirror AFAIK
14:49:32 <ewoud> do we also plan to use them instead of resources.ovirt.org?
14:49:40 <eedri> ewoud, i don't think we can..
14:49:42 <knesenko> ewoud: I am not sure we can
14:49:48 <eedri> ewoud, they have a retention policy
14:49:54 <eedri> ewoud, deleting old builds for starts
14:50:05 <eedri> ewoud, and they don't support non centos/fedora builds
14:50:11 <knesenko> ewoud: they provide a repo per project ...
14:50:17 <knesenko> means - vdsm:repo, engine:repo
14:50:20 <knesenko> and so on
14:51:26 <eedri> #info new jobs were added to support official ovirt builds via copr system
14:51:29 <ewoud> ok
14:51:36 <eedri> #link http://jenkins.ovirt.org/view/copr/
14:51:58 <eedri> dcaro, lets move to review some critical tickets?
14:52:03 <dcaro> eedri: sure
14:52:08 <ewoud> so they're not meant to be used by end users
14:52:13 <dcaro> #topic Tickets
14:52:23 <eedri> dcaro, btw, i prepared a query for tickets solved in june
14:52:24 <eedri> dcaro,     http://goo.gl/ddyIvl
14:52:48 <dcaro> nice :)
14:53:11 <dcaro> #link https://fedorahosted.org/ovirt/query?status=closed&changetime=4+weeks..1+day&order=priority&report=9&col=id&col=summary&col=changetime&col=status&col=type&col=priority&col=milestone&col=component
14:53:19 <dcaro> wow... xd
14:53:31 <eedri> hehe
14:53:54 <dcaro> https://fedorahosted.org/ovirt/ticket/188
14:54:00 <dcaro> ovirt-appliance can not be build because of unreachable mirror
14:54:25 <eedri> fabiand, rbarry here?
14:54:36 <eedri> dcaro, seems like a new dep is needed on the slaves?
14:54:38 <fabiand> here
14:54:46 <fabiand> The unreachable mirror problem is fixed, thanks to dcaro!
14:54:52 <eedri> \0/
14:55:06 <dcaro> fabiand: you sent a patch for the deps right? Was it merged?
14:55:16 <dcaro> http://gerrit.ovirt.org/#/c/29584/
14:55:19 <fabiand> I am not sure yet ..
14:55:19 <dcaro> no it's not :)
14:55:25 <fabiand> dcaro, but that#s just a "drop on the hot stsone".
14:55:32 <dcaro> I'll review
14:55:32 <fabiand> There are other issues which prevent building the appliance ..
14:55:56 <dcaro> fabiand: so only f19 slaves will be used for that job?
14:56:03 <fabiand> dcaro, yes
14:56:12 <dcaro> fabiand: okok
14:56:17 <fabiand> dcaro, I also just pinned it to one specific host, to make it more reproducable.
14:56:53 <dcaro> fabiand: take into account that it will get blocked when loaded and will fail when that slave is not there anymore, I recommend pinning it only for the tests or when debugging
14:57:05 <rbarry> eedri: Here
14:57:14 <fabiand> dcaro, I fully agree! Once it's stable I'm happy to pin it to the f19 tag
14:57:20 <dcaro> okok :)
14:59:19 <eedri> dcaro, fabiand next one is also from you :)
14:59:25 * fabiand hides :)
14:59:34 <eedri> fabiand, https://fedorahosted.org/ovirt/ticket/177
14:59:43 <eedri> update ovirt-node package in jenkins servers
14:59:49 <fabiand> Oh that one.
14:59:54 <eedri> not sure if there is still action item there
14:59:56 <fabiand> I honestly did not understand that one ..
15:00:20 <dcaro> hehehe
15:00:22 <eedri> dougsland, ?
15:00:28 <eedri> dougsland, can you elaborate on that ticket?
15:00:40 <eedri> dougsland, is it still needed
15:02:15 <dougsland> eedri, closed
15:02:20 <eedri> dougsland, great
15:02:31 <eedri> fabiand, 3rd one is the charm! (or icecream)
15:02:40 <eedri> https://fedorahosted.org/ovirt/ticket/136
15:02:47 <eedri> you're keeping the infra team busy :)
15:04:47 <dcaro> I think we are out of time
15:04:56 <eedri> dougsland, can you help with https://fedorahosted.org/ovirt/ticket/123?
15:05:01 <dcaro> any issue we must discuss before losing?
15:05:15 <eedri> dcaro, let's have a quick look on the urgent tickets
15:05:16 <dcaro> hahaha, s/losing/closing/
15:05:31 <eedri> dcaro, hehe
15:05:38 <eedri> dcaro, we always win! never lose
15:05:56 <dougsland> eedri, I think so.
15:06:04 <eedri> dcaro, unless there is some needinfo on those critical tickets, i guess we can close and tackle it offline
15:06:09 <eedri> dougsland, so you can provide el7 builds?
15:06:14 <eedri> dougsland, for vdsm deps..
15:06:23 <dougsland> eedri, is it downstream or upstream ?
15:06:27 <eedri> dougsland, ovirt
15:06:34 <eedri> dougsland, using copr or koji i guess
15:06:51 <dcaro> ok, so closing the meeting
15:06:56 <eedri> dcaro, +1
15:07:08 <dcaro> #endmeeting