15:08:59 <knesenko> #startmeeting oVirt Infra
15:08:59 <ovirtbot> Meeting started Mon Dec 16 15:08:59 2013 UTC.  The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:08:59 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:09:07 <knesenko> #chair ewoud eedri dcaro
15:09:07 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko
15:09:11 * orc_orc_ is here
15:09:14 <obasan> #chair obasan
15:09:17 <knesenko> #chair orc_orc
15:09:17 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko orc_orc
15:09:26 <knesenko> #chair obasan
15:09:26 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan orc_orc
15:10:23 <knesenko> #topic Hosting
15:10:28 <knesenko> Hello
15:10:54 <knesenko> lets start ...
15:11:08 <obasan> knesenko, hello
15:11:13 <knesenko> #chair eedri_
15:11:13 <ovirtbot> Current chairs: dcaro eedri eedri_ ewoud knesenko obasan orc_orc
15:11:24 <knesenko> some news regarding the rackspace03
15:11:27 <knesenko> NO NEWS !
15:11:36 <knesenko> they didn't fixed the problem
15:11:46 <orc_orc_> hunh
15:11:56 <orc_orc_> but you found the workaround, no?
15:12:05 <orc_orc_> or SvenKieske ... I fergit
15:12:17 <knesenko> orc_orc: no I am .... I can't ssh from rackspace01 to rackspace03
15:12:26 <knesenko> they have routing issues
15:12:30 <knesenko> not on our side ...
15:12:41 <orc_orc_> ;(
15:12:46 <eedri_> knesenko, i'll escalate this with out rax contact
15:13:10 <knesenko> eedri_: ok
15:13:29 <dcaro> knesenko: no, rackspace problem is 'solved', meaning now I can connect using the console
15:13:31 <knesenko> also there is a another ticket to update centos slaves to 6.5
15:13:46 <knesenko> dcaro: there is a another problem :)
15:13:56 * ewoud here
15:14:09 <knesenko> dcaro: no ssh between rackspace01 and 03
15:14:14 <knesenko> #chair ewoud
15:14:14 <ovirtbot> Current chairs: dcaro eedri eedri_ ewoud knesenko obasan orc_orc
15:15:07 <knesenko> so we are still blocked on its installation
15:15:21 <dcaro> knesenko: can't we work around that?
15:16:07 <knesenko> dcaro: how ?
15:16:21 <dcaro> knesenko: maybe creating a tunnel or something
15:16:32 <dcaro> knesenko: at least until it's fixed
15:16:33 <knesenko> dcaro: why should we ?!
15:16:48 <dcaro> knesenko: to get work done
15:16:50 <knesenko> dcaro: :) they just need to fix it ...
15:16:52 <orc_orc_> dcaro: a tunnel would need a woprking endpoint near '03 to reach the ILO    and that is the problem
15:16:56 <knesenko> dcaro: I don't know ...
15:18:09 <knesenko> ok what about Centos slaves upgrades ?
15:18:25 <knesenko> updating the repo and run yum update should be enough ?
15:18:26 <dcaro> orc_orc_: we know that we can connect through jenkins.ovirt.org to both machines, maybe we can tunnel through there (if we are not going to send big files there should be no problem)
15:18:53 <knesenko> dcaro: orc_orc I don't want to make ugly workarounds ...
15:18:55 <ewoud> knesenko: I offered to do it, but haven't gotten around to it
15:19:03 <knesenko> dcaro: orc_orc they should fix it ...
15:19:09 <knesenko> ewoud: ok ...
15:19:21 <knesenko> ewoud: I hope I will have some time to do it this week .
15:19:25 <ewoud> knesenko: I can do it this evening
15:19:31 <knesenko> eedri_: how many Centos slaves do we have ?
15:19:34 <knesenko> ewoud: +1
15:19:35 <knesenko> !
15:19:43 <ewoud> knesenko: 2 I think
15:20:03 <eedri_> ewoud, yea 2
15:20:09 <knesenko> ewoud: ok good
15:20:18 <ewoud> https://foreman.ovirt.org/hosts?utf8=%E2%9C%93&search=+hostgroup+%3D++%22jenkins+slave%22+and+os+%3D++CentOS shows 2 results
15:20:18 <knesenko> ewoud: thanks for the effort
15:20:43 <ewoud> just a matter of marking the slave as offline, wait till there are no jobs, yum update, reboot and mark it as online, right?
15:20:45 <eedri_> ewoud, there is also a f18 slave i think we're using, so we should reprovision it as f20 or centos / ubuntoi
15:20:53 <eedri_> ewoud, yea
15:20:58 <ewoud> and then the next
15:21:54 <bigclouds_> mskrivanek:could you tell me in which file and which function it starts vm from a live snapshot?
15:22:18 <ewoud> #action ewoud update centos slaves to 6.5 this evening
15:22:37 <ewoud> what about notifications / announcements?
15:23:01 <knesenko> #info rackspace03 is still blocked. Rackspace should fix their routing problem
15:23:16 <mskrivanek> bigclouds_: well, preview would do that
15:24:09 <knesenko> ewoud: which notifications ?
15:24:22 <knesenko> ewoud: I am sorry maybe I am missing something
15:24:57 <bigclouds_> mskrivanek:i mean vdsm code, i just want to know how to assign vm state file when start it  in xml?
15:25:50 <ewoud> knesenko: about updating the slaves
15:26:18 <knesenko> ewoud: I think that sending email to infra should be enough
15:26:25 <ewoud> knesenko: ok
15:26:34 <knesenko> ok what next here ?
15:27:09 <knesenko> #topic Foreman and Puppet
15:27:20 <dneary> Finally on the network
15:27:42 <knesenko> dneary: want to join the meeting ?
15:27:55 <dneary> knesenko, I thought I just did ;-)
15:28:00 <knesenko> dcaro: ewoud any updates there ? :)
15:28:04 <knesenko> #chair dneary
15:28:04 <ovirtbot> Current chairs: dcaro dneary eedri eedri_ ewoud knesenko obasan orc_orc
15:28:11 <knesenko> dneary: ))
15:29:08 <ewoud> knesenko: we set up a jenkins job to do updates of the puppet repo on foreman.ovirt.org
15:29:19 <dcaro> knesenko: yep, we pushed the puppet manifest testing too
15:29:28 <dcaro> knesenko: not finished though
15:29:34 <ewoud> http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/system_update_puppet/
15:29:58 <ewoud> as you can see on http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/system_update_puppet/3/console there's decent logging
15:30:00 <knesenko> #info dcaro and ewoud create a new job to update puppet - http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/system_update_puppet/
15:30:23 <ewoud> haven't tested if the gerrit trigger on merge now works though, but it should
15:30:31 <knesenko> dcaro: where can I see the puppet manifest testing job ?
15:30:38 <knesenko> ewoud: ok
15:30:40 <ewoud> all reviews against master should be resubmitted against production
15:30:46 <dcaro> knesenko: http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/infra_check_puppet_manifests/
15:30:59 <dcaro> knesenko: it's a work in progress though
15:31:03 <ewoud> http://gerrit.ovirt.org/#/q/status:open+project:infra-puppet+branch:master,n,z
15:31:29 <knesenko> #info ewoud and dcaro are working on http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/infra_check_puppet_manifests/
15:31:39 <knesenko> dcaro: ewoud great progress guys ! thanks
15:32:10 <ewoud> one thing I'd like to add is monitoring
15:32:48 <ewoud> at $work we use https://github.com/jrottenberg/check_puppet and I submitted a check that uses the foreman dashboard API
15:33:39 <ewoud> orc_orc_: mind helping me setting that up?
15:34:02 <orc_orc_> ewoud: I am find w this, later this week -- have to get a report out on the rhel 7 candidate
15:34:09 <orc_orc_> Thurdsay?
15:34:20 <ewoud> I think I'll manage, but in case I don't that sounds fine
15:34:28 <orc_orc_> ewoud: ok
15:36:03 <knesenko> great
15:36:08 <knesenko> anything else here ?
15:36:10 <dcaro> ewoud: it's fairly simple to get info from foreman using python-foreman ;)
15:37:29 <ewoud> dcaro: I did see that and haven't had the time to try it, but certainly looked promising
15:38:09 <ewoud> dcaro: but that check script was made to work on EL5 without additional packages installed
15:38:10 <dcaro> ewoud: we use it at work for our scripts
15:38:25 <ewoud> dcaro: otherwise I'd at least have used requests
15:38:48 <dcaro> ewoud: you should update your nagios machine os :)
15:39:00 <eedri_> knesenko, i think we have issues with minidell running puppet
15:39:31 <ewoud> dcaro: yes, that's a long running issue
15:39:38 <knesenko> eedri_: which one of them :?
15:40:00 <ewoud> dcaro: I won't bore you with the details, but lots of custom scripts, unpackaged nagios etc
15:40:10 * eedri_ checking
15:40:43 <dcaro> ewoud: yep, seen it before. Maybe it's better to have just one script that checks if any hosts are failing on foreman rather that one script per host?
15:41:12 <ewoud> dcaro: https://github.com/jrottenberg/check_puppet/blob/master/check_foreman_dashboard.py does exactly that: reports the number of hosts that are failing
15:41:53 <dcaro> ewoud: +1
15:42:17 <eedri_> knesenko, ok, i see it's not connected to foreman the 2nd host
15:42:24 <eedri_> knesenko, jenkins-slave-host02
15:42:52 <eedri_> knesenko, Exiting; no certificate found and waitforcert is disabled
15:42:58 <knesenko> eedri_: so we have one slave that is not registered to foreman/puppet
15:43:30 <eedri_> dcaro, ewoud i think it's because the hostname was renamed
15:43:31 <knesenko> eedri_: ok need to add it then
15:43:45 <knesenko> eedri_: ah ...
15:43:50 <eedri_> i think just deleting /var/lib/puppet/ssl and running certificate request again
15:43:59 <eedri_> should fix that (and signing the cert on foreman)
15:44:08 <knesenko> eedri_: what is the new hostname ?
15:44:20 <knesenko> jenkins-slave-vm01.ovirt.org ?
15:44:22 <eedri_> jenkins-slave-host02.ovirt.org
15:44:26 <knesenko> eedri_: ok
15:44:33 * eedri_ trying to do that now
15:44:42 <knesenko> +1
15:44:43 <eedri_> knesenko, and then we can merge obasan jnlp puppet clkass
15:45:01 <knesenko> #topic Jenkins
15:45:13 <ewoud> eedri_: https://foreman.ovirt.org/smart_proxies/1-foreman-ovirt-org/puppetca already shows a request vor jenkins-slave-host02.ovirt.org
15:45:20 <ewoud> s/vor/for
15:45:48 <eedri_> ewoud, yea, just going there now to sign it
15:46:02 <obasan> eedri_, knesenko before that we'll have to disconnect the current jnlp connection. because now it on it will be a systemd service
15:48:38 <knesenko> any news on Jenkins ?
15:49:09 <eedri_> knesenko, any update on the upgrade params?
15:50:30 <knesenko> eedri_: seemslike there is a new but there ....
15:50:34 <knesenko> but I am not sure.
15:50:41 <knesenko> will send email to Eli
15:50:58 <eedri_> knesenko, we had issues with dao tests, obasan any update on that?
15:51:23 <obasan> eedri_, knesenko yes. I refactored the jobs and splitted them. (two jobs for 33. and two jobs for master)
15:51:27 <obasan> eedri_, knesenko it seems to be working fine now
15:51:35 <eedri_> knesenko, also, there is an open patch on vdsm functional tests, waiting for answer from apuimedo
15:51:40 <eedri_> knesenko, iirc
15:52:02 <eedri_> obasan, i still see failures on master- http://jenkins.ovirt.org/job/ovirt_engine_dao_unit_tests_gerrit_master/
15:52:21 <eedri_> obasan, people reported that its running tests from 3.3, so maybe a cache/ws issue?
15:52:50 <obasan> eedri_, I'll try to clean the ws of them both . I took a look at some of the failures and they are real failures
15:53:27 <knesenko> #info knesenko send email to Eli regarding the upgrade job
15:53:48 <obasan> eedri_, although I doubt that because I changed the jobs names and splitted name so new w/s have been created
15:54:17 <eedri_> obasan, yea, so needs to be investigated still
15:54:42 <obasan> eedri_, I can send emails to the committers
15:55:07 <eedri_> obasan, try to understand the problem maybe 1st
15:55:34 <SvenKieske> sbonaazo: ping?
15:56:05 <eedri_> knesenko, i know we only have 5 min left for the meeting, but it seems we never have time to go over trac tickets
15:56:36 <eedri_> knesenko, so either we'll try to do another meeting every once in a while to review them, or maybe go over them in the begining?
15:57:03 <ewoud> we could start with the tickets that went the longest without any activity
15:57:21 <knesenko> lets review the urgent tickets ?
15:58:33 <knesenko> #topic Trac review
15:58:54 <knesenko> ewoud: will assign thois one to u https://fedorahosted.org/ovirt/ticket/101
15:59:02 <knesenko> ewoud: as you are going to do it today
15:59:29 <ewoud> knesenko: was about to
15:59:47 <ewoud> not sure how to modify the label so I'll just update centos for now
15:59:53 <ewoud> if that's sufficient
16:01:10 <knesenko> eedri_: is this relevant ? https://fedorahosted.org/ovirt/ticket/73
16:01:53 <eedri_> knesenko, this is basically the upgrade params job
16:01:56 <eedri_> knesenko, so you can close it
16:02:14 <knesenko> eedri_: no its not ... its another job
16:02:58 <eedri_> knesenko, i know, but upgrade from 3.2 to latest is not critical anymore imo for ovirt
16:03:23 <eedri_> knesenko, the one thing that should be added is running 3.3.x stable -> latest 3.4 ovirt (built from head)
16:03:39 <eedri_> knesenko, and also upgrade from lastest 3.4 stable -> latest 3.4 (head)
16:04:16 <YamakasY> ade: ?? my log doesn't go back that far
16:04:41 <SvenKieske> eedri_: So you do not want to test upgrades which users actually perform? (3.2.x -> 3.3.x)?
16:07:11 <ade> YamakasY, ok np. So Im going to do a full update tomorrow and see if Im still getting selinux denials - going out for a curry now though :D
16:08:19 <knesenko> #endmeeting