15:08:59 #startmeeting oVirt Infra 15:08:59 Meeting started Mon Dec 16 15:08:59 2013 UTC. The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:08:59 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:09:07 #chair ewoud eedri dcaro 15:09:07 Current chairs: dcaro eedri ewoud knesenko 15:09:11 * orc_orc_ is here 15:09:14 #chair obasan 15:09:17 #chair orc_orc 15:09:17 Current chairs: dcaro eedri ewoud knesenko orc_orc 15:09:26 #chair obasan 15:09:26 Current chairs: dcaro eedri ewoud knesenko obasan orc_orc 15:10:23 #topic Hosting 15:10:28 Hello 15:10:54 lets start ... 15:11:08 knesenko, hello 15:11:13 #chair eedri_ 15:11:13 Current chairs: dcaro eedri eedri_ ewoud knesenko obasan orc_orc 15:11:24 some news regarding the rackspace03 15:11:27 NO NEWS ! 15:11:36 they didn't fixed the problem 15:11:46 hunh 15:11:56 but you found the workaround, no? 15:12:05 or SvenKieske ... I fergit 15:12:17 orc_orc: no I am .... I can't ssh from rackspace01 to rackspace03 15:12:26 they have routing issues 15:12:30 not on our side ... 15:12:41 ;( 15:12:46 knesenko, i'll escalate this with out rax contact 15:13:10 eedri_: ok 15:13:29 knesenko: no, rackspace problem is 'solved', meaning now I can connect using the console 15:13:31 also there is a another ticket to update centos slaves to 6.5 15:13:46 dcaro: there is a another problem :) 15:13:56 * ewoud here 15:14:09 dcaro: no ssh between rackspace01 and 03 15:14:14 #chair ewoud 15:14:14 Current chairs: dcaro eedri eedri_ ewoud knesenko obasan orc_orc 15:15:07 so we are still blocked on its installation 15:15:21 knesenko: can't we work around that? 15:16:07 dcaro: how ? 15:16:21 knesenko: maybe creating a tunnel or something 15:16:32 knesenko: at least until it's fixed 15:16:33 dcaro: why should we ?! 15:16:48 knesenko: to get work done 15:16:50 dcaro: :) they just need to fix it ... 15:16:52 dcaro: a tunnel would need a woprking endpoint near '03 to reach the ILO and that is the problem 15:16:56 dcaro: I don't know ... 15:18:09 ok what about Centos slaves upgrades ? 15:18:25 updating the repo and run yum update should be enough ? 15:18:26 orc_orc_: we know that we can connect through jenkins.ovirt.org to both machines, maybe we can tunnel through there (if we are not going to send big files there should be no problem) 15:18:53 dcaro: orc_orc I don't want to make ugly workarounds ... 15:18:55 knesenko: I offered to do it, but haven't gotten around to it 15:19:03 dcaro: orc_orc they should fix it ... 15:19:09 ewoud: ok ... 15:19:21 ewoud: I hope I will have some time to do it this week . 15:19:25 knesenko: I can do it this evening 15:19:31 eedri_: how many Centos slaves do we have ? 15:19:34 ewoud: +1 15:19:35 ! 15:19:43 knesenko: 2 I think 15:20:03 ewoud, yea 2 15:20:09 ewoud: ok good 15:20:18 https://foreman.ovirt.org/hosts?utf8=%E2%9C%93&search=+hostgroup+%3D++%22jenkins+slave%22+and+os+%3D++CentOS shows 2 results 15:20:18 ewoud: thanks for the effort 15:20:43 just a matter of marking the slave as offline, wait till there are no jobs, yum update, reboot and mark it as online, right? 15:20:45 ewoud, there is also a f18 slave i think we're using, so we should reprovision it as f20 or centos / ubuntoi 15:20:53 ewoud, yea 15:20:58 and then the next 15:21:54 mskrivanek:could you tell me in which file and which function it starts vm from a live snapshot? 15:22:18 #action ewoud update centos slaves to 6.5 this evening 15:22:37 what about notifications / announcements? 15:23:01 #info rackspace03 is still blocked. Rackspace should fix their routing problem 15:23:16 bigclouds_: well, preview would do that 15:24:09 ewoud: which notifications ? 15:24:22 ewoud: I am sorry maybe I am missing something 15:24:57 mskrivanek:i mean vdsm code, i just want to know how to assign vm state file when start it in xml? 15:25:50 knesenko: about updating the slaves 15:26:18 ewoud: I think that sending email to infra should be enough 15:26:25 knesenko: ok 15:26:34 ok what next here ? 15:27:09 #topic Foreman and Puppet 15:27:20 Finally on the network 15:27:42 dneary: want to join the meeting ? 15:27:55 knesenko, I thought I just did ;-) 15:28:00 dcaro: ewoud any updates there ? :) 15:28:04 #chair dneary 15:28:04 Current chairs: dcaro dneary eedri eedri_ ewoud knesenko obasan orc_orc 15:28:11 dneary: )) 15:29:08 knesenko: we set up a jenkins job to do updates of the puppet repo on foreman.ovirt.org 15:29:19 knesenko: yep, we pushed the puppet manifest testing too 15:29:28 knesenko: not finished though 15:29:34 http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/system_update_puppet/ 15:29:58 as you can see on http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/system_update_puppet/3/console there's decent logging 15:30:00 #info dcaro and ewoud create a new job to update puppet - http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/system_update_puppet/ 15:30:23 haven't tested if the gerrit trigger on merge now works though, but it should 15:30:31 dcaro: where can I see the puppet manifest testing job ? 15:30:38 ewoud: ok 15:30:40 all reviews against master should be resubmitted against production 15:30:46 knesenko: http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/infra_check_puppet_manifests/ 15:30:59 knesenko: it's a work in progress though 15:31:03 http://gerrit.ovirt.org/#/q/status:open+project:infra-puppet+branch:master,n,z 15:31:29 #info ewoud and dcaro are working on http://jenkins.ovirt.org/view/System%20and%20Monitoring/job/infra_check_puppet_manifests/ 15:31:39 dcaro: ewoud great progress guys ! thanks 15:32:10 one thing I'd like to add is monitoring 15:32:48 at $work we use https://github.com/jrottenberg/check_puppet and I submitted a check that uses the foreman dashboard API 15:33:39 orc_orc_: mind helping me setting that up? 15:34:02 ewoud: I am find w this, later this week -- have to get a report out on the rhel 7 candidate 15:34:09 Thurdsay? 15:34:20 I think I'll manage, but in case I don't that sounds fine 15:34:28 ewoud: ok 15:36:03 great 15:36:08 anything else here ? 15:36:10 ewoud: it's fairly simple to get info from foreman using python-foreman ;) 15:37:29 dcaro: I did see that and haven't had the time to try it, but certainly looked promising 15:38:09 dcaro: but that check script was made to work on EL5 without additional packages installed 15:38:10 ewoud: we use it at work for our scripts 15:38:25 dcaro: otherwise I'd at least have used requests 15:38:48 ewoud: you should update your nagios machine os :) 15:39:00 knesenko, i think we have issues with minidell running puppet 15:39:31 dcaro: yes, that's a long running issue 15:39:38 eedri_: which one of them :? 15:40:00 dcaro: I won't bore you with the details, but lots of custom scripts, unpackaged nagios etc 15:40:10 * eedri_ checking 15:40:43 ewoud: yep, seen it before. Maybe it's better to have just one script that checks if any hosts are failing on foreman rather that one script per host? 15:41:12 dcaro: https://github.com/jrottenberg/check_puppet/blob/master/check_foreman_dashboard.py does exactly that: reports the number of hosts that are failing 15:41:53 ewoud: +1 15:42:17 knesenko, ok, i see it's not connected to foreman the 2nd host 15:42:24 knesenko, jenkins-slave-host02 15:42:52 knesenko, Exiting; no certificate found and waitforcert is disabled 15:42:58 eedri_: so we have one slave that is not registered to foreman/puppet 15:43:30 dcaro, ewoud i think it's because the hostname was renamed 15:43:31 eedri_: ok need to add it then 15:43:45 eedri_: ah ... 15:43:50 i think just deleting /var/lib/puppet/ssl and running certificate request again 15:43:59 should fix that (and signing the cert on foreman) 15:44:08 eedri_: what is the new hostname ? 15:44:20 jenkins-slave-vm01.ovirt.org ? 15:44:22 jenkins-slave-host02.ovirt.org 15:44:26 eedri_: ok 15:44:33 * eedri_ trying to do that now 15:44:42 +1 15:44:43 knesenko, and then we can merge obasan jnlp puppet clkass 15:45:01 #topic Jenkins 15:45:13 eedri_: https://foreman.ovirt.org/smart_proxies/1-foreman-ovirt-org/puppetca already shows a request vor jenkins-slave-host02.ovirt.org 15:45:20 s/vor/for 15:45:48 ewoud, yea, just going there now to sign it 15:46:02 eedri_, knesenko before that we'll have to disconnect the current jnlp connection. because now it on it will be a systemd service 15:48:38 any news on Jenkins ? 15:49:09 knesenko, any update on the upgrade params? 15:50:30 eedri_: seemslike there is a new but there .... 15:50:34 but I am not sure. 15:50:41 will send email to Eli 15:50:58 knesenko, we had issues with dao tests, obasan any update on that? 15:51:23 eedri_, knesenko yes. I refactored the jobs and splitted them. (two jobs for 33. and two jobs for master) 15:51:27 eedri_, knesenko it seems to be working fine now 15:51:35 knesenko, also, there is an open patch on vdsm functional tests, waiting for answer from apuimedo 15:51:40 knesenko, iirc 15:52:02 obasan, i still see failures on master- http://jenkins.ovirt.org/job/ovirt_engine_dao_unit_tests_gerrit_master/ 15:52:21 obasan, people reported that its running tests from 3.3, so maybe a cache/ws issue? 15:52:50 eedri_, I'll try to clean the ws of them both . I took a look at some of the failures and they are real failures 15:53:27 #info knesenko send email to Eli regarding the upgrade job 15:53:48 eedri_, although I doubt that because I changed the jobs names and splitted name so new w/s have been created 15:54:17 obasan, yea, so needs to be investigated still 15:54:42 eedri_, I can send emails to the committers 15:55:07 obasan, try to understand the problem maybe 1st 15:55:34 sbonaazo: ping? 15:56:05 knesenko, i know we only have 5 min left for the meeting, but it seems we never have time to go over trac tickets 15:56:36 knesenko, so either we'll try to do another meeting every once in a while to review them, or maybe go over them in the begining? 15:57:03 we could start with the tickets that went the longest without any activity 15:57:21 lets review the urgent tickets ? 15:58:33 #topic Trac review 15:58:54 ewoud: will assign thois one to u https://fedorahosted.org/ovirt/ticket/101 15:59:02 ewoud: as you are going to do it today 15:59:29 knesenko: was about to 15:59:47 not sure how to modify the label so I'll just update centos for now 15:59:53 if that's sufficient 16:01:10 eedri_: is this relevant ? https://fedorahosted.org/ovirt/ticket/73 16:01:53 knesenko, this is basically the upgrade params job 16:01:56 knesenko, so you can close it 16:02:14 eedri_: no its not ... its another job 16:02:58 knesenko, i know, but upgrade from 3.2 to latest is not critical anymore imo for ovirt 16:03:23 knesenko, the one thing that should be added is running 3.3.x stable -> latest 3.4 ovirt (built from head) 16:03:39 knesenko, and also upgrade from lastest 3.4 stable -> latest 3.4 (head) 16:04:16 ade: ?? my log doesn't go back that far 16:04:41 eedri_: So you do not want to test upgrades which users actually perform? (3.2.x -> 3.3.x)? 16:07:11 YamakasY, ok np. So Im going to do a full update tomorrow and see if Im still getting selinux denials - going out for a curry now though :D 16:08:19 #endmeeting