15:01:19 #startmeeting oVirt Infra Team Weekly 15:01:19 Meeting started Mon Jan 14 15:01:19 2013 UTC. The chair is quaid. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:19 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:01:23 yay 15:01:30 #topic Agenda & roll call 15:01:39 * quaid finds the right channel and starts things ... 15:01:45 http://www.ovirt.org/Infrastructure_team_meetings#2012-01-14 15:02:08 * Hosting 15:02:08 * Puppet 15:02:08 * Jenkins 15:02:08 * Gerrit 15:02:08 * Other business? 15:02:10 * Trac review 15:02:14 15:02:53 sorry I'm so bad about reminders for this one, it comes so early for me on Monday that unless I remember to do the reminder on Sunday night ... 15:03:26 Morning all 15:04:06 re Other business: Add website space issue & adding redundancy for OpenShift administration, please 15:04:33 (actually, I guess it's afternoon for most of us - morning Karsten :-) ) 15:04:51 ping for meeting: ewoud RobertM dcaro eedri 15:05:04 quaid, pong 15:05:36 dneary: added, thanks 15:05:56 ok, just a few of us, we'll move quickly then :) 15:06:00 dcaro: g'day 15:06:10 #topic Hosting 15:06:53 #info I had a kickoff meeting last Friday with RackSpace for the new hosting there; eedri & dcaro were included (either in the meeting or the materials) 15:07:11 #info 10 to 15 business days from Friday 11 Jan for the new hosts to arrive 15:08:42 I'll post the details about those hosts to the etherpad 15:08:45 http://etherpad.ovirt.org/p/new_hosting_design_Jan_2013 15:09:13 #action Need to follow up with Kevin from Alter Way for status 15:09:44 Hi :) 15:09:53 When you spoke of me 15:10:10 Server are beeing installed and will be delivered on wednesday 15:11:36 goacid: oh hi! 15:11:42 Hi :) 15:11:52 #info AlterWay servers are being installed and will be delivered on Wednesday 15:12:02 quaid, Would also like to add eedri's request that we add dcaro as Jenkins admin for the master and slaves (and get him to help migrate to the new server). 15:12:04 goacid: great, thanks 15:12:16 dneary: +1 15:12:20 quaid, Given your last info, it's timely :-) 15:12:41 DNS will be setup in few minutes with the good reverse 15:12:51 dneary: was that just a request of eedri's looking for group +1 or something further needed? 15:13:07 iirc, eedri is able to grant those rights, yes? 15:13:49 goacid: I was thinking of having the hosts go by the unimaginative alterway{01,02}.ovirt.org 15:14:28 Well that is ok for me :) If you want more funny name juste let me know 15:15:27 yeah, we can name hosts after spells from Harry Potter :) 15:15:37 * quaid just happy for clear and meaningful 15:15:55 ok, we have a hosting design document that is coming together fairly well 15:16:07 we can make a plan from it 15:16:44 (or modify what is there, that is) 15:17:29 I'll work on improving that one today, then throw the plan to the list 15:17:38 #action quaid to work on final migration plan for list review 15:17:46 I'm also ok with the boring names 15:17:51 anything else for hosting? 15:18:09 * eedri here 15:18:14 howdy eedri 15:18:25 hey, sorry for being away... had to take care of something 15:19:00 +1 on the alterway news... 15:19:19 np 15:19:41 quaid, i can grant access to jenkins ui 15:19:56 quaid, what is the official way to grant ssh sudo access? 15:20:33 eedri: honestly, I don't think we did decide anything official :) but ... 15:20:42 eedri: is there an open question on the mailing list? 15:20:47 * quaid hasn't read all his email :) 15:21:03 I think a few +1s from existing Infra folks and no -1 is the usual way 15:21:15 so +1 from me 15:21:19 quaid, i meant the technical way 15:21:22 ah! 15:21:30 quaid, of adding users access 15:21:46 I have a wiki page that explains my method I've been using, I think 15:22:10 http://www.ovirt.org/Adding_a_new_system_administrator_to_a_host 15:22:47 quaid, I imagined that it would be in AOB, but didn't want to forget it. eedri just sent an email to infra proposing dcaro as Jenkins admin. 15:22:58 +1 from me 15:23:20 I'd also like to hear from ewoud 15:23:34 great, we can +1 on list to make it formal 15:24:03 thanks :) 15:24:05 ok, we'll bring this up again in a few minutes, do the final approval on the mailing list ASAP 15:24:13 going to move on to Puppet ... 15:24:25 #topic Puppet 15:24:41 #info dcaro has sent initial Puppet manifests to the list 15:25:08 http://lists.ovirt.org/pipermail/infra/2013-January/001799.html 15:25:15 https://github.com/david-caro/puppet-jenkins 15:25:35 I have zero experience with these 15:26:34 dcaro: can you contact ewoud to make sure he saw & can reply to that thread? 15:27:59 #action Infra team needs especially ewoud need to review initial Jenkins Puppet manifests 15:28:06 #undo 15:28:06 Removing item from minutes: 15:28:09 that was sloppy :) 15:28:13 yes, he already commented some things in the pull request: https://github.com/rtyler/puppet-jenkins/pull/40 15:28:23 #action Infra team especially ewoud need to review initial Jenkins Puppet manifests 15:28:29 oh, great 15:28:55 sweet, that's underway then 15:29:29 dcaro: are we going to try to do our config for the AlterWay hosts using Puppet? 15:29:45 meaning ... do we have anywhere to put Puppet master? 15:30:40 checking the design doc, we have it slated to be a VM on alterway02 15:30:47 so ... chicken & egg problem 15:31:02 quaid, i think we said we'll install jenkins manually 1st 15:31:16 quaid, and after we'll have puppetmaster install we can enforce the puppet classes on it 15:31:27 quaid, and make sure it's stays aligned to the configuration 15:31:39 ah, yes, thanks! 15:31:44 I do recall that 15:32:15 #agreed We'll install Jenkins manually, then have puppet master enforce the puppet classes and make sure it stays aligned to the configuration 15:32:31 anything else for Puppet topic? 15:33:27 quaid, are we going to have it on gerrit? 15:33:35 quaid, a puppet git repo 15:33:51 quaid, imported from github, that means 15:34:02 eedri, using submodules? 15:34:31 makes sense to me 15:34:38 dcaro, not necessarily, just to better manage it internally in ovirt 15:34:43 dcaro, and not on github 15:35:01 +1 to puppet git repo on gerrit 15:35:11 dcaro, we can keep having it on github, and cherry pick what we want to use to gerrit.ovirt.org 15:35:19 +1 15:35:22 +1 15:35:27 ewoud, ? 15:36:17 I think I recall when discussing this similar topic, we wanted to make sure we were packaging some of our configs - is that correct? 15:38:21 http://docs.puppetlabs.com/guides/best_practices.html 15:38:23 dcaro, Did you commit your private RSA key to github? 15:38:33 https://github.com/david-caro/puppet-jenkins/tree/master/files 15:39:04 * quaid hates when he does that 15:39:09 nono, it's a custom generated one, only for that, actually you are supposed to change it (I'm still trying to figure out how to avoid having it there) 15:39:55 eedri: would you be able to setup the repo on gerrit? 15:40:17 quaid, i don't have access/perms to add new repos, only itamar 15:40:27 ok 15:40:43 quaid, which one? 15:40:46 eedri: can you bring up the request on infra@ and we'll get ewoud's input there? 15:40:48 Ah :-) 15:40:49 sorry if i should interrupt the meeting / need to know though how to detach an export domain ? do i need to set the DC into maintenance mode ? if i do that - does anything happen to the VMs running ? 15:40:54 dcaro, This makes sense 15:40:58 Excuse me! 15:41:45 itamar: I think we're talking about one or more new repos for Infra, this one for Puppet manifests 15:41:57 * quaid doesn't know the answer to ale's question 15:42:02 please send an email to infra mailing list (cc me) asking for it creation... 15:42:12 itamar: +1 15:42:49 actually ... dcaro can you make that request on infra@ and we'll get final input there? 15:43:58 quaid: sure 15:44:10 i tried copying the exported templates to another export domain, but it complains that the templates already exist in the system ... :\ .... so it does find the VM Templates but i cannot import them into the new DC 15:44:46 #action dcaro to request Puppet module on Gerrit 15:45:32 #agreed Puppet repo on Gerrit generally a good idea, will finish detail discussions on list 15:45:36 moving on ... 15:45:48 #topic Jenkins 15:46:07 #action All reply to the on-list request to grant admin access to dcaro 15:46:51 #info Jenkins migration plan embedded in the overall new hosting design 15:46:53 http://etherpad.ovirt.org/p/new_hosting_design_Jan_2013 15:47:04 anything else on Jenkins here? 15:47:33 recently we had some failures on jenkins jobs, it's motly cause of infra 15:47:44 i suspect slow vm response 15:47:50 and out of space issues 15:48:05 hopefully will be solved once we'll move to new infra 15:48:19 definitely hoping so :) 15:48:49 #info recent job failure on Jenkins are likely due to infrastructure - slow VM response and out of space issues 15:49:29 any more on Jenkins? 15:49:44 topic going once .... 15:49:56 topic going twice .... 15:50:01 sold 15:50:03 #topic Gerrit 15:50:23 #info Gerrit is generally stable currently 15:50:37 previously we agreed we'll be planning that move after we get Jenkins et al settled 15:51:15 #agreed Still planning to move Gerrit to new hosting after things settle down from the Jenkins and Mailman moves 15:51:40 #idea Should do the Gerrit install using the new Foreman/Puppet system 15:51:50 * quaid knows that's an obvious one, just getting it in the record :) 15:52:09 ok, going to close this one quickly ... anything more? 15:53:18 um HTTPS on gerrit? 15:53:41 ah 15:53:52 I think we agreed to do that, yes 15:54:11 #action quaid to get an SSL Cert for one or more ovirt.org domains 15:54:22 I'll make sure we have a ticket open for that, and start working it 15:55:12 * vfeenstr luckily looked into here at the right moment :) 15:55:18 I want to get this one last topic in before the hour 15:55:20 * ewoud here now 15:55:23 stupid meetings 15:55:23 so I'll do that ticket in parallel 15:55:34 #topic OpenShift - space & admin redundancy 15:56:00 #info we ran out of space on our OpenShift gear over the weekend, which causes an unhelpful error that dneary fixed by cleaning up things 15:56:12 #action quaid is asking #openshift for a much bigger quota 15:56:22 I just heard back from the OpenShift guys 15:56:24 dneary: what were you thinking about admin redundancy? 15:56:40 We have a bigger quota (did you see Grant Shipley's email? 15:57:13 quaid, I learned that this is as simple as "rhc log-clean" or something like that 15:57:36 quaid, So - you're the creator of the "app", I only have access through git 15:57:59 I asked if there was any way for me to access admin tasks using rhc, and that's only possible for the app creator 15:58:04 Apparently 15:58:11 dneary: didn't see that, thanks 15:58:20 #info Bigger quota is now here, thanks dneary 15:58:23 Just figuring out how we can make sure that we don't need you if there is a problem :-) 15:58:31 #info 'rhc log-clean' helps keep things clean 15:58:42 #idea can we automate openshift cleanup using cron? 15:59:10 is there any kind of log rotation policy? I don't like the idea of having to run the cleanup from another machine 15:59:13 #action quaid needs to figure out how to add other admins to OpenShift, then add folks from Infra 15:59:43 #idea can we use OpenShift log rotation automagic? 16:00:10 dcaro, These are the questions to which I do not have answers 16:00:16 me either :) 16:00:33 I did google the error message I got, and apparently "we're working on a better solution" is the answer people got in forums 16:00:35 but we'll get some 16:01:23 I found someone suggesting using the cron cartdrige for that 16:01:42 #idea could use cron cartridge for our own log rotation 16:01:55 mburns, ping 16:02:42 eedri: pong 16:02:49 so i wrote a little script a while back to generate a spice link for all my vm's using ovirt-shell, and all-uva-sudden it's complaining that --password is no longer allowed...do i have to generate keys to get this working again? 16:03:37 basically it ran list vms, then looped through them and generated output like vmname: ___ port: ___ password: ___ to make my life easier 16:03:53 ok, I've got another meeting about to distract me, so .... 16:04:04 anything else before we close this topic and/or the meeting? 16:04:30 :| .. any1 know what happens to the VMs if I put a DC into Maintenance ? Are they being stopped / or you just cant do anything while the host is in Maint mode ? 16:05:32 it'd be cool if a DC could use multiple export domains that do not belong to the DC .. 16:06:30 closing meeting in one minutes 16:08:49 Thanks folks! 16:08:53 #endmeeting