15:01:08 #startmeeting oVirt infra weekly 15:01:08 Meeting started Mon Feb 25 15:01:08 2013 UTC. The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:01:19 copy+paste from http://www.ovirt.org/Infrastructure_team_meetings#2013-02-25 15:01:26 Agenda 15:01:30 Hosting Puppet Jenkins Gerrit Other business? Trac review 15:01:55 #chair Rydekull garrett_ dcaro eedri_ 15:01:55 Current chairs: Rydekull dcaro eedri_ ewoud garrett_ 15:01:59 ewoud: what about new guys introduction ? :) 15:02:13 knesenko: I'd like to start with that then :) 15:02:32 so let's 15:02:40 #topic Introductions 15:02:44 Hi all my name is Kiril I am from integration team of oVirt product 15:03:01 obasan: your turn :) 15:03:10 knesenko: did you also start the thread 'New ideas to grow the community'? 15:03:22 ewoud: Yes I am 15:03:38 hi my name is Ohad 15:03:53 I am from int. team as well . 15:04:11 For someone who is also quite new, whats the integration team? :-) 15:04:19 Rydekull: I was about to ask :) 15:04:34 Rydekull, i think knesenko meant integration team for the rhevm product 15:04:50 that what I meant 15:05:00 obasan, knesenko: welcome 15:05:07 knesenko, obasan: welcome!! 15:05:10 jhernand, ping 15:05:16 ewoud, dcaro thanks :) 15:05:17 anything in particular you're interested in picking up? 15:05:22 +1 welcome 15:05:32 ewoud: hey, when you get a second, in this meeting, i have some input on the alterway servers 15:05:33 welcome guys 15:05:36 ewoud: not sure what that means either 15:05:43 ewoud: err, sorry 15:05:48 mburns: I think we can do that under hosting 15:05:54 eedri_: not sure what that means either. But, ok :-) 15:06:01 ewoud: sure, just ping me when you get there... 15:06:05 mburns: will do 15:06:06 system administration, jenkins, automation jobs - sounds great 15:06:22 Rydekull, in a few words: integration team is in charge of ci testing for rhevm, building rhevm various projects 15:06:36 cool 15:07:00 Aha 15:07:10 That explains a bit more :-) 15:07:37 Rydekull, using foreman & puppet of course 15:07:53 knesenko, obasan: Welcome, glad you're willing to join us :-) 15:08:10 So me and obasan don't have problems to work on jenkins, puppet and systems administration 15:08:15 we are here to contribute 15:08:16 :) 15:08:24 FYI: I'm ewoud, mostly interested in setting up foreman & puppet now 15:08:43 so I'm sure we'll work together 15:08:51 ewoud: np 15:09:02 looking forward to it 15:09:04 anyone else wants to introduce or shall we move on to hosting? 15:09:10 I finished 15:09:15 #topic hosting 15:09:34 #info eedri started on the jenkins migration 15:09:49 #undo 15:09:49 Removing item from minutes: 15:09:56 yes. new server is alive @ alterway01.ovirt.org 15:09:59 #info eedri started on the jenkins migration, details on the ML and migration planned for thursday 15:10:24 ewoud, we might start it on wed if possible even 15:10:33 ewoud: can i jump in here before we plan any more migrations or set timelines? 15:10:40 mburns: sure 15:11:05 i just talked to acathrow and we can get full RHEL licenses (with support) for the alterway boxes 15:11:22 we need those for the RAX boxen 15:11:44 i'm trying to do that, took over from dneary 15:11:58 quaid: we can probably get them for that too, but i'd need to talk to acathrow 15:12:00 then we can have Jenkins slaves there that run RHEL 15:12:00 mburns: I think one reason we chose for centos is that our users are also likely to run centos 15:12:08 mburns, we have a talk about which OS to install on the alterway servers, and we included rhel as one of the options. but we decided on centos 15:12:16 not because we couldn't get rhel 15:12:37 yep, we decided also to not keep revisiting the decision 15:13:17 * Rydekull has tabled the question long ago in his mind :-) 15:13:38 it was a very long discussion indeed :) 15:13:38 quaid: eedri_ that's fine, i wasn't paying attention to the discussion, and the topic was raised to me directly that we perhaps "should" be using rhel... 15:13:50 mburns: so if you now have 2 licenses then we can use those for RAX, right? 15:14:04 * Rydekull has no clue what RAX is 15:14:06 mburns: we should, as well as should be using centos too 15:14:10 rackspace I'm guessing 15:14:11 rackspace 15:14:48 (stock ticker) 15:14:53 ewoud: i don't have anything at the moment, i was just told that we can get supported entitlements for the alterway boxes 15:15:12 i expect it's also true of the RAX boxes... 15:15:16 mburns: ok 15:15:35 I hope quaid will talk to the right people to arrange that :) 15:15:37 if it's decided that we don't want them on alterway, that's fine, i'll back and tell them 15:15:38 mburns: i'm going to include acathrow on my work there, so we can get to this desired state too 15:15:45 quaid: ack 15:16:12 #info redhat offered RHEL licences for alterway servers, but it has been decided that we want to run centos 15:16:14 mburns: yeah, it's decided and in the ml archives in Dec & Jan 15:16:46 so I think we can start the jenkins migration as soon as eedri_ has time 15:17:18 ewoud, yes. new server is ready, so next in line is the migration. hopefully will start either on wed/thu 15:17:41 ewoud, quaid how do we want to handle the dns change? 15:18:04 eedri_: I don't know since I have no idea how RH DNS works 15:18:26 which may also become relevant when we start using foreman to provision: it would be nice if we could talk to a DNS API somehow 15:18:31 though that's optional 15:18:37 hmm 15:19:01 right now i'm under the assumtion that the new server is alterway01.ovirt.org 15:19:09 in this case, we want to move jenkins.ovirt.org? 15:19:12 i'm guessing that after the migration will be over and verified, we'll want to change it 15:19:32 maybe change jenkins.ovirt.org to archive-jenkins.ovirt.org for backward recovery 15:19:33 the TTL is already 300, so it should be fast but it should be coordinated 15:19:52 quaid: I think so 15:20:32 eedri_: you can point 2 ips to the same fqdn 15:20:44 eedri_: ignore 15:21:09 we can request a specific time window, especially with days to plan 15:21:36 quaid: and if we run into problems, how easy is it to cancel that change? 15:21:56 we can even use a proxy to show the new server from the old one while the dns change applies 15:21:59 what we also can do is set up a reverse proxy on the old host 15:22:04 dcaro: :) 15:22:21 that's great 15:22:30 no need for dns change then 15:22:36 until we'll be sure the new server works 15:22:41 yes 15:22:44 eedri_: yep 15:22:48 _! 15:22:49 +1 15:23:17 #info we'll use a reverse proxy for jenkins.ovirt.org until we've verified the migration is completed, then change DNS 15:23:31 ok, I think that's enough about the jenkins migration 15:23:53 ewoud: should be easy to cancel 15:23:57 anyone is working on alterway02? 15:24:03 ewoud: mhh ok.. what shall I try ? 15:24:11 waiting for centos reinstall? 15:24:27 I saw Rydekull mailed to arch@ just before the meeting, got a +1 but no centos 6 packages yet 15:24:40 YamaKasY: not sure, bit busy too now 15:24:52 Yeah, I will follow up by trying to contact the specific maintainers directly during this week. I was a bit swamped on this issue 15:25:37 quaid: both blocked on centos 6 reinstall, but after that on EL6 packages 15:25:39 ewoud: okay! 15:26:35 given we're blocked I'd propose that we do start setting up a foreman + puppet base 15:27:10 I can update my foreman PoC environment so we at least have a base to start with 15:27:35 ewoud, are we planning to install foreman as a VM on ovirt instance (on alterway02)? 15:27:55 then stuff like creating the user accounts can be set up, inclusing SSH keys; we can play with our puppet repo in gerrit 15:27:59 eedri_: eventually yes 15:28:05 eedri_: but until we 15:28:20 've arranged that I think we shouldn't block on that completely 15:29:05 #action ewoud update foreman PoC to something functional 15:29:27 I'll work on that we aren't stuck 15:29:45 I think that covers hosting, puppet & jenkins 15:30:05 can we move on to the next point on the agenda? 15:30:11 +1 15:30:23 #topic gerrit 15:30:44 we are talking about gerrit.ovirt.org ? 15:30:45 about the topic: it's been on our agenda template, but we haven't actually discussed it for a few weeks 15:30:48 knesenko: yes 15:31:15 I think the eventual goal is that we're going to maintain it, but right now it's not really 15:31:28 ewoud, plan is to migrate it also right? 15:31:33 eedri_: correct 15:32:08 yes, when we have things stabilized, including team processes sonitMs clear how we support it, etc. 15:32:37 should we leave it off the agenda for next week? 15:32:42 +1 15:33:01 #topic other business 15:33:21 knesenko mentioned http://www.ovirt.org/Becoming_an_Infrastructure_team_member needs some work on the ML 15:33:27 I think he's right 15:34:32 ewoud: We need to create some steps 15:34:38 * Rydekull would be surprised if anyone thought otherwise 15:34:40 enjoying mailing lists 15:34:44 tasks 15:34:47 etc ... 15:35:05 knesenko: I agree, but not sure how it can be established 15:35:24 I am not sure what is the process today :\ 15:35:40 traditionally in open source you submit patches and you grow into a maintainer, but with infra that's harder 15:35:53 agree 15:35:59 Well 15:36:03 but its very important 15:36:10 right, we want to be able to give out little tasks people can grow with 15:36:22 Without any qaccess what so ever, its hard. But you can still be given accounts on machines etc 15:36:33 and with screen etc, you can in theory, with a mentor, do stuff/be surpervised 15:36:38 there are tools to collaborate 15:36:39 one problem is trust; just giving root is a bit scary but without it you can't do a lot 15:36:46 ewoud: +1 15:36:50 +1 to working out improvements to that page on the ML 15:37:12 ewoud: we can figure out where people can help via transparency 15:37:24 I'll kick of a thread 15:37:26 I think the essential part is to have a mentor being picked out. And that mentor giving tasks, personally 15:37:29 i.e. give out sudoless ssh so people can poke around 15:37:40 I also vote for givin 'per user' accounts and give sudo permissions, to have more control than just root pass 15:37:44 #action ewoud start a thread on the ML to start documention the becoming an infra member 15:37:49 and offer solutions that an other can be rootnfor 15:37:50 dcaro: +1 15:38:23 right, we can create a backup group & get help with that 15:38:27 I think that no one should work as root . only with sudo 15:38:34 or log gathering & analysis 15:38:34 maybe we can set up sudo profiles for give access to do some tasks 15:38:42 +1 15:38:46 +1 15:38:51 dcaro: that implies we have some standard tasks 15:38:52 1. ssh for looking aroun d 15:39:02 2. ssh + limited sudo 15:39:04 would be a good thing to have, but I'm not sure we currently have that 15:39:07 3. full sudo 15:39:11 ewoud: sure 15:39:17 do we have some git with conf. files of the machines or something like that 15:39:17 ? 15:39:29 progress 1 to 3 15:39:30 knesenko: not yet, but we aim to set up puppet for that 15:39:36 i think we can start with monitoring jenkins jobs, that can be easilly solved with limited permissions for jenkins ui 15:39:46 quaid: and maybe a 0. attend a meeting / active on ML 15:39:48 eedri_: +1 15:39:54 ewoud: +1 15:40:00 eedri_: Wouldnt that depend on which part you'd be involved with 15:40:33 let's continue this on the ML 15:40:57 any other business? 15:41:00 yes 15:41:12 what is the procedure of getting a poweruser access to jenkins? 15:41:26 I'd like to add a job that will generate a nightlies of ovirt-live 15:41:31 basically is sending a request to infra list 15:41:44 and getting approval from the list or community 15:41:48 I sent such request. who is supposed to authorize that? 15:42:36 +1 fpr obasan 15:42:58 obasan, i don't think your request made it to the list 15:42:59 maybe we want to normally use tickets so we don't forget the task? 15:43:00 obasan: when did you send it? 15:43:10 obasan: please dig it up from the archives 15:43:13 2/3 days ago. 15:43:25 quaid: +1 for tickets 15:43:41 quaid: do we have some ticket system ? 15:43:53 https://fedorahosted.org/ovirt/ticket 15:44:16 I think it didn't make it. I will resend it. 15:44:50 Rydekull: 10x 15:45:22 ewoud: do you need any help with puppet/foreman installation/configuration ? 15:45:53 As for other business, short report from the outage last week(which I think should be a item on the agenda) 15:46:06 We went out for a few hours as you may have seen on the email i sent 15:46:35 this was due to errors on openshift's side that made our website behave strangely. Only letting through a request once every decade or so 15:46:36 knesenko: I could use some 15:47:06 is there any technical doc about snapshot ? i search in ovirt wiki but doesn't found. this is not lvm snapshot ,maybe a qcow method but where is put the file ? 15:47:26 sorry, back 15:47:31 ewoud: ok np 15:48:18 knesenko: some time ago I made https://github.com/ekohl/ovirt-infra-puppet as a proof of concept, but it could use some updates 15:48:59 knesenko: I think the first goal should be to create the user accounts so giving access to all our infra is easy 15:49:42 knesenko: and when it's practical we should move the repo to gerrit.ovirt.org 15:49:59 ewoud, knesenko: we can discuss that in the lists maybe? create a list of what we want on puppet and split the tasks? 15:50:07 dcaro: will do 15:50:10 so you want to create users with puppet on all machines ? 15:50:20 #action ewoud re-ignite the puppet discussion on the ML 15:50:46 knesenko: you can define virtual resources in puppet and only create a subset 15:51:06 I think this brings us to the trac review 15:51:35 #topic Trac ticket review 15:51:56 we haven't done this in a while 15:52:34 quaid: could you start with https://fedorahosted.org/ovirt/ticket/5 so we at least have some sort of workflow? 15:53:35 heh, yes 15:53:37 I did close a few tickets last week 15:53:46 Rydekull: I noticed the list was a bit shorter 15:53:52 rydekull++ was determined 15:54:07 I sorted out the ovirtbot, and ConfirmAccount 15:54:10 it was nice to close out some items 15:54:17 and I prodded some others to close some things that had solved themselves 15:54:30 ewoud: i'll work #5 today 15:54:45 quaid: cool 15:55:07 https://fedorahosted.org/ovirt/ticket/31 Licensing (rights) link disappears when graphic is not used 15:55:11 I'm not sure what has to be done 15:56:26 garrett: ping 15:56:40 we have a feew theme items to discuss with garrett 15:56:49 Yeah 15:57:15 I tried to figure out the licensing link, couldn't solve it 15:57:25 quaid: https://fedorahosted.org/ovirt/ticket/27 Obtain entitlements for RHEL for Infrastructure hosts at RackSpace was picked up by you instead of dneary? 15:57:34 yes 15:57:47 go ahead and reassign plz if not done already 15:57:59 Is there any specific issue, you'd like me to focus on? 15:58:01 * quaid is on his phone not laptop :) 15:58:26 quaid: done 15:58:35 https://fedorahosted.org/ovirt/report/1 is the report btw 15:58:36 i'd like us to share in the Gerrit knowledge 15:59:00 quaid: I'm setting up gerrit at $dayjob and getting some knowledge 15:59:15 was Gerrit something Rydekull wanted to learn? /me not clearly recalling 15:59:25 Yes 15:59:36 But it sounds as its starting to get crowded also 15:59:52 nah, more the merrier, and you did ask :) 15:59:59 :-) 16:00:19 maybe we can setup a one-off meeting with Itamar to learn? 16:00:36 eedri: who else admins gerrit? 16:00:57 Gerrit - gerrit.ovirt.org: iheim, rydekull 16:01:24 rydekull: do you want to setup a sprint/meeting with iheim? we can coordinate on infra@ 16:02:04 all of us who want to learn & help can pick a time, etc. 16:02:06 Sure, i'll put gerrit as my next task as an apprentice :-) 16:02:09 quaid: I do not administer gerrit, but I could use the knowledge :) 16:02:14 +qq 16:02:18 +1 16:02:48 #action Rydekull set up a sprint/meeting about gerrit to spread knowledge 16:02:59 I'd like to close 16:03:20 closing in 3 16:03:20 +1 16:03:23 Feel free 16:03:23 2 16:03:26 1 16:03:29 thanks all 16:03:30 #endmeeting