13:59:07 <knesenko> #startmeeting Infra Weekly 13:59:07 <ovirtbot> Meeting started Mon Sep 9 13:59:07 2013 UTC. The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:59:07 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic. 13:59:08 <dneary> (I mean, you have that right in the wiki) 13:59:12 <dneary> hi 13:59:13 <mburns> eedri: knesenko: that was my assumption, but wanted to ask to be sure. thanks 13:59:19 <itamar> dneary, I'm not sure if to be happy or sad ;) 13:59:19 <knesenko> #chair ewoud eedri dcaro 13:59:19 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko 13:59:24 <knesenko> Rydekull: here ? 13:59:40 <knesenko> #chair obasan 13:59:40 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan 13:59:50 <dneary> itamar, You're welcome, I think :-) 13:59:55 <bilias> thnx 13:59:57 <knesenko> dneary: hello ... want to join ? 14:00:48 <knesenko> #topic Hosting and Issues 14:00:53 <knesenko> Hi all 14:00:57 <dcaro> knesenko: now Im here :) 14:01:03 <knesenko> so lets start 14:01:17 <knesenko> last week dcaro installed new HDs on rackspace servers . 14:01:23 <knesenko> dcaro: want to update us ? 14:01:59 <ewoud> knesenko: I think Rydekull was looking at setting up gluster on it 14:02:17 <eedri> knesenko, yes, the HDs are already installed for 2 weeks iirc 14:02:18 <ewoud> knesenko: but he found out someone created a filesystem on it and ran into issues there 14:02:21 <knesenko> ewoud: yes I know ... I tried to ping him ... 14:02:36 <dcaro> yep, it seems that the hd on rackspace 2 was 'connected' but it lacks the raid configuration, I opened a ticket and we are gonna make sure we can use it after 14:02:52 <knesenko> dcaro: ok thanks 14:03:08 <knesenko> so team .. we need to decide what should we do with these HDs 14:03:20 <knesenko> dcaro: on which server we have RAID ? 14:03:32 <dcaro> knesenko: 01 has raid configured 14:03:44 <dneary> knesenko, As a chair, you mean? 14:03:47 <knesenko> ok so I assume 01 will be used for backups 14:03:51 <knesenko> dneary: yes 14:04:10 <knesenko> but what about gluster ? 14:04:17 <knesenko> which mode do we want to run there ? 14:04:18 * dneary is not bothered about being a chair 14:05:55 <ewoud> knesenko: I'd say we create gluster on it and create a backup VM on top of that 14:06:12 <ewoud> knesenko: I'd rather not log in to the hypervisor for backups 14:06:52 <knesenko> ewoud: so you mean we need to change the setup that we are running on rackspace servers right now 14:06:52 <eedri> ewoud, +1 14:07:03 <knesenko> right now we have local storage there ... 14:07:14 <ewoud> knesenko: we can combine them 14:07:34 <ewoud> knesenko: leave the current jenkins slaves on local storage, create a gluster volume on the new storage 14:07:45 <knesenko> ewoud: its impossible to add more than one storage domain into the local DC 14:07:46 <ewoud> knesenko: that's an extra storage pool we can create VMs from 14:07:50 <ewoud> knesenko: argh 14:08:05 <knesenko> ewoud: that's why we need to change the setup 14:08:08 <eedri> knesenko, can't we create another DC? 14:08:17 <knesenko> we don't have hosts for that 14:08:52 <knesenko> I think that we should export the slaves we have right now 14:08:59 <knesenko> remove the setup ... 14:09:03 <ewoud> knesenko: I think it could be worth it to change it, but I have insufficient experience with gluster to know if our jenkins slaves will run on them 14:09:03 <knesenko> create a gluster 14:09:17 <eedri> knesenko, where will you export it to? 14:09:17 <knesenko> and use this gluster volume for the engine setup 14:09:44 <knesenko> will create some tmp export domain or something 14:09:44 <eedri> knesenko, and since almost all of our jenkins slaves is running on rackspace now, how much downtime of the slaves are we talking about 14:09:59 <knesenko> hehee .... it depends ... 14:10:18 <knesenko> we need to prepare a plan for it 14:10:46 <knesenko> I think that the project can take 3-4 hours 14:10:47 <ewoud> exportdomain can be NFS 14:12:05 <knesenko> I think that it should be pushed asap 14:12:12 <knesenko> +1 from me 14:12:18 <ewoud> +1 from me as well 14:12:37 <eedri> i have a suggestion 14:12:44 <knesenko> eedri: yes please 14:12:45 <eedri> why not test it on a single host 1st? 14:12:53 <knesenko> eedri: test what ? 14:12:55 <eedri> so we'll still have a few vms running 14:13:03 <eedri> adding gluster storage 14:13:11 <ewoud> eedri: sounds like a good plan 14:13:13 <danken> sorry for popping into the meeting - could you add the EL6 gluster-3.4.0 packages to the nightly repo? ybronhei asked for that in vdsm meeting. 14:13:19 <eedri> you can create a new DC 14:13:25 <eedri> and move one host to there 14:13:33 <eedri> also - might worth asking rackspace for a 3rd host 14:13:36 <knesenko> eedri: possible ... 14:13:40 <eedri> only if for temporary 14:13:42 <eedri> for migration 14:13:49 <knesenko> eedri: that's why we need a plan 14:14:02 <eedri> dneary, can we request an additional host from rackspace? 14:14:15 <dneary> eedri, It's a question of budget 14:14:15 <ewoud> eedri: given the trouble we had with getting running, would that be better this time? 14:14:16 <eedri> dneary, it will reduce risk and downtime for migrating our DC to gluster storage 14:14:31 <ewoud> danken: I think that's possible, but is it for every EL6 host? 14:14:31 <dneary> eedri, Do you know how much it would cost? 14:14:36 <eedri> dneary, if only as temporaray 14:14:42 <eedri> dneary, no idea 14:15:00 <knesenko> #action knesenko create a plan for gluster and engine installation/migration for rackspace servers 14:15:04 <knesenko> next .... 14:15:07 <dneary> eedri, We can take it up with Red Hat IT; we could also take it up with the board 14:15:10 <knesenko> anything else on hosting ? 14:15:16 <eedri> danken, nighlies builds are for projects built from jenkins 14:15:25 <dneary> We'd need to document everything that's on the existing servers first, to show they're all over-subscribed 14:15:35 <eedri> danken, since we're not building gluster, i don't think that the place for it. 14:15:55 <eedri> danken, users should have another repo enabled with those (we can add it to stable/testing( 14:16:11 <knesenko> #topic Foreman and Puppet 14:16:18 <knesenko> ewoud: dcaro any news here ? 14:16:26 <eedri> dneary, yes, any specific contact? 14:16:40 <dneary> eedri, I've been dealing with Jim Strong in the past 14:16:43 <knesenko> ewoud: dcaro any chance we merge this 0- http://gerrit.ovirt.org/#/c/16907/ :) :) 14:16:45 <dneary> We should CC Itamar too 14:16:46 <eedri> dneary, same here 14:16:58 <dcaro> knesenko: not on my side, I played a littel with r10k, but haven't finished anything yet 14:17:06 <ewoud> knesenko: same here 14:17:20 <knesenko> so lets merge the patch ... 14:17:21 <eedri> #action eedri to query on request an additonal server on rackspace for migration to gluster DC 14:17:22 <knesenko> :) ? 14:17:28 <knesenko> eedri: +1 14:17:47 <knesenko> eedri: update me with the answer ... or cc me to the email ... 14:17:47 <dneary> We could also ask on the board list 14:18:29 <Yamakasi> mburns: yes I looked into that but I was not able to figure it out 14:18:43 <Yamakasi> mburns: some sort of screencast ? 14:18:55 <knesenko> #topic Jenkins 14:18:57 <danken> eedri: frankly, I find this convincing. If you work with nightlies, you're probably crazy enough to download gluster yourself. 14:19:01 <danken> eedri: thanks. 14:19:09 <knesenko> eedri: dcaro obasan news here ? 14:19:17 <mburns> Yamakasi: basically, yes 14:19:34 <knesenko> ewoud: please merge the ntp patch ... or abandon it :) 14:19:36 <ewoud> danken: I am wondering, since we use the EL6 slaves for all versions if it won't conflict 14:19:47 <eedri> knesenko, we had an open issue on migrating network functional test to per patch 14:19:53 <ewoud> knesenko: will do after the meeting 14:19:58 <dcaro> knesenko: we have a new job for ovirt-log-collector, just checks pep8 14:20:28 <knesenko> dcaro: yes saw it today :) and it works :) 14:20:34 <eedri> danken, anyone picking up on that? i think giseppua isn't available anymore? 14:21:50 <danken> eedri: sorry for not being attentive - I need some context. 14:22:15 <eedri> danken, last week giusspa (sorry for misspelling) enabled the network function tests on jenkins 14:22:26 <eedri> danken, we talked about converting it to run per patch on only network patches 14:22:47 <eedri> danken, so he had an action item to provide us a way to different network patches from others 14:22:57 <eedri> knesenko, we have another issue on jenkins 14:23:29 <eedri> knesenko, http://jenkins.ovirt.org/job/ovirt_db_report_engine/ stopped working after the f18-vm02 from ec2 went offline 14:23:47 <eedri> knesenko, it seems that it used local install / files on that specific vm 14:24:11 <eedri> knesenko, i'm not sure if anyone is still using that job, but we need to send email to engine-devel and ask if it's still relevant 14:24:46 <eedri> in that case, we can either start the vm and restore that data or the owner of the job should create a puppet class to enable running that job on any vm 14:24:58 <knesenko> eedri: yes ... 14:25:00 <dcaro> eedri: I think that job was created by Libor, he also left 14:25:02 <danken> eedri: I see no reason to limit it to network patches. 14:25:07 <knesenko> eedri: are you taling about - /var/lib/dbreport/schemaspy.sh ? 14:25:12 <eedri> knesenko, yes 14:25:24 <knesenko> eedri: ok .... can you send the email regarding the job ? 14:25:29 <danken> eedri: if the test eats too much cpu we can limit its rate to once an hour 14:25:30 <eedri> knesenko, not sure if it's something custom or install 14:25:41 <eedri> danken, it's not cpu i'm worried about 14:25:46 <eedri> danken, it's capacity 14:25:53 <knesenko> #action eedri send email regarding http://jenkins.ovirt.org/job/ovirt_db_report_engine 14:26:00 <eedri> danken, and amount of patches running on jenkins 14:26:09 <knesenko> other issues regarding jenkins ? 14:26:35 <danken> so limit this job's rate to once in 2 hours. 14:26:55 <knesenko> #topic Other Issues 14:26:58 <eedri> danken, i'm not sure we can, dcaro ^ 14:27:03 <knesenko> ok guys .... other issues ? 14:27:05 <eedri> knesenko, wait, there's another jenkins issue 14:27:17 <eedri> knesenko, i've enabled another per patch job - engine unit tests 14:27:37 <eedri> knesenko, so now we have 3 verifies on each patch (findbugs, unit tests, checkstyle) 14:27:53 <eedri> knesenko, actually we can disable checkstyle if unit tests now runs 14:28:02 <knesenko> eedri: sounds good 14:28:29 <dcaro> eedri: danken: I don't know of any way of limiting the number of job runs per hour, not sure if it's useful either, will you queue them ot just discard the ones out of quota? 14:28:56 <eedri> dcaro, danken i agree, i think jenkins will trigger a job once a gerrit event will happen 14:29:52 <knesenko> anything else guys ? 14:31:19 <eedri> knesenko, trac tickets? 14:32:11 <knesenko> I don't think that somebody touched tickets 14:32:45 <knesenko> eedri: we can check the status of the assigned tickets 14:32:47 <eedri> knesenko, we should probably review them, so if some can be closed 14:32:52 <eedri> knesenko, or are still relevant 14:33:00 <knesenko> I didn't had time to work on my tickets 14:33:24 <knesenko> eedri: ok 14:36:44 <knesenko> #endmeeting