14:02:20 <ewoud> #startmeeting Infra weekly
14:02:21 <ovirtbot> Meeting started Mon Jul 29 14:02:20 2013 UTC.  The chair is ewoud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:21 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:02:28 <ewoud> #chair dneary
14:02:28 <ovirtbot> Current chairs: dneary ewoud
14:02:49 * dcaroest is here
14:05:24 <ewoud> #chair dcaroest
14:05:24 <ovirtbot> Current chairs: dcaroest dneary ewoud
14:05:45 * ewoud failed to send the minutes last week
14:06:11 <dneary> If you send them now, does that mean I have extra minutes in the hour?
14:06:21 <dneary> I could use them
14:07:10 <ewoud> dneary: mailing now
14:07:15 <ewoud> http://ovirt.org/meetings/ovirt/2013/ovirt.2013-07-22-14.02.html is the URL
14:07:37 <dneary> Whee! Extra minutes!
14:07:51 * dneary feels like I'm in a Philip K Dick story
14:08:13 <dneary> OK - outage reports first?
14:08:27 <ewoud> dneary: go ahead
14:09:12 <dneary> I noticed the disk on linode01 filled up again early last week, and I got a report that emails were not being sent from the wiki. Were these the same issue, or is the wiki email still an issue?
14:09:40 <dneary> Also, who cleaned up the hard disk again? And do we have a solution for the hard disk problem ready to execute now?
14:09:45 <ewoud> dneary: yes, there was an issue and it was resolved by removing the unpacked backup of gerrit
14:10:13 <ewoud> dneary: dcaroest cleaned up the backup and now there's 29G free
14:10:21 <ewoud> /dev/xvda              59G   31G   29G  52% /
14:10:30 <dcaroest> \o/ 50%
14:10:30 <dneary> ewoud, OK - that soukld be good for a couple of weeks
14:10:37 <eedri> +1
14:10:42 <ewoud> #chair eedri
14:10:42 <ovirtbot> Current chairs: dcaroest dneary eedri ewoud
14:10:45 <dneary> What does "cleaned up" mean? Deleted? Compressed? Moved elsewhere?
14:10:50 <eedri> finnaly some wiggle room
14:11:10 <dcaroest> dneary: deleted as we already have tose directories backed up ina compressed manner
14:11:25 <ewoud> #info disk space issue on linode earlier this week, solved by removing unpacked gerrit backups and now we're back to 48% free space (29G)
14:12:09 <ewoud> eedri: there's an action item to remove the f17 nightlies from jenkins
14:12:17 <ewoud> eedri: any progress there?
14:12:46 <dneary> I must admit that in a world where a terabyte of data costs a few hundred dollars now, I'm mystified as to why we spend so much time looking for gigabytes
14:13:08 <eedri> ewoud, done
14:13:22 <eedri> ewoud, no more f17, only f18,f19 & el6
14:13:47 <dneary> And the email issue from the wiki - is that resolved too, or was it a different issue?
14:13:52 <ewoud> eedri: ok, because the cluster that slave is running in will be phased out so I either have to move or remove that guest
14:13:56 <dneary> The error was "relay access denied"
14:13:56 <eedri> ewoud, i think you can reinstall your host as f18
14:14:02 <dneary> which doesn't look like a disk full error
14:14:25 <eedri> ewoud, you mean you'll need to take that slave offline?
14:15:23 <ewoud> eedri: yes, $employer is shutting down the last RHEV clusters
14:15:42 <eedri> ewoud, sorry to hear, OK.
14:15:55 <eedri> ewoud, we're working on adding additional f18 & el6 vms on rackspace soon
14:15:56 <ewoud> eedri: me too, but we no longer manage our own virtualisation
14:17:08 <ewoud> eedri: so how do we proceed on phasing it out? disable in jenkins, remove from puppet and delete?
14:18:27 <ewoud> eedri: and what's the progress on the addtional slaves on rackspace?
14:19:13 <dcaroest> ewoud: I can answer that, I created the fedora 18 template and a first vm (not confiogured yet). This week we should have at least the fedora 18 slaves and the centos template
14:19:35 <eedri> ewoud, dcaroest is working on that afaik
14:19:43 <eedri> ewoud, right now we have 4 f19 slaves running
14:19:59 <eedri> ewoud, so in terms of os support, you can remove it, not sure about current jobs load
14:20:58 <ewoud> #info dcaroest created a fedora 18 template and is working on centos6 and expects slaves to be added this week
14:21:38 <ewoud> eedri: with the rackspace hosts filled, will we have sufficient power?
14:23:03 <eedri> ewoud, i believe so
14:23:11 <eedri> ewoud, we have also one amazon vm disabled
14:23:19 <eedri> ewoud, so we can enable it if needed
14:23:27 <ewoud> eedri: good
14:23:37 <eedri> dneary, i remember there was planning for a 3rd host on rackspace, right?
14:23:50 <dneary> eedri, I'm afraid I don't know
14:23:53 <dneary> Still catching up
14:23:59 <eedri> dneary, will have to check on that
14:24:03 <dneary> Please do
14:24:20 <eedri> dneary, ewoud i think we'll be good for a while once all vms on rackspace are up
14:24:31 <dneary> ewoud, You're moving away from RHEV?
14:24:32 <dcaroest> eedri: I do not remeber that neither
14:25:27 <ewoud> dneary: yes, that decission was made a long time ago and I'll gladly rant about it over a beer, but is a bit offtopic now
14:26:08 <ewoud> speaking of architecture, last meeting we spoke about having a google hangout to address the 'how, what, where'
14:26:52 <ewoud> I was supposed to send a mail about it, but should I also plan it or can I wait for Kiril?
14:26:57 <dneary> ewoud, OK
14:27:47 <dneary> ewoud, My primary concern is: will we have a full disk while I'm on vacation, either on openshift or on linode01?
14:27:58 <dneary> ewoud, So, if we won't, it can wait :-)
14:28:49 <eedri> ewoud, i would say let's wait
14:29:13 <ewoud> dneary: I don't expect linode, not sure about openshift
14:29:23 <dneary> Just checking stats now
14:29:30 <dneary> I don't understand quota output
14:29:42 <dneary> Disk quotas for user 847edb45aea84198838f915be6faa066 (uid 3689):
14:29:42 <dneary> Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
14:29:42 <dneary> /dev/mapper/EBSStore01-user_home01
14:29:42 <dneary> 3841536       0 5242880            9016       0  200000
14:30:09 <ewoud> dneary: try quota -s
14:30:28 <ewoud> 3752M / 5120M
14:30:33 <dneary> So, if I'm reading that correctly, we're using 3.8M blocks and have a quota of 5.2M blocks
14:30:47 <dneary> So using 1.9GB, of a quota of 2.6GB
14:31:07 <dneary> ewoud, I'm assuming they're 512K blocks
14:31:08 <ewoud> though I'd like some trending on that
14:31:17 <dneary> Not 512K
14:31:20 <dneary> 512 byte
14:31:44 <ewoud> dneary: -s shows it human readable and that points to a 3.7G in use of a 5G quota
14:31:45 <dneary> Ah, they are 1K blocks
14:31:59 <dneary> We should be good for 3 weeks
14:32:27 <ewoud> dcaroest: weren't you working on trending with ganglia?
14:33:22 <dcaroest> ewoud: yep, I installed a server with graphite on it, but I wanted to set up the clients using puppet, I have to retake that issue
14:33:22 <ewoud> so, is there any more about hosting?
14:33:51 <eedri> ewoud, we should plan migration on resources.ovirt.org to alterway, but kiril said he'll do it when he gets back
14:34:32 <ewoud> eedri: ok
14:34:53 <eedri> ewoud, btw, any other servers we plan to migrate / create on alterway?
14:34:59 <eedri> ewoud, like gerrit?
14:35:45 <ewoud> eedri: not sure if we want to first re-organize the ovirt setup to a cluster there
14:36:17 <ewoud> eedri: I'd like to move jenkins master to a rackspace VM so we can have a proper HA cluster
14:36:58 <ewoud> not sure if we can given we have no shared storage and I don't know enough about gluster performance/reliability in production but I'd like to
14:38:29 <eedri> ewoud, i think the idea was to keep all production vms on alterways, and keep rackspace to stateless slaves
14:38:44 <eedri> ewoud, so if we want to migrate jenkins, we should do it to alterway02 ovirt instance
14:39:08 <ewoud> eedri: or that
14:39:16 <dcaroest> seems reasonable to me
14:39:42 <dcaroest> ovirt does not need gluster to give HA right?
14:39:56 <ewoud> dcaroest: no, but you need some kind of shared storage
14:40:16 <ewoud> and if you use NFS from one of the hosts, you don't win anything in terms of reliability
14:40:24 <dcaroest> ewoud: true, I think we are using nfs
14:41:45 <dcaroest> ewoud: well.. you win if the one that goes down is jenkins itself. But yes, you'll rely on one of the hosts (moving the single point of failure from the jenkins machine to one of the hsots)
14:42:57 <ewoud> I think we should leave this to the architecture discussion
14:43:42 <dcaroest> ewoud: sure
14:45:35 <ewoud> I'm taking a look at the ticket list
14:46:13 <ewoud> https://fedorahosted.org/ovirt/ticket/36 Fix PHP errors in theme
14:46:20 <ewoud> dneary: do you know if this was fixed?
14:46:40 <dneary> ewoud, Yes, several weeks ago
14:46:59 <ewoud> good, then we can close it
14:47:01 <dneary> We have some other errors but all of thoise in this ticket are fixed.
14:48:03 <ewoud> dneary: did you have any time to look at using piwik to monitor stats?
14:48:10 <ewoud> https://fedorahosted.org/ovirt/ticket/44
14:48:41 <dneary> ewoud, I have not
14:49:00 <ewoud> https://fedorahosted.org/ovirt/ticket/31 Licensing (rights) link disappears when graphic is not used
14:49:09 <ewoud> I'm not even sure what it means
14:49:25 <ewoud> https://fedorahosted.org/ovirt/ticket/39 Upgrade MediaWiki Strapping theme
14:49:34 <ewoud> dneary: I think that's the upgrade that was done some time ago?
14:49:36 <dneary> ewoud, Other issues still on the plate are renaming the mailing list, protecting against robots (I believe that's on dcaro's plate), and I've abandoned uploading lost archives,
14:49:49 <dneary> Yes - the upgrade and the PHP errors were the same thing
14:50:58 <ewoud> dneary: https://fedorahosted.org/ovirt/ticket/25 can be closed then?
14:51:17 <dneary> ewoud, Yes - WONTFIX
14:52:19 <ewoud> https://fedorahosted.org/ovirt/ticket/21 Search field made visible when window is slim
14:52:47 <ewoud> looks good to me, so close?
14:53:08 <dneary> I also want to take care of SSL for oVirt domùains (or ensure it gets taken care of)
14:53:22 <ewoud> dneary: feel free to claim the tickets
14:54:14 <dcaroest> dneary: I have it assigned... but it's a little low in my priority list :S
14:54:15 <dneary> ewoud, dcaroest is a good man for the SSL issue
14:54:33 <dneary> dcaroest, Let me know if you have any issues, I've gone through the process internally before
14:54:40 <dcaroest> dneary: ok
14:54:57 <ewoud> we should also discuss how we want to use trac when we do the architecture
14:55:23 <ewoud> and if we even need the version component
14:56:18 <eedri> guys sorry, i have to go
14:56:22 <eedri> i'll catch up later
14:56:24 <ewoud> eedri: no problem
14:56:35 <ewoud> I want to close #31 with WONTFIX unless there's more info
14:56:46 <ewoud> open for 6 months, so it can't be that critical
14:57:05 <dcaroest> ewoud: which one is it?
14:57:17 <ewoud> https://fedorahosted.org/ovirt/ticket/31 Licensing (rights) link disappears when graphic is not used
15:00:16 <dcaroest> ewoud: no idea of it, and not too familiar with the wiki code neither. But I can try to take a look at it if I have some spare time
15:00:50 <ewoud> dcaroest: I closed it now, so re-open if you want to take a look at it
15:00:59 <dcaroest> ewoud: ok
15:01:34 <ewoud> that leaves 21 open issues, quite a few jenkins so let's take a look at those next week
15:02:26 <dcaroest> ewoud: perfect
15:02:48 <ewoud> I'd like to close the meeting now, unless there are more issues
15:04:36 <dcaroest> ewoud: shall we count down?
15:04:55 <ewoud> going once
15:05:00 <ewoud> going twice
15:05:09 <ewoud> #endmeeting