13:55:21 <knesenko> #startmeeting oVirt Infra meeting
13:55:21 <ovirtbot> Meeting started Mon Jul  1 13:55:21 2013 UTC.  The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:55:21 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:55:32 <knesenko> #chair obasan
13:55:32 <ovirtbot> Current chairs: knesenko obasan
13:55:39 <knesenko> #chair eedri dcaro
13:55:39 <ovirtbot> Current chairs: dcaro eedri knesenko obasan
13:55:56 * eedri here
13:56:12 <eedri> ewoud, can you attend it?
13:57:08 <knesenko> #topic Hosting
13:57:39 * obasan here
13:57:39 <knesenko> #chair obasan
13:57:39 <ovirtbot> Current chairs: dcaro eedri knesenko obasan
13:57:50 <knesenko> ok lets start guys
13:58:09 <knesenko> in a few minutes
13:58:43 <eedri> dneary, around?
13:58:50 <eedri> dneary, if you want to attend the meeting
13:59:02 <dneary> eedri, Yup, here
13:59:16 <eedri> #chair dneary
13:59:16 <ovirtbot> Current chairs: dcaro dneary eedri knesenko obasan
13:59:31 <knesenko> eedri: want to lead the meeting ?
13:59:54 <eedri> knesenko, no, go ahead
13:59:57 <knesenko> eedri: ok
14:00:06 <knesenko> ok lets start then
14:00:24 <knesenko> I have some updates regarding the hosting
14:00:35 <knesenko> dcaro: found some issue on rackspace01 ...
14:00:56 <knesenko> seems like each reboot , the default gateway was deleted somehow ...
14:01:04 <knesenko> I"ll need to retest it ...
14:01:50 <eedri> knesenko, did you tried disabling the firewall?
14:01:51 <knesenko> I will test the iptables rules and the default gateway issues + reboot . if everything will work ok - I will install ovirt_engine + gluster service on them
14:02:18 <eedri> knesenko, if rackspace has external firewall, maybe the local ip tables is not needed
14:02:25 <knesenko> eedri: I can ... but I am not sure that we are allowed to do it
14:02:41 <eedri> dneary, ?
14:02:46 <eedri> dneary, what do you think?
14:02:52 <dcaro> eedri: no need neither, the iptables/firewalld are ok, the problem is just the gateway
14:02:54 <dneary> eedri, Following along
14:03:00 <dneary> Afraid I have no advice to offer
14:03:08 <dcaro> that's why it works from the internal hosts (same network)
14:03:15 <knesenko> eedri: it will be great if we just can disable iptables rules and work without them
14:03:25 <dneary> knesenko, eedri: Is NM managing the networking there?
14:03:33 <eedri> knesenko, i think we can do whatever we want, it's our servers to manage, just need to be sure the external firewall is working
14:03:50 <knesenko> eedri: external firewall is working
14:03:52 <ewoud> eedri: not actively, highlight if you need me
14:04:11 <eedri> dneary, NM?
14:04:29 <dneary> Network Manager
14:04:42 <eedri> knesenko, ?
14:04:54 <knesenko> eedri: let me check
14:05:11 <knesenko> dneary: eedri yes its up
14:06:21 <dcaro> knesenko: we have this in the net config for em1: NM_CONTROLLED=no
14:07:08 <knesenko> dcaro: right ... so actually we don't use it
14:08:54 <knesenko> anything else on hosting ?
14:08:56 <eedri> knesenko, let's send a ticket to rackspace querying about disabling the iptables service
14:09:20 <knesenko> eedri: we can update the existing one
14:09:35 <eedri> new alterway contact - Hervé Leclerc (CCed) will be taking over from Kévin Mazière
14:09:46 <knesenko> dcaro: can you ask rackspace if we can disable the iptables rules on the hosts ?
14:09:59 <dcaro> knesenko: sure
14:10:12 <knesenko> #action dcaro  ask rackspace if we can disable the iptables rules on the hosts
14:10:26 <knesenko> dcaro: ok good . thanks
14:10:40 <eedri> #info new alterway contact herve.leclerc@alterway.fr ( Hervé Leclerc )
14:10:46 <knesenko> dcaro: what about the graphite ? any progress there ?
14:11:48 <dneary> dcaro, OK thanks - in that case the default gateway is set in /etc/sysconfig/network-scripts/ifconfig-em1
14:11:59 <dneary> Or something similar to that (that path was from memory)
14:12:18 <eedri> dneary, i think usualy it's ifcfg-em1
14:12:42 <dcaro> knesenko: nop, lack of time, will try to address it this week
14:12:46 <knesenko> eedri: we have ovirt-engine on that host ... so in that case its ifcfg-ovirtmgmt
14:12:52 <knesenko> dcaro: ok thanks
14:12:57 <eedri> knesenko, ok
14:13:03 <dcaro> dneary: almost, /etc/sysconfig/network-scripts/ifcfg-em1
14:13:07 <dcaro> :)
14:13:15 <knesenko> obasan: what about openshift quota ?
14:13:48 <obasan> knesenko, that won't be a problem. I found the command that does it
14:13:58 <obasan> knesenko, if we want to check quota we can use an ssh command and get the result
14:14:09 <eedri> obasan, can you add it to our monitoring server then?
14:14:17 <obasan> eedri, yes
14:14:20 <knesenko> obasan: good
14:14:47 <knesenko> #action obasan add monitoring openshift quota to our monitoring server
14:15:07 <knesenko> does anybody knows how can i remove items from actions ?
14:15:17 <eedri> undo?
14:15:23 <eedri> --#undo
14:15:51 <eedri> http://wiki.debian.org/MeetBot
14:16:14 <knesenko> eedri: I have this action from the previous meeting - ACTION: obasan look into monitoring openshift quota
14:16:24 <knesenko> eedri: and I want to remove because its done ... and finished
14:16:48 <eedri> knesenko, i think the meeting summary won't show it unless you add it again
14:16:56 <knesenko> eedri: ok good.
14:17:02 <knesenko> so anything else on hosting ?
14:17:28 <eedri> knesenko, let's move on to jenkins?
14:17:33 <knesenko> ok
14:17:39 <knesenko> #topic jenkins
14:17:49 <knesenko> who wants to start ?
14:17:59 <eedri> knesenko, any update on the error on backup job?
14:18:18 * dneary has some updates on the website from last week - ping me when ready?
14:18:30 <knesenko> eedri: backup was fixed ...
14:18:50 <knesenko> eedri: it will run the full backup on a weekly basis ...
14:19:00 <eedri> #info jenkins config backup job was fixed, changed to backup full back weekly
14:19:07 <eedri> knesenko, thanks
14:19:45 <eedri> last ovirt meeting we agreed to upgrade the only f17 slave to f19
14:19:48 <knesenko> so it should reduce the space on alterway02
14:19:57 <eedri> ewoud, any progress with that?
14:20:05 <eedri> knesenko, yes
14:20:54 <knesenko> also we are lack of resources .... so we need to push rackspace* installation asap
14:21:05 <knesenko> me and obasan will do it today
14:21:10 <eedri> knesenko, +1
14:21:38 <eedri> quaid, around?
14:21:41 <dcaro> +1, last week a couple slaves killed the slave thread because they were out of resources :S
14:22:50 <knesenko> i know I know .... I need to push it ....I am sorry .... but I was blocked on some issues there ....
14:22:57 <knesenko> I hope I will finish it this week ....
14:23:33 <eedri> ewoud, ?
14:23:57 <knesenko> eedri: seens like he is not here
14:24:18 <eedri> knesenko, ok, let's just add it as action
14:24:29 <eedri> #action need to upgrade f17 jenkins slave to f19
14:24:42 <eedri> we'll see later who can pick this up
14:24:51 <knesenko> I don;t think that upgrade will work ... need to reinstall it
14:25:04 <eedri> knesenko, might work f17-f18 then f18-f19
14:25:06 <ewoud> eedri: no, no progress on f17 => f19
14:25:17 <eedri> knesenko, i upgraded the f17 to f18 and it worked
14:25:59 <knesenko> eedri: we can try .
14:26:11 <knesenko> eedri: but having a clean f19, sounds better
14:26:15 <eedri> knesenko, if rackspace can be ready this week, that's best
14:26:21 <dcaro> xd
14:26:23 <eedri> knesenko, that way we can install a f19 vm
14:26:54 <knesenko> eedri: I hope it will be ready today
14:26:58 <knesenko> right obasan ?
14:27:11 <dcaro> ok, we can focus then on getting rackspace ready, knesenko obasan, if you need help just tell me
14:27:16 <knesenko> :)
14:27:26 <knesenko> dcaro: ok thanks
14:27:57 <eedri> dneary, ?
14:28:03 <dneary> eedri!
14:28:05 <eedri> dneary, you had updates on the site right?
14:28:10 <dneary> Yup
14:28:18 <dneary> #topic oVirt website
14:28:39 <dneary> As you all noticed, we have had some outages and performance issues over the past couple of weeks
14:28:50 <dneary> Last week I spent some time digging into them to understand
14:29:13 <dneary> Summary (much of this has gone to the mailing list already, but there are a few topics for discussion)
14:29:50 <dneary> # Thanks to garrett we have now got an upgraded Strapping theme with many fixes - we should now be in less danger of filling up the disk in error logs
14:30:04 <dneary> Also plugged 3 security holes at the same time
14:30:19 <garrett> and improved the responsive layout for phones and tablets
14:30:28 <garrett> (and improved rendering in Internet Explorer as well)
14:31:15 <dneary> # There was an OpenShift issue due to the move from V1 to V2 cartridges in OpenShift - an environment file which was expected in V2, but missing in V1, was causing the environment to be set incorrectly, so the Pear Mail module was not being found - since last week, emails are now working again
14:31:36 <dneary> We can now deploy clear & properly with Mail in deplist.txt
14:32:31 <dneary> # The performance issue - I was connected to our server during a severe slow-down, and noticed a lot of Bot traffic - seemed like there were 2-3 spiders trawling the entire history of the wiki indexing everything all at the same time
14:32:54 <eedri> dneary, did you manage to block it via httpd/robot.txt?
14:32:57 <dneary> So - this is the action/discussion part - we need to figure out how to defend ourselves against misbehaving spiders
14:33:02 <dneary> eedri, No
14:33:05 <dneary> Nothing done yet
14:33:15 <eedri> dcaro, has some experience with it on gerrit
14:33:33 <eedri> dneary, also, we can consider adding monitoring graphs to ovirt.org
14:33:40 <eedri> dneary, and check load statistics
14:33:45 <dneary> I have a bunch of promising links, but I need to learn more about the area
14:33:50 <dneary> http://www.robotstxt.org/orig.html#code
14:34:00 <dneary> http://www.thesitewizard.com/apache/block-bots-with-htaccess.shtml
14:34:15 <eedri> dcaro, how did you block the bots from gerrit.ovirt.org?
14:34:28 <dneary> Need some MediaWiki specific recipes
14:34:30 <dcaro> dneary: actually we have all the robots blockd under gerrit.ovirt.org, specially bing was loading too much the server
14:34:50 <dneary> dcaro, we do want wiki pages to be indexed by Google
14:35:01 <dneary> dcaro, But not the entire history - just the latest revision
14:35:38 <dneary> One last thing
14:35:39 <dcaro> dneary: I hope they have a common format
14:36:11 <dneary> dcaro, I bey you can just filter out anything that calls index.php directly
14:36:32 <dneary> dcaro, They all have "revision=" GET queries for history
14:36:40 <dneary> OK - last point:
14:36:59 <dneary> # Our wiki bot appears not to be running any more on resources.ovirt.org
14:37:11 <dneary> Rydekull set it up, but I don't know how to restart it
14:37:25 <dneary> There are errors for every new page revision in error.log now
14:37:43 <dneary> We should either disable the bot, or ensure it's running on resources
14:38:28 <dcaro> looks like a monitoring task
14:38:40 <dneary> Yes
14:38:47 <dneary> For all of these we need monitoring
14:39:08 <dneary> And we need a full list of all of the servers we have & services they're running & how to stop & restart them and so on
14:39:38 <dneary> In general, I think we need to have someone (as per quaid's email last week) to take on the mantle of infra team co-ordinator
14:39:41 <dcaro> also what we want to monitor and when to restart them (hugh load, error log...)
14:40:03 * dneary looks at dcaro, eedri
14:40:16 <dneary> Do either of you have some time to whip everything into shape?
14:40:18 <eedri> can we add a wiki page on infra on what we are currently monitoring?
14:40:38 <dcaro> xd, I can try to give it some time
14:40:44 <eedri> dneary, obasan installed a monitoring server lately, we can look into adding more services to monitor
14:41:01 <eedri> dneary, can you add it as task to the trac?
14:41:15 <eedri> dneary, so we won't forget about it, which services we want to monitor,etc...
14:41:39 <dneary> eedri, Just to be precise, what's "it" in your question above?
14:41:59 <eedri> dneary, :), the list of servers/services we want to monitor
14:42:38 <dneary> eedri, OK
14:42:48 <eedri> dneary, from your perspective (i.e ovirt.org or the service that didn't work on resources.ovirt.org)
14:42:59 <eedri> dneary, we can update that ticket as we go along with more services that come into mind
14:43:04 <dneary> eedri, New Trac, or a comment on the monitoring ticket?
14:43:20 <eedri> dneary, if there is already a ticket, then a new comment will be good :)
14:43:42 <eedri> obasan, can you review the monitoring ticket later and see what can be added?
14:43:52 <obasan> eedri, yes
14:44:49 <eedri> #action need to add monitoring to ovirt.org load/bots
14:45:25 <eedri> #action look into blocking overloading bots on wiki page
14:45:45 <eedri> dneary, anything else on ovirt site?
14:46:25 <mburns> did dneary mention the missing new project incubation page?
14:47:24 <eedri> mburns, not that i recall. mainly on outage and upgrades
14:48:10 <knesenko> ok guys anything else ?
14:48:32 <eedri> knesenko, we should quicky scan/review trac tickets
14:48:32 <knesenko> lets move to puppet then ?
14:48:37 <eedri> knesenko, sure
14:48:46 <knesenko> eedri: lets do it in the end ?
14:48:57 <eedri> knesenko, +1
14:48:58 <knesenko> in the last topic
14:49:04 <knesenko> #topic puppet
14:49:11 <knesenko> dcaro: any news here ?
14:49:35 <dcaro> knesenko: not on my side :S, still have to catch up
14:49:46 <eedri> knesenko, maybe ewoud can update
14:50:01 <knesenko> ewoud: ? :)
14:50:07 * ewoud has been lacking time
14:50:12 <eedri> ewoud, do we have a running puppet master ?
14:50:23 <ewoud> eedri: we do, and we manage user accounts with it for a few admins
14:50:24 <eedri> ewoud, so new rackspace vms can use..
14:50:25 * quaid lurking now if there are any open questions for him
14:50:36 <ewoud> eedri: there's even a git repo on gerrit
14:50:40 <eedri> ewoud, ok, so what is missing is just new puppet classes...
14:50:56 <knesenko> dcaro: we will need your help to add rackspace* servers to the puppet server
14:50:57 <ewoud> eedri: exactly
14:51:07 <eedri> knesenko, ewoud let's try to push new classes after the rackspace install
14:51:26 <eedri> knesenko, we'll see what's missing and add it (ntp/nfs/services/etc...)
14:51:27 <ewoud> I'd like to install nrpe on each machine using puppet and generate the icinga config
14:51:44 <eedri> i can help with it if needed
14:51:49 <knesenko> obasan: ^^ :)
14:52:08 <eedri> obasan, can you send nrpe puppet class for review?
14:52:29 <obasan> eedri, I already sent one. and then ewoud introduced an improved one
14:52:37 <eedri> obasan, ok, so it's merged?
14:52:45 <ewoud> eedri: no, we should add it to our infra
14:52:55 <ewoud> eedri: and it doesn't use exported resources yet
14:53:10 <ewoud> I'm a bit unfamiliar with how nagios/icinga configs are writen
14:53:16 <eedri> ewoud, oh... so after we merge it, there is still a manual process to load it to foreman?
14:53:23 <ewoud> eedri: yes
14:53:36 <eedri> ewoud, can't we just add a cron job to do it?
14:53:47 <eedri> dcaro, ?
14:53:57 <ewoud> eedri: we could replace that by jenkins
14:54:09 <eedri> ewoud, +1
14:54:10 <dcaro> eedri: ewoud: yep
14:54:18 <eedri> ewoud, jenkins job is already better than a local/hidden cron
14:54:18 <ewoud> in essence it's just on gerrit merge a git push
14:54:51 <eedri> dcaro, can you add it to jenkins?
14:54:52 <ewoud> we just need to add the jenkins public key to puppet@foreman and define the job
14:55:09 <dcaro> eedri: ok, I'll crate a ticket for me
14:55:15 <eedri> dcaro, thanks
14:55:18 <knesenko> dcaro: good
14:55:28 <ewoud> did you guys talk about quaids mail?
14:55:29 <eedri> #action dcaro to add jenkins job to load new puppet classes to foreman
14:55:42 <eedri> ewoud, no, you mean the ovirt.org cert?
14:56:09 <knesenko> I think he ewoud talks about the project coordinator
14:56:10 <ewoud> eedri: no, the project leader
14:56:12 <ewoud> that
14:56:21 <ewoud> I have yet to respond to it, but I fully agree with it
14:56:34 <eedri> ewoud, i didn't respond yet as well
14:56:45 <ewoud> I'd also like to note that I'm not the ideal person for it
14:57:00 <knesenko> eedri: we need to raise it up and discuss it with our manager
14:57:43 <eedri> knesenko, ewoud let's continue the discussion on the ml and see who can take lead on it
14:57:50 <ewoud> eedri: let's
14:57:50 <knesenko> eedri: ok
14:57:59 <knesenko> anything else on puppet ?>
14:58:27 <knesenko> #topic other business + review tickets
14:58:41 <knesenko> ok lets review upstream tickets
14:58:50 <knesenko> https://fedorahosted.org/ovirt/report/1
15:00:05 <knesenko> lets start with the jenkins component tickets
15:00:17 <knesenko> https://fedorahosted.org/ovirt/ticket/59
15:00:37 <knesenko> oh that mine :) I will handle it
15:00:46 <knesenko> https://fedorahosted.org/ovirt/ticket/43
15:01:06 <knesenko> this one ^ is related to F19 upgrade
15:01:19 <knesenko> do we have a volunteer here ?
15:01:25 <eedri> knesenko, yes, so if we finish rackspace this week, 1st vm should be f19
15:01:26 <dcaro> and to the rackspace servers
15:01:36 <eedri> knesenko, otherwise we'll upgrade the existing f17
15:01:50 <knesenko> eedri: ok ... so I will take this ticket as well
15:02:12 <knesenko> next
15:02:13 <knesenko> https://fedorahosted.org/ovirt/ticket/51
15:02:25 <eedri> also pending new vms on rackspace
15:02:29 <knesenko> eedri: seems like you are working on it right ?
15:02:34 <eedri> and possible baremetal server
15:02:40 <eedri> to run vdsm tests
15:02:40 <knesenko> eedri: ok . can I assign this ticket to you ?
15:02:47 <eedri> sure
15:03:23 <knesenko> https://fedorahosted.org/ovirt/ticket/6
15:03:56 <eedri> i think foreman is up
15:03:58 <knesenko> I think that we did most here ... puppet and foreman are up and running
15:04:00 <eedri> and puppet as well
15:04:06 <knesenko> need to add new classes to the puppet ...
15:04:11 <dcaro> yep, basic installation is running
15:04:11 <knesenko> so we can close it then ?
15:04:20 <eedri> knesenko, i'd close it and open a new one
15:04:26 <eedri> for puppet classes needed
15:04:27 <knesenko> eedri: ok
15:04:59 <knesenko> https://fedorahosted.org/ovirt/ticket/17
15:05:29 <knesenko> ok we have a backup for jenkins ....what about other services ?
15:05:33 <eedri> dneary, ?
15:05:43 <eedri> dneary, openshift/media wiki is backedup?
15:05:47 <dneary> eedri, I do not know
15:05:56 <knesenko> :\
15:06:04 <dneary> eedri, quaid may know
15:06:12 <eedri> quaid, ?
15:06:14 <knesenko> ok ... I think that we should open a thread on it on ML ...
15:06:30 <eedri> knesenko, agree, can you send email on it?
15:06:31 <knesenko> we need a defined server/storage to store there our backups
15:06:36 <knesenko> eedri: yes
15:07:02 <knesenko> #action knesenko send email regarding the backups . related to https://fedorahosted.org/ovirt/ticket/17
15:07:12 <knesenko> https://fedorahosted.org/ovirt/ticket/28
15:07:31 <dneary> knesenko, I agree
15:07:49 <knesenko> quaid: any update on https://fedorahosted.org/ovirt/ticket/28 ?
15:08:18 <knesenko> dneary: ^^ maybe you know ?
15:08:24 * quaid back
15:08:58 <quaid> hmm, I'm drawing a blank on mediawiki backup
15:09:29 <quaid> knesenko: about SSL, I just asked dcaro & eedri if they can finish that one for me, the IT group is ready to get the CSR and have it signed, then we drop it in place everywhere, etc.
15:09:32 <dneary> quaid, knesenko: I've gone through the SSL process for openstack.redhat.com
15:09:37 <dneary> It's not complicated
15:09:42 <quaid> we worked out the details, but I ran out of time to finish
15:09:53 <quaid> not sure if either can take it over, haven't seen their reply
15:10:11 <knesenko> ok thanks for update ...
15:10:17 <dcaro> quaid: yes, we will :)
15:10:41 <knesenko> dneary: quaid guys if you are lack of time , I think that we can handle it
15:10:57 <quaid> +1 thanks
15:11:01 <knesenko> dcaro: can you take this task ?
15:11:03 <quaid> this new project is eating me up
15:11:25 <ewoud> quaid: if only you could just tell us what is and we could help ;)
15:11:51 <knesenko> quaid: just update the ticket with the info , or email will be great ...
15:11:54 <quaid> ewoud: you know how companies are, they love the Big Reveal :)
15:11:59 <quaid> knesenko: will do
15:12:04 <dcaro> knesenko: ok
15:12:09 <knesenko> dcaro: thanks a lot
15:12:30 * knesenko moving the ticket to dcaro
15:12:49 <knesenko> quaid: please update the ticket + talk to dcaro if he needs any info to finish this task
15:13:01 <knesenko> https://fedorahosted.org/ovirt/ticket/53.
15:13:14 <knesenko> eedri: ^^ ?
15:13:35 <knesenko> eedri: want to handle it ?
15:14:03 <eedri> knesenko, sure. should be easy now that we have backuips
15:14:32 <quaid> *clash*
15:14:35 <knesenko> good
15:14:36 <knesenko> thanks
15:15:04 * quaid puts in ssl notes
15:15:21 <knesenko> https://fedorahosted.org/ovirt/ticket/10
15:15:58 <knesenko> its already assigned to dneary quaid .... moving on
15:16:07 <dneary> It's assigned to me?
15:16:14 <knesenko> https://fedorahosted.org/ovirt/ticket/41
15:16:30 <knesenko> dneary: to quaid
15:16:38 <knesenko> dneary: you reported
15:16:42 <knesenko> https://fedorahosted.org/ovirt/ticket/41
15:16:42 <dneary> Yup
15:16:57 <eedri> 41 i think is pending dcaro patches to be merged
15:17:06 <knesenko> aha ...
15:17:13 <knesenko> so I will assign it to dcaro then
15:17:14 <quaid> if we can confirm that any open tickets that I own are actually for me/best done by me, I can just schedule to do them -- that's the sort of proj management I'm needing :)
15:17:19 * quaid looks to see what he howns
15:17:23 <dneary> OK - quaid, we should figure out who will do that list rename. I'm ready to give it a go, but would want to have a fallback plan in place if things go wrong
15:17:42 <dcaro> knesenko: yes please
15:18:28 <knesenko> https://fedorahosted.org/ovirt/ticket/42
15:18:36 * quaid sees he has only one ticket anyway
15:18:48 <knesenko> quaid: seems like this one ^^ is related to SSL ticket right ?
15:19:01 <quaid> ah-ha, yes
15:19:05 <knesenko> ok
15:19:36 <knesenko> quaid: please update more info on it , so we can handle it from there . thanks
15:19:37 <knesenko> next
15:19:54 <quaid> dneary: perhaps we can coordinate so I'm available in case you have problems, etc.? i.e. pick a time early in my day when we have crossover of a few hours
15:20:23 <knesenko> quaid: can you handle mailing lists tasks ? can i assign them to you ?
15:20:34 <dneary> yeah
15:20:42 <dneary> perhaps this week?
15:20:59 <knesenko> quaid: there are 2 ...
15:21:11 <quaid> dneary: sure
15:21:26 <quaid> knesenko: which ones are they?
15:21:34 <knesenko> https://fedorahosted.org/ovirt/ticket/25
15:21:43 <knesenko> https://fedorahosted.org/ovirt/ticket/12
15:21:49 <dneary> quaid, Tomorrow is looking promising for me
15:22:19 <dneary> For #25, I think I need quaid's archives - I don't have all of the missing messages
15:24:13 <quaid> I don't either, and that's probably the biggest catch there
15:24:25 <quaid> someone sent them to me but I lost that archive, I'm sure someone else has them though
15:24:30 <knesenko> guys seems like we are lack of time . we will continue to review the tickets on our next meeting.
15:24:33 * quaid has them but all in one mailbox
15:25:01 <knesenko> thank you all . Please make sure to handle your tickets (If you have time)
15:25:13 <eedri> thanks
15:25:21 <knesenko> anything else before I finish this meeting ?
15:25:38 <knesenko> 5
15:25:39 <knesenko> 3
15:25:41 <knesenko> 2
15:25:42 <knesenko> 1
15:25:43 <knesenko> ?
15:25:51 <knesenko> ok thank you all ...
15:26:00 <knesenko> have a nice week
15:26:01 <knesenko> #endmeeting