15:03:22 <quaid> #startmeeting oVirt Infra Weekly 15:03:22 <ovirtbot> Meeting started Mon Mar 4 15:03:22 2013 UTC. The chair is quaid. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:03:22 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic. 15:03:29 <quaid> #topic Roll call & agenda 15:04:30 <quaid> Introductions 15:04:30 <quaid> Hosting 15:04:30 <quaid> Puppet 15:04:30 <quaid> Summarize the ML thread 15:04:30 <quaid> Jenkins 15:04:32 <quaid> Migration to alterway 15:04:35 <quaid> Other business? 15:04:37 <quaid> Trac review 15:04:40 <quaid> <eolist> 15:05:01 <quaid> I see eedri_ & Rydekull here 15:05:18 <eedri_> i think dcaro is on a puppet course today 15:05:26 <quaid> that's useful :) 15:05:33 <eedri_> :) indeed 15:05:34 <Rydekull> is he becoming a puppeteer? 15:05:47 <eedri_> Rydekull, you can ask him when he's back... 15:06:00 * ewoud here now 15:06:01 <eedri_> Rydekull, more like a 'puppet master' 15:06:56 <Rydekull> I'm going to break quite soon and be a bit more afk, even though i'll be connected through a mobile connection 15:06:57 * quaid narrated an improve piuppet show this weekend, about as non-IT as you can get 15:07:03 <Rydekull> but I need to change location and travel by subway :-) 15:07:12 <quaid> #chair eedri_ ewoud Rydekull 15:07:12 <ovirtbot> Current chairs: Rydekull eedri_ ewoud quaid 15:07:39 <quaid> ok, we'll get rolling and see where it goes 15:07:46 <quaid> #topic Introductions 15:07:59 <quaid> anyone around who is new and wants to introduce themselves? 15:08:44 <dneary> hi all 15:08:54 <eedri_> dneary, hi dave! you' 15:08:58 <eedri_> new? :) 15:10:36 <dneary> eedri_, Just learning the ropes 15:10:51 <Rydekull> quaid: I think sufficient time has been given to rewrite a sentence a few times over :-P 15:11:04 <quaid> sorry, moving between rooms :) 15:11:12 <quaid> #topic Puppet 15:11:35 <quaid> aside from dcaro getting training today, what news have we? 15:12:00 <ewoud> I did have an action item to update foreman, but I haven't had time to work on that 15:12:24 <eedri_> ewoud, is the current foreman running on rpms install? 15:12:32 <eedri_> ewoud, so you'd want to update to latest 1.1 stable? 15:12:36 <ewoud> eedri_: yes 15:12:39 <eedri_> ewoud, cool 15:14:10 <quaid> #info update to latest Foreman is in progress 15:14:27 <ohadlevy> ewoud: yum update ? :) 15:14:28 * ohadlevy runs 15:14:36 <ewoud> ohadlevy: already started :P 15:14:38 <quaid> are we installing the puppet master on one of the VMs at Alter Way? 15:14:53 <quaid> meaning we also need the other services running, etc. 15:14:55 <ewoud> quaid: that is the end goal yes, but I don't want to wait for that 15:15:07 * quaid notices he switched topics around, will have something for Hosting next 15:15:37 <quaid> #info Working up Puppet system in pieces, not waiting for perfect end-state, but we will have various useful services running as VMs at Alter Way 15:15:38 <ewoud> but I think we also need to build a base that's installed on every host 15:16:02 <ewoud> such as configure SSH, ensure sudo is installed etc 15:16:10 <ohadlevy> ewoud: do you know that foreman 1.2 supports spice html5 out of the box for ovirt? :) 15:16:24 <ewoud> ohadlevy: nice :) 15:16:59 <ohadlevy> ewoud: yeah, got it to work even on my phone :) 15:17:09 <quaid> heh 15:17:25 <quaid> ewoud: do we have a page or running list of what goes on each host? 15:17:32 <ewoud> quaid: don't think so 15:17:56 <ewoud> quaid: I think related is the topic of how we get new infra members, but I haven't seen a reply on the ML 15:18:25 <quaid> oops, is there a thread I missed? 15:18:56 <Rydekull> Im not sure I see the relation there 15:19:07 <Rydekull> Relation between puppet and how we get new infra members? 15:19:12 * quaid starts a page 15:19:18 <ewoud> http://lists.ovirt.org/pipermail/infra/2013-February/002157.html & http://lists.ovirt.org/pipermail/infra/2013-February/002158.html 15:19:20 <quaid> http://www.ovirt.org/Infrastructure_Puppet_details 15:19:55 <ewoud> Rydekull: I mixed it up, the first is puppet design, second is new member process 15:20:09 <Rydekull> Right 15:21:17 <ewoud> so please reply to the first link and we can work from there 15:21:18 <ovirtbot> 14[[07Infrastructure Puppet details14]]4 N10 02http://www.ovirt.org/index.php?oldid=7811&rcid=8000 5* 03Quaid 5* (+291) 10adding a new page to track what needs to be done by Puppet 15:21:40 <quaid> ok, we can go back to new member process 15:21:52 <eedri_> didn't we write some of it on the etherpad that we made for the new services 15:22:33 <ewoud> eedri_: I vaguely recall something like that, but I wouldn't have a link 15:22:46 * eedri_ looking 15:23:08 <eedri_> http://etherpad.ovirt.org/p/new_hosting_design_Jan_2013 15:23:21 * Rydekull departs, will still respond, just with a longer delay 15:24:00 <ewoud> eedri_: thanks 15:24:24 <eedri_> bless google for browser history... 15:24:53 * quaid thought that was the browser who tracked that 15:25:37 <eedri_> i meant chrome browser... so i guess it's google 15:26:18 <ewoud> ok, can we continue this on the ML? 15:26:32 <quaid> yes, and we can put our answers and stuff in that wiki page? 15:26:54 <quaid> ok, I'm ready to go back and do the Hosting topic, if we're ready 15:27:03 <ewoud> ok 15:27:05 <eedri_> +1 15:27:24 <quaid> #topic Hosting 15:27:46 <quaid> #info quaid now has the base details he needs to be able to install on the RackSpace hosts 15:27:53 <eedri_> +1 15:27:58 <quaid> #info quaid still working on RHEL entitlements for those hosts 15:28:18 <quaid> #info hopeful for later this same week for being able to install, still learning the RackSpace how-to do stuff 15:28:20 <eedri_> quaid, we're going to install rhevm on one of them ? 15:28:37 <eedri_> quaid, we still need jenkins vm slaves to replace current ec2 15:28:38 <quaid> ooh, hmm, I don't know if we discussed that? sounds familiar 15:28:51 <ewoud> 1 thing to note is that RHEL doesn't support nested virtualisation 15:28:59 <eedri_> ewoud, true 15:29:01 <ewoud> eedri_: I thought we wanted ovirt on rackspace as well 15:29:01 <quaid> oh carp, i forgot that 15:29:09 <eedri_> ewoud, i don't mind 15:29:22 <eedri_> ewoud, but i'm not sure we can install ovirt on rhel yet, no? 15:29:32 <ewoud> eedri_: correct 15:29:45 <quaid> so we need to actually get RHEV to have virt 15:30:12 <eedri_> rhev entitelments instead of rhel? 15:30:24 <quaid> right 15:30:30 <ewoud> but with rhev you don't have an all in one 15:30:40 <ewoud> so would you host the manager in a rackspace VM? 15:30:51 <eedri_> ewoud, i think you do, in 3.1 15:30:52 <dyasny> ewoud, sure you do, not supported for production 15:31:58 <quaid> how many entitlements do we need, worst case? 15:32:08 <eedri_> i'd say 3 15:32:18 <eedri_> if we're talking about bare metal servers 15:32:24 <quaid> and at least one RHEL so we can just run that as a slave for Jenkins? 15:32:26 <ewoud> dyasny: ok 15:32:29 <quaid> we've got 2 bare metal there 15:32:32 <eedri_> or do you need the amount of vms we'll run 15:32:47 <quaid> right, how many VMs running RHEL and RHEV? 15:33:09 <eedri_> quaid, when you say RHEV, you mean RHEVH? 15:33:40 <ewoud> but if we want to run node tests don't we want nested virt? 15:33:52 <ewoud> which would argue for fedora 18 + ovirt 15:34:06 <eedri_> ewoud, nested virt is a good option to save "wasting" bare metal hosts 15:34:07 * Rydekull doesnt understand why sd need rhel / rhev at all 15:34:18 <eedri_> ewoud, we'll need that also for testing ovirt auto tests 15:34:23 <ewoud> eedri_: exactly 15:34:27 <eedri_> ewoud, i.e using rest/sdk for adding hosts ,etc... 15:34:56 <eedri_> quaid, i'd say maybe install f18 on one of the racksapce 15:35:07 <eedri_> so we'll be able to use both options 15:35:11 <eedri_> and we won't be limited 15:35:44 <ewoud> eedri_: and on the other just rhel + jenkins slave? 15:35:57 <quaid> Rydekull: primarily, test coverage - being able to test on RHEL is a nice thing; secondarily, a sponsor offered it because they would like to see us running some RHE* in our environment. 15:36:16 <eedri_> ewoud, we'll need multiple jenkins slaves 15:36:23 <eedri_> ewoud, i imagine they will all be vms 15:36:38 <eedri_> ewoud, so they can be eithr on ovirt running on f18 or on rhel running rhevm 15:37:11 <eedri_> ewoud, we might want to configure foreman to dynamicly create vms on demand 15:37:12 <ewoud> I'd prefer ovirt then 15:37:45 <ewoud> I know jenkins has a plugin to create VMs using some hypervisors, maybe we can modify that 15:38:02 <eedri_> ewoud, i started to work on a plugin to integration foreman & jenkins 15:38:15 <eedri_> ewoud, i think david can help pick it up 15:38:17 <quaid> heh, so are we thinking the environemnt should be F18-based instead? 15:38:29 <eedri_> quaid, thinking on our requirements 15:38:47 <eedri_> quaid, nested virt is something important to have, since we're limited on bare-metal hosts 15:38:48 <ewoud> quaid: we'll likely still have RHEL VMs, but the host may be better on F18 15:38:51 <quaid> we can still run some RHEL as slaves 15:39:02 <quaid> right 15:39:10 <eedri_> quaid, and it will take some time till nested will be supported on rhel 15:40:12 <quaid> ok 15:41:03 <quaid> ok, so we're not blocking on entitlements, because we are going to install Fedora 18 for at least one RackSpace host, as that will let us simulate the bare metal experience for other VMs 15:41:09 <Rydekull> im still all for centos 15:41:23 <quaid> does CentOS support nested? 15:41:47 <ewoud> no 15:42:10 <eedri_> Rydekull, you need some free bare-metal hosts or nested virt 15:42:12 <ewoud> CentOS aims to be compatible with rhel so very unlikely they add feature 15:42:17 <eedri_> Rydekull, that won't run ovirt instance 15:42:44 <ewoud> so rackspace01 => f18 + ovirt all-in-one, rackspace02 => ? 15:42:45 <eedri_> Rydekull, in order to run automation jobs or other tasks that vms can't 15:43:12 <Rydekull> seriously. can we please put up a design on what we need and why. and stop revisiting the same discussions? :-) 15:43:12 <eedri_> ewoud, i'd say let's keep it open 15:43:34 <eedri_> Rydekull, it's not revisiting, so far we've disscussted on alterway servers 15:43:47 <eedri_> Rydekull, we didn't really agreed on the purpuse of rackspace, afair 15:43:57 <Rydekull> but why would you run it differently everywhere else? 15:43:58 <quaid> on generally, not in detail 15:44:26 <eedri_> Rydekull, we're trying to see if we've coverting all our needs 15:44:27 <ewoud> Rydekull: nested virtualisation is something that was recently asked on the ML and not something we can do with rhel(-based) in the short term 15:45:13 <eedri_> i think we specified the needs here: http://etherpad.ovirt.org/p/new_hosting_design_Jan_2013 15:46:07 <Rydekull> what prevents us from doing nested virtualisation with centos? 15:46:20 <eedri_> Rydekull, not supported on the os 15:46:34 <eedri_> Rydekull, it's a new feature that just came out on fedora recently 15:48:37 <quaid> ok, but we do need to update our plan from that etherpad and finish a discussion, maybe move back to the mailing list with this since it's going on for a bit? 15:48:39 <Rydekull> i dont understand where "not supported" is coming from. what is the enabler that is missing in centos for nested virtualisation? it must be a kernel thing sikce its a cpucall? 15:50:58 <quaid> Rydekull: i'n ot sure off-hand 15:51:09 * quaid has keyboard weirdness 15:51:31 <quaid> shall we finish the RackSpace design discussions on-list? 15:51:33 <eedri_> Rydekull, it was annonced recently as support only on fedora, i imagine it's release on newer kernel/qemu 15:51:38 <ewoud> Rydekull: kernel doesn't support it 15:51:42 <eedri_> Rydekull, that is not yet avaliable on centos or rhel 15:52:07 <eedri_> can we move on to the next topic and move this disscusson to the list? 15:52:22 <eedri_> quaid, ? 15:52:40 <quaid> +1 15:52:57 <quaid> #action moving discussion to list to finish 15:53:03 <quaid> #topic Jenkins 15:53:13 <quaid> anything we didn't cover yet? 15:53:24 <eedri_> jenkins server has migrated to alterway 15:53:37 <eedri_> jenkins.ovirt.org is now redirecting to alterway01.ovirt.org 15:53:53 <eedri_> once we're satisfied with it, we can change the dns record to point to the new server 15:53:53 <ovirtbot> 14[[07Special:Log/newusers14]]4 create210 02 5* 03DNeary 5* 10DNeary created a user account Theron 15:53:54 <ovirtbot> 14[[07User:Theron14]]4 N10 02http://www.ovirt.org/index.php?oldid=7812&rcid=8002 5* 03DNeary 5* (+29) 10Creating user page for new user. 15:53:55 <ovirtbot> 14[[07User talk:Theron14]]4 N10 02http://www.ovirt.org/index.php?oldid=7813&rcid=8003 5* 03DNeary 5* (+246) 10Welcome! 15:54:31 <quaid> #info Jenkins has migrated to Alter Way 15:54:31 <eedri_> only think i saw that some users might have issues with thier permissions, so i'll publish it on the lists after the meeting 15:54:45 <quaid> #info jenkins.ovirt.org now redirecting to alterway01.ovirt.org 15:54:46 <eedri_> so anyone with issues can contact infra for issues 15:54:53 <quaid> #info once satisified, we can change DNS 15:55:01 <eedri_> openid plugin has be enabled 15:55:02 <quaid> #action quaid to schedule DNS change with eedri_ 15:55:19 <quaid> #info openid plugin is enabled (yay) 15:55:20 <eedri_> each user that has an existing account, can assosicate his user with open id 15:55:31 <eedri_> though it's not without errors unfourtunately 15:55:42 <eedri_> the plugin isn't perfect, i managed to use 'google' provide so far 15:55:53 <eedri_> anyhow, basic auth will still work 15:56:05 <eedri_> new design similar to ovirt.org 15:56:27 <eedri_> if there are any suggestions on colors or ui, i have the custom css so it's possbile to change it 15:57:04 <quaid> cool 15:57:09 <quaid> that's all good news, thank you! 15:57:52 <quaid> anything else on jenkins? 15:58:07 <eedri_> quaid, we need to push forward adding new slaves 15:58:23 <eedri_> quaid, since current ec2 won't be able to hold real jobs per patch from gerrit as we planned 15:58:38 <eedri_> quaid, fastest way is to install ovirt on alterway02 as we planned 15:59:24 <eedri_> also, obasan added new job for ovirt-live builds, that will be published nightly 15:59:36 <ewoud> aren't the rackspace machines already available? 16:00:04 <quaid> ewoud: they are, I didn't get IP & login info until last week though 16:00:07 <ewoud> if so, maybe we can push for f18 fast on at least rackspace01 and wait for EL6 support on alterway02 16:00:51 <quaid> I don't know what is required to do the install yet, I'll start my research but won't be available to focus on it until Wed 16:01:07 <quaid> I'll work on making sure to spread help around ASAP - not sure how to enable others, but I'll look at that too 16:01:09 <ewoud> ovirt all in one shouldn't be that hard 16:01:23 <quaid> should I start witha base F18 install? 16:01:28 <eedri_> i can assist in install if needed 16:01:47 <eedri_> obasan, also has experience with ovirt as well 16:02:00 <ewoud> +1 16:02:27 <obasan> ewoud, pretty simple actually. just install the all in one rpm and run the setup 16:02:39 <ewoud> it's a waste to let them run and do nothing, we can always reinstall if needed 16:02:40 <obasan> I will gladly help 16:02:51 <eedri_> +1 16:02:52 <ewoud> obasan: I think that covers 'not that hard' :) 16:03:02 <quaid> +1 16:03:13 <quaid> I'll plan on doing that on Wed 16:03:19 <eedri_> quaid, i agree.. they can provide us with credentials 16:03:22 <ewoud> only question is if we want to make the 2 machines a cluster with gluster, but we can answer that later 16:03:23 <eedri_> quaid, and we'll do the rest 16:03:33 <eedri_> quaid, just need to take care of storage 16:03:46 <eedri_> quaid, if we're using external storage such as iscsi or fcp 16:03:46 <ewoud> for performance reasons I would go for local storage now 16:03:47 <quaid> yep, storage is a bit open - lots of it available 16:04:18 <quaid> rax01 has the fast SSD on RAID 10 16:05:17 <ovirtbot> 14[[07Intel Workshop May 2013 CFP14]]4 !N10 02http://www.ovirt.org/index.php?oldid=7814&rcid=8004 5* 03Theron 5* (+4032) 10Created page with "Intel oVirt Workshop May 2013 | Call For Participation All submissions must be received by March 29th, 2013 at 11:55pm (PST). Ovirt strives to become the first and best t..." 16:06:17 * theron notes that page is under heavy rewrite 16:06:32 <quaid> no worries, not likely you'll get any competitor writers here right now :) 16:06:47 <quaid> ok, are we done with Jenkins topic? 16:06:59 <eedri_> quaid, so just to understnad 16:07:05 <ovirtbot> 14[[07Features/UIPlugins14]]4 !10 02http://www.ovirt.org/index.php?diff=7815&oldid=6995&rcid=8005 5* 03Vszocs 5* (+356) 10 16:07:19 <eedri_> quaid, what is the next server we're expected to use for jenkins slaves? alterway02 or raxs? 16:07:29 <quaid> #action quaid to install F18 + oVirt all-in-one on rackspace01 by Thursday 7 March 16:07:40 <eedri_> +1 16:08:06 <eedri_> final quesion on alterway01 16:08:13 <ovirtbot> 14[[07Special:Log/upload14]]4 upload10 02 5* 03Vszocs 5* 10Vszocs uploaded "[[02File:Ovirt-monitoring hosts graph.png10]]" 16:08:24 <eedri_> i imagine that it's not that loaded with running jenkind currently 16:08:41 <quaid> but it sounds as if it will be? 16:08:45 <eedri_> what our thoughs on running additional services (as virtual servers) on it? such as artifactory 16:08:51 <eedri_> quaid, can't tell for sure 16:09:02 * quaid sees rax01 as hosting Jenkins Slaves 16:09:30 <ewoud> eedri_: I think we should use something like munin to monitor the load and check it 16:09:30 <quaid> eedri_: RobertM did work last year to show how much load it would take, and it sounded like it was pretty busy. 16:09:40 <Rydekull> i dont see why we are running it on bare metal to start with. and not in a vm. but i say, ru other services aswell 16:09:47 <eedri_> ewoud, it has internal monitoring plugin 16:10:15 <eedri_> Rydekull, we had a disscussion on it, and you might be right. we'll need to monitor it in time and perhaps migrate it to a vm 16:10:19 <ovirtbot> 14[[07Intel Workshop May 2013 CFP14]]4 !10 02http://www.ovirt.org/index.php?diff=7817&oldid=7814&rcid=8007 5* 03Theron 5* (+55) 10 16:10:34 <eedri_> Rydekull, but since we're not too short on servers now (still got 3 to use), it's not top priority 16:10:49 <ewoud> eedri_: maybe something to keep in mind, but a central monitoring service might be useful, especially if it also does trending 16:10:54 <quaid> +1 16:10:55 <eedri_> +1 16:10:59 <eedri_> nagios/ganglia 16:11:03 <eedri_> can be great 16:11:17 <eedri_> ewoud, and we have puppet classes ready for them already :) 16:11:25 <ovirtbot> 14[[07Intel Workshop May 2013 CFP14]]4 !10 02http://www.ovirt.org/index.php?diff=7818&oldid=7817&rcid=8008 5* 03Theron 5* (+1) 10/* All submissions must be received by March 29th, 2013 at 11:55pm (PST). */ 16:11:39 * ewoud upgraded foreman.ekohl.nl to 1.1 btw 16:11:41 <eedri_> ewoud, they can run on a vm on the ovirt instance 16:11:42 <ovirtbot> 14[[07Special:Log/upload14]]4 upload10 02 5* 03Vszocs 5* 10Vszocs uploaded "[[02File:Foreman view details.png10]]" 16:11:43 <quaid> so rax01 is bare metal dedicated to jenkins slaves 16:11:54 <quaid> rax02 is for testing, also having bare metal for KVM testing 16:11:56 <ewoud> will update the config later to do something useful and install some slaves 16:11:57 <eedri_> ewoud, quaid btw, we can install all in one on raxs 16:12:03 <eedri_> and add the other 2 servers as hosts 16:12:10 <eedri_> so all our services will run on vms 16:12:18 <ovirtbot> 14[[07Special:Log/upload14]]4 upload10 02 5* 03Vszocs 5* 10Vszocs uploaded "[[02File:ShellBox SubTab.png10]]" 16:12:19 <ewoud> eedri_: can you add the servers over WAN? 16:12:24 <quaid> alterway02 is supposed to house the list of random VMs - mailman, gerrit, etc. 16:12:24 <eedri_> ewoud, good q 16:12:37 <quaid> eedri_: all in one is an RPM, yes? 16:12:47 <eedri_> quaid, yes. it's a plugin for ovirt 16:12:52 <quaid> cool 16:13:03 <ewoud> eedri_: but +1 on having both rackspace servers in a single ovirt 16:13:41 <ewoud> ideally we'd have both alterway servers as well, but that may be something for the long run 16:16:59 <quaid> ok, my brain is starting to drift a bit :) ... as I have a call in a few minutes 16:17:20 <quaid> I think, are we done for the day? 16:17:21 <ewoud> same, so let's conclude 16:17:46 <ewoud> please have a look at the topics I started last week on the ML as linked before 16:17:48 <quaid> ok, I think I'll have to relook at what I'm doing Wed to be sure I do it right, but I'll start early when I can catch people 16:18:04 <quaid> #action all to read and possibly reply to open ML thread on Puppet 16:18:12 <ovirtbot> 14[[07Intel Workshop May 2013 CFP14]]4 !10 02http://www.ovirt.org/index.php?diff=7821&oldid=7818&rcid=8011 5* 03Theron 5* (-7) 10/* Submission Requirements */ 16:18:21 <ewoud> and new member process hasn't received any love either 16:19:53 <quaid> true that 16:19:56 <ovirtbot> 14[[07Intel Workshop May 2013 CFP14]]4 !10 02http://www.ovirt.org/index.php?diff=7822&oldid=7821&rcid=8012 5* 03Theron 5* (-20) 10/* Submission Requirements */ 16:20:03 <quaid> let's put in a new section in the minutes to address, too 16:20:23 <quaid> I suppose that's something I should be able to do since there is so much trapped in my brain still 16:20:38 * quaid has been bad at documenting 16:21:16 <ovirtbot> 14[[07Intel Workshop May 2013 CFP14]]4 !10 02http://www.ovirt.org/index.php?diff=7823&oldid=7822&rcid=8013 5* 03Theron 5* (+25) 10/* Contact information */ 16:21:37 <quaid> #action quaid to add new member discussion to ongoing agenda 16:21:49 <quaid> #action quaid to make some progress on documentation and such around new members by next week 16:22:27 <ovirtbot> 14[[07Intel Workshop May 2013 CFP14]]4 !10 02http://www.ovirt.org/index.php?diff=7824&oldid=7823&rcid=8014 5* 03Theron 5* (+32) 10/* Suggested Topics */ 16:23:30 <ovirtbot> 14[[07Infrastructure team meetings14]]4 10 02http://www.ovirt.org/index.php?diff=7825&oldid=7702&rcid=8015 5* 03Quaid 5* (+253) 10adding in agenda for next meeting 16:23:32 <quaid> ok, closing out this one 16:23:34 <quaid> in 5 16:23:36 <quaid> 4 16:23:37 <quaid> 3 16:23:39 <quaid> 2 16:23:40 <quaid> 1 16:23:43 <quaid> #endmeeting