14:01:02 #startmeeting Infra Weekly 14:01:02 Meeting started Mon Aug 12 14:01:02 2013 UTC. The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:02 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:01:07 I gotta jet within 30 minutes, but yeah, hello 14:01:18 #chair eedri dcaro ewoud obasan 14:01:18 Current chairs: dcaro eedri ewoud knesenko obasan 14:01:29 #chair Rydekull 14:01:29 Current chairs: Rydekull dcaro eedri ewoud knesenko obasan 14:03:12 #topic Hosting and Issues 14:03:19 ok hello guys . 14:03:59 $hello 14:03:59 I am back from the reserve duty . I wasn't here for 3 weeks. So I would like to get a small update from you if we have some news/issues 14:04:25 Rydekull: nice to meet you ! I am kiril 14:04:52 So do we have some urgent issues ? 14:05:12 knesenko, I know that there was a glitch in ovirt.org that directed to openshift 14:05:22 knesenko, but I am not sure what caused that 14:05:38 obasan: do you know if somebody fixed it ? 14:05:55 obasan: that sounds like a misconfiguration in openshift and not something we can handle 14:06:00 knesenko: We've met before, but you had just joined back then :-) 14:06:15 Rydekull: :) 14:06:17 knesenko: im Alexander (Rydekull) either way 14:06:48 anything else urgent ? 14:06:55 knesenko: there was one thing we discussed in the past meetings about having a session to talk about our architecture and workflows, but maybe using a google hangout to make it a bit more interactive 14:07:21 ewoud: its possible 14:07:24 +1 for it 14:07:42 but we need an agenda for it ... to understand what we want to discuss there 14:07:42 knesenko: but we thought that we should wait for you to be back 14:07:55 ewoud: appreciated 14:09:00 ewoud: do you want to prepare the agenda for this kind of meeting ? 14:09:18 knesenko: maybe we should prepare it together somewhere after the meeting? 14:10:17 ewoud: possible. lets do it then 14:10:19 ok ... 14:10:23 lets contniue 14:10:42 I heard that there were some issues with ovirt.org emails 14:10:47 Am I right ? 14:10:54 Yes 14:11:13 Atleast two different issues 14:11:22 1. Disk space on linode01 14:11:38 ^ already solved 14:11:50 2. Since we're running in cloud, we dont know from where our e-mail get sent. IP it was being sent from was changed. 14:11:57 Both of them has been resolved 14:12:13 Rydekull: dcaro good to hear 14:12:34 and the last one, is the issues with the rackspace setup I've installed 14:12:52 some disk space issues as well ... solved by eedri for now 14:13:15 we need some external storage solution asap 14:13:41 i will try to push it 14:13:46 can we move on ? 14:13:53 anything else on hosting ? 14:14:15 no 14:14:31 #topic Foreman and Puppet 14:14:42 dcaro: ewoud any news here ? 14:15:16 knesenko: not on my side :( 14:15:38 knesenko: none here eithe r:( 14:15:43 knesenko, same here 14:15:53 dcaro: i think that I sent you an email about the sudo puppet class before i left ... did you get it ? 14:16:26 ewoud: dcaro what about this one - http://gerrit.ovirt.org/#/c/16907/ ? 14:17:09 knesenko: not really I think 14:17:33 ewoud: do we want it ? 14:17:50 knesenko: we do want NTP, question is mostly how we're going to manage our module repo 14:17:59 knesenko: seems ok to me, but I already made my comment, I prefer 19 lines than 182 14:18:19 and I'd prefer some meta format that tells you where to get the module 14:18:34 be it git submodules or a Puppetfile based solution 14:18:48 ewoud: yep, that's another story, we have also to decide 14:19:43 dcaro: ewoud so let's decide how we are going to manage puppet classes 14:20:27 I am not such a puppet guru, so I think it will be better if dcaro and ewoud will decide the design 14:20:30 sounds good ? 14:21:02 I vote for puppetfile based (I've been warned about submodules by 3 different sysadmins), but I have no experience with neither 14:21:31 Uh 14:21:32 I'd vote for git submodules in the short term and replace that with a Puppetfile based solution in the future, but that's because I do have experience with git submodules 14:21:45 git submodules? unsure what you mean 14:21:53 but I vote for modules in puppet, and stored in git 14:22:06 Rydekull: http://git-scm.com/book/en/Git-Tools-Submodules 14:22:06 Rydekull: http://git-scm.com/book/en/Git-Tools-Submodules 14:22:09 jejeje 14:22:14 dcaro: :) 14:22:41 Rydekull: both use modules in puppet and are stored in git :) 14:23:08 ewoud: how much will it cost you to setup the submodule solution? 14:23:17 dcaro: 0 because it's already in place ;) 14:23:33 ewoud: what's the decision about then? :) 14:23:52 when are we gonna switch to puppetfiles? 14:24:07 dcaro: I'd say when we have time, but that may be quite far in the future 14:24:08 I vote for the version that dont use submodules, looking at the URL you referenced atleast 14:24:29 so, filebased 14:25:05 Voting for filebased 14:25:25 is puppet librarian out of the question? 14:25:43 librarian is one of the puppetfile based solutions 14:25:46 obasan: no, but I don't think anyone has experience with it so we would need to fix our deployment scripts 14:26:22 ewoud, ok :) 14:27:53 can someone summarize the differences between the solutions we are talking about ? 14:28:39 * ewoud starts http://etherpad.ovirt.org/p/puppet_module_layout 14:29:19 well, my time is up 14:29:22 see you guys 14:29:32 Rydekull: bye! 14:29:44 Rydekull: thanks bye 14:32:31 dcaro: ewoud eedri obasan please vote http://etherpad.ovirt.org/p/puppet_module_layout 14:35:24 if we want to make changes, so lets do it from the beginning for the long term 14:35:44 make no sense to me to make things for a short term 14:37:15 can we consult some puppet experts? 14:37:22 maybe ohadlevy 14:37:33 can pitch in his opinion 14:38:26 knesenko, +1 for not doing short term, i don't think it's that critical to apply for having a temp solution 14:38:34 eedri: aha 14:38:42 so i think that r10k is the winner 14:39:35 it does appear that way 14:40:17 #action dcaro and ewoud please work to implement r10k for puppet 14:40:23 ok 14:40:28 ok 14:40:34 ewoud: dcaro please create a ticket for it, so we will track it 14:40:46 ewoud: do you do the honors? 14:41:04 dcaro: will do 14:41:05 ewoud: dcaro thanks for the summary 14:41:20 anything else on puppet ? 14:42:07 #topic Jenkins 14:42:11 eedri: news ? 14:42:12 dcaro: mind if I rewrite https://fedorahosted.org/ovirt/ticket/71 to be r10k based? 14:42:32 ewoud: sure 14:42:40 meaning, of course not :9 14:42:50 knesenko, not from my side, there were issues with gwt compilations, which i think are fixed now 14:42:59 knesenko, but i did saw some email on missing mom from nightly rpeo 14:43:12 knesenko, so it should be checked, maybe mom rpm job isn't building latest ver? 14:43:30 eedri: i have no idea ... 14:43:36 * eedri checking 14:43:51 eedri: thanks 14:44:25 dcaro: i heard that you prepared F18 template for jenkins slaves, right ? 14:45:02 knesenko, seems to me it's there 14:45:06 knesenko: yep, but maybe we shoudl shrink the disk size to fit 5 machines instead of 4 on the host, seeing the latest space issues 14:45:32 dcaro: i used 50GB or 40GB per vm 14:45:40 I thunk so 14:45:50 knesenko, email from aug 1, might not be relevant anymore 14:46:02 eedri: ok ... thanks for checking that 14:46:29 knesenko: looking 14:46:30 eedri: dcaro we need to decide how many VMs do we want to install on racksapce 14:46:35 and which OS 14:47:04 knesenko: 45GB 14:47:27 dcaro: ok 14:47:35 knesenko, we need to keep in mind gwt now requires more mem 14:47:48 I'd say 4f19, 3f18 and 1rhel was the starting point right? 14:47:52 dcaro: and whats the data domain size ? 14:48:00 knesenko, now we have an un optimal solution where one vm has 1 execution only 14:48:13 knesenko, maybe schedule gwt to run nightly? 14:48:27 knesenko: 196GB - 10% = 177GB 14:48:43 176 is more close 14:48:43 knesenko, cause right now we're missing alot of power on 1 vm due to that restriction 14:49:09 eedri: F19 slaves has 5 executors 14:49:11 eedri: I though it was decided to use vms as big as possible? 14:49:26 knesenko, except 1, which is f19-gwt 14:49:29 eedri: sorry, yep 14:49:32 knesenko, it was changed while you weren't here 14:49:39 eedri: instead of - fedora19-vm02 14:49:46 knesenko, yes 14:49:52 knesenko, that was the only things that worked 14:50:00 knesenko, for running new gwt 2.5 compilations 14:50:07 :\ 14:50:19 so we need to rethink how to run them 14:50:30 eedri: gwt job takes 15 min 14:51:03 knesenko, ok, just need to make sure it runs alone 14:51:13 eedri: ok 14:51:25 dcaro: so how many VMs can we create on rackspace 2 ? 14:51:46 right now 3 with 45GB 14:51:55 dcaro: and 5 with 30 GB ? 14:51:55 but if we use 40 instead we can put 4 14:52:10 yep 14:52:20 i vote for 4 , each one 40G 14:52:58 eedri: which OS ? 14:53:01 +1 14:53:16 knesenko, i propose 2 f18 + 2 centos 6.4 14:53:47 eedri: no more f19 ? 14:53:59 good for me, but we should shrink the f19 machines too, to fit 4 of them 14:54:06 maybe gradually 14:54:09 knesenko, we have 3.. you think we need more? 14:54:24 knesenko, most of the jobs are maven compilations, so it doesn't really matter 14:54:31 one of them uses 1 executor 14:54:36 eedri: ok 14:54:47 knesenko, and we'll be able to upgrade f18 in the future if needed 14:55:00 eedri: ok ... 14:55:18 #action knesenko install f18 + 2 centos 6.4 on rackspace2 14:55:23 anything else here ? 14:55:46 #topic Other Issues 14:56:04 I think the demand for spam filtering becomes bigger and bigger 14:56:43 maybe we should increase the priority on that? 14:56:54 ewoud: yes we need ... 14:57:08 I could look into using $employers anti spam cluster where we already have many customers 14:57:08 today I reviewed the infra mailing list, there were alot of spam messages 14:57:20 knesenko, i think there was a request for network testing on jenkins, dcaro can elaborate on that 14:57:59 ewoud, there is spamassasing opensource project which was used in last $company 14:58:02 yes, There's a request to create a new jenkins job to run networking functional tests 14:58:20 eedri: $employers cluster is also based on spamassassin 14:58:27 ewoud, http://spamassassin.apache.org/ 14:58:51 in my last one we also used calmav 14:59:00 www.clamav.net 14:59:31 dcaro, +1, also is familiar 14:59:33 for antimalware, combined with spamassassin 14:59:33 thing is, the service is already running and is mostly a case of creating the right records in our database and changing the MX records 14:59:36 and what do we have now 14:59:41 and that's better than nothing 15:01:02 who can handle it ? 15:01:05 dcaro: :) :) :) 15:01:12 * knesenko is looking on dcaro 15:01:29 ewoud: eedri we don't have any spf/ domainkeys records right? 15:01:38 dcaro: not as far as I know 15:01:48 then it's easier xd 15:01:57 knesenko: ok, put me in ;) 15:02:34 dcaro: https://fedorahosted.org/ovirt/ticket/12 15:02:35 dcaro: thanks . I can help ... 15:02:41 btw. should we have spf/dkim in place? Do we send messages from @ovirt.org addresses? 15:03:12 #action dcaro try to push spam solution - https://fedorahosted.org/ovirt/ticket/12 15:05:06 dcaro: what about the network test eedri just mentioned 15:05:25 we have no time for this meeting , so we will finish in few minutes 15:06:04 knesenko: there's an email on the list about it, but they also asked for a way to start only certain jobs with a tag in the commit message 15:06:46 dcaro: will search for this email 15:06:53 ok thanks guys ! 15:06:57 have a nice week 15:07:00 #endmeeting