14:01:02 <knesenko> #startmeeting Infra Weekly
14:01:02 <ovirtbot> Meeting started Mon Aug 12 14:01:02 2013 UTC.  The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:02 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:01:07 <Rydekull> I gotta jet within 30 minutes, but yeah, hello
14:01:18 <knesenko> #chair eedri dcaro ewoud obasan
14:01:18 <ovirtbot> Current chairs: dcaro eedri ewoud knesenko obasan
14:01:29 <knesenko> #chair Rydekull
14:01:29 <ovirtbot> Current chairs: Rydekull dcaro eedri ewoud knesenko obasan
14:03:12 <knesenko> #topic Hosting and Issues
14:03:19 <knesenko> ok hello guys .
14:03:59 <Rydekull> $hello
14:03:59 <knesenko> I am back from the reserve duty . I wasn't here for 3 weeks. So I would like to get a small update from you if we have some news/issues
14:04:25 <knesenko> Rydekull: nice to meet you ! I am kiril
14:04:52 <knesenko> So do we have some urgent issues ?
14:05:12 <obasan> knesenko, I know that there was a glitch in ovirt.org that directed to openshift
14:05:22 <obasan> knesenko, but I am not sure what caused that
14:05:38 <knesenko> obasan: do you know if somebody fixed it ?
14:05:55 <ewoud> obasan: that sounds like a misconfiguration in openshift and not something we can handle
14:06:00 <Rydekull> knesenko: We've met before, but you had just joined back then :-)
14:06:15 <knesenko> Rydekull: :)
14:06:17 <Rydekull> knesenko: im Alexander (Rydekull) either way
14:06:48 <knesenko> anything else urgent ?
14:06:55 <ewoud> knesenko: there was one thing we discussed in the past meetings about having a session to talk about our architecture and workflows, but maybe using a google hangout to make it a bit more interactive
14:07:21 <knesenko> ewoud: its possible
14:07:24 <knesenko> +1 for it
14:07:42 <knesenko> but we need an agenda for it ... to understand what we want to discuss there
14:07:42 <ewoud> knesenko: but we thought that we should wait for you to be back
14:07:55 <knesenko> ewoud: appreciated
14:09:00 <knesenko> ewoud: do you want to prepare the agenda for this kind of meeting ?
14:09:18 <ewoud> knesenko: maybe we should prepare it together somewhere after the meeting?
14:10:17 <knesenko> ewoud: possible. lets do it then
14:10:19 <knesenko> ok ...
14:10:23 <knesenko> lets contniue
14:10:42 <knesenko> I heard that there were some issues with ovirt.org emails
14:10:47 <knesenko> Am I right ?
14:10:54 <Rydekull> Yes
14:11:13 <Rydekull> Atleast two different issues
14:11:22 <Rydekull> 1. Disk space on linode01
14:11:38 <dcaro> ^ already solved
14:11:50 <Rydekull> 2. Since we're running in cloud, we dont know from where our e-mail get sent. IP it was being sent from was changed.
14:11:57 <Rydekull> Both of them has been resolved
14:12:13 <knesenko> Rydekull: dcaro good to hear
14:12:34 <knesenko> and the last one, is the issues with the rackspace setup I've installed
14:12:52 <knesenko> some disk space issues as well ... solved by eedri for now
14:13:15 <knesenko> we need some external storage solution asap
14:13:41 <knesenko> i will try to push it
14:13:46 <knesenko> can we move on ?
14:13:53 <knesenko> anything else on hosting ?
14:14:15 <Rydekull> no
14:14:31 <knesenko> #topic Foreman and Puppet
14:14:42 <knesenko> dcaro: ewoud any news here ?
14:15:16 <dcaro> knesenko: not on my side :(
14:15:38 <ewoud> knesenko: none here eithe r:(
14:15:43 <eedri> knesenko, same here
14:15:53 <knesenko> dcaro: i think that I sent you an email about the sudo puppet class before i left ... did you get it ?
14:16:26 <knesenko> ewoud: dcaro what about this one - http://gerrit.ovirt.org/#/c/16907/ ?
14:17:09 <ewoud> knesenko: not really I think
14:17:33 <knesenko> ewoud: do we want it ?
14:17:50 <ewoud> knesenko: we do want NTP, question is mostly how we're going to manage our module repo
14:17:59 <dcaro> knesenko: seems ok to me, but I already made my comment, I prefer 19 lines than 182
14:18:19 <ewoud> and I'd prefer some meta format that tells you where to get the module
14:18:34 <ewoud> be it git submodules or a Puppetfile based solution
14:18:48 <dcaro> ewoud: yep, that's another story, we have also to decide
14:19:43 <knesenko> dcaro: ewoud so let's decide how we are going to manage puppet classes
14:20:27 <knesenko> I am not such a puppet guru, so I think it will be better if dcaro and ewoud will decide the design
14:20:30 <knesenko> sounds good ?
14:21:02 <dcaro> I vote for puppetfile based (I've been warned about submodules by 3 different sysadmins), but I have no experience with neither
14:21:31 <Rydekull> Uh
14:21:32 <ewoud> I'd vote for git submodules in the short term and replace that with a Puppetfile based solution in the future, but that's because I do have experience with git submodules
14:21:45 <Rydekull> git submodules? unsure what you mean
14:21:53 <Rydekull> but I vote for modules in puppet, and stored in git
14:22:06 <ewoud> Rydekull: http://git-scm.com/book/en/Git-Tools-Submodules
14:22:06 <dcaro> Rydekull: http://git-scm.com/book/en/Git-Tools-Submodules
14:22:09 <dcaro> jejeje
14:22:14 <ewoud> dcaro: :)
14:22:41 <dcaro> Rydekull: both use modules in puppet and are stored in git :)
14:23:08 <dcaro> ewoud: how much will it cost you to setup the submodule solution?
14:23:17 <ewoud> dcaro: 0 because it's already in place ;)
14:23:33 <dcaro> ewoud: what's the decision about then? :)
14:23:52 <dcaro> when are we gonna switch to puppetfiles?
14:24:07 <ewoud> dcaro: I'd say when we have time, but that may be quite far in the future
14:24:08 <Rydekull> I vote for the version that dont use submodules, looking at the URL you referenced atleast
14:24:29 <Rydekull> so, filebased
14:25:05 <knesenko> Voting for filebased
14:25:25 <obasan> is puppet librarian out of the question?
14:25:43 <dcaro> librarian is one of the puppetfile based solutions
14:25:46 <ewoud> obasan: no, but I don't think anyone has experience with it so we would need to fix our deployment scripts
14:26:22 <obasan> ewoud, ok :)
14:27:53 <knesenko> can someone summarize the differences between the solutions we are talking about ?
14:28:39 * ewoud starts http://etherpad.ovirt.org/p/puppet_module_layout
14:29:19 <Rydekull> well, my time is up
14:29:22 <Rydekull> see you guys
14:29:32 <dcaro> Rydekull: bye!
14:29:44 <knesenko> Rydekull: thanks bye
14:32:31 <knesenko> dcaro: ewoud eedri obasan please vote http://etherpad.ovirt.org/p/puppet_module_layout
14:35:24 <knesenko> if we want to make changes, so lets do it from the beginning for the long term
14:35:44 <knesenko> make no sense to me to make things for a short term
14:37:15 <eedri> can we consult some puppet experts?
14:37:22 <eedri> maybe ohadlevy
14:37:33 <eedri> can pitch in his opinion
14:38:26 <eedri> knesenko, +1 for not doing short term, i don't think it's that critical to apply for having a temp solution
14:38:34 <knesenko> eedri: aha
14:38:42 <knesenko> so i think that r10k is the winner
14:39:35 <ewoud> it does appear that way
14:40:17 <knesenko> #action dcaro and ewoud please work to implement r10k for puppet
14:40:23 <ewoud> ok
14:40:28 <dcaro> ok
14:40:34 <knesenko> ewoud: dcaro please create a ticket for it, so we will track it
14:40:46 <dcaro> ewoud: do you do the honors?
14:41:04 <ewoud> dcaro: will do
14:41:05 <knesenko> ewoud: dcaro thanks for the summary
14:41:20 <knesenko> anything else on puppet ?
14:42:07 <knesenko> #topic Jenkins
14:42:11 <knesenko> eedri: news ?
14:42:12 <ewoud> dcaro: mind if I rewrite https://fedorahosted.org/ovirt/ticket/71 to be r10k based?
14:42:32 <dcaro> ewoud: sure
14:42:40 <dcaro> meaning, of course not :9
14:42:50 <eedri> knesenko, not from my side, there were issues with gwt compilations, which i think are fixed now
14:42:59 <eedri> knesenko, but i did saw some email on missing mom from nightly rpeo
14:43:12 <eedri> knesenko, so it should be checked, maybe mom rpm job isn't building latest ver?
14:43:30 <knesenko> eedri: i have no idea ...
14:43:36 * eedri checking
14:43:51 <knesenko> eedri: thanks
14:44:25 <knesenko> dcaro: i heard that you prepared F18 template for jenkins slaves, right ?
14:45:02 <eedri> knesenko, seems to me it's there
14:45:06 <dcaro> knesenko: yep, but maybe we shoudl shrink the disk size to fit 5 machines instead of 4 on the host, seeing the latest space issues
14:45:32 <knesenko> dcaro: i used 50GB or 40GB per vm
14:45:40 <knesenko> I thunk so
14:45:50 <eedri> knesenko, email from aug 1, might not be relevant anymore
14:46:02 <knesenko> eedri: ok ... thanks for checking that
14:46:29 <dcaro> knesenko: looking
14:46:30 <knesenko> eedri: dcaro we need to decide how many VMs do we want to install on racksapce
14:46:35 <knesenko> and which OS
14:47:04 <dcaro> knesenko: 45GB
14:47:27 <knesenko> dcaro: ok
14:47:35 <eedri> knesenko, we need to keep in mind gwt now requires more mem
14:47:48 <dcaro> I'd say 4f19, 3f18 and 1rhel was the starting point right?
14:47:52 <knesenko> dcaro: and whats the data domain size ?
14:48:00 <eedri> knesenko, now we have an un optimal solution where one vm has 1 execution only
14:48:13 <eedri> knesenko, maybe schedule gwt to run nightly?
14:48:27 <dcaro> knesenko: 196GB - 10% = 177GB
14:48:43 <dcaro> 176 is more close
14:48:43 <eedri> knesenko, cause right now we're missing alot of power on 1 vm due to that restriction
14:49:09 <knesenko> eedri: F19 slaves has 5 executors
14:49:11 <dcaro> eedri: I though it was decided to use vms as big as possible?
14:49:26 <eedri> knesenko, except 1, which is f19-gwt
14:49:29 <dcaro> eedri: sorry, yep
14:49:32 <eedri> knesenko, it was changed while you weren't here
14:49:39 <knesenko> eedri: instead of - fedora19-vm02
14:49:46 <eedri> knesenko, yes
14:49:52 <eedri> knesenko, that was the only things that worked
14:50:00 <eedri> knesenko, for running new gwt 2.5 compilations
14:50:07 <knesenko> :\
14:50:19 <eedri> so we need to rethink how to run them
14:50:30 <knesenko> eedri: gwt job takes 15 min
14:51:03 <eedri> knesenko, ok, just need to make sure it runs alone
14:51:13 <knesenko> eedri: ok
14:51:25 <knesenko> dcaro: so how many VMs can we create on rackspace 2 ?
14:51:46 <dcaro> right now 3 with 45GB
14:51:55 <knesenko> dcaro: and 5 with 30 GB ?
14:51:55 <dcaro> but if we use 40 instead we can put 4
14:52:10 <dcaro> yep
14:52:20 <knesenko> i vote for 4 , each one 40G
14:52:58 <knesenko> eedri: which OS ?
14:53:01 <dcaro> +1
14:53:16 <eedri> knesenko, i propose 2 f18 + 2 centos 6.4
14:53:47 <knesenko> eedri: no more f19 ?
14:53:59 <dcaro> good for me, but we should shrink the f19 machines too, to fit 4 of them
14:54:06 <dcaro> maybe gradually
14:54:09 <eedri> knesenko, we have 3.. you think we need more?
14:54:24 <eedri> knesenko, most of the jobs are maven compilations, so it doesn't really matter
14:54:31 <knesenko> one of them uses 1 executor
14:54:36 <knesenko> eedri: ok
14:54:47 <eedri> knesenko, and we'll be able to upgrade f18 in the future if needed
14:55:00 <knesenko> eedri: ok ...
14:55:18 <knesenko> #action knesenko install f18 + 2 centos 6.4 on rackspace2
14:55:23 <knesenko> anything else here ?
14:55:46 <knesenko> #topic Other Issues
14:56:04 <ewoud> I think the demand for spam filtering becomes bigger and bigger
14:56:43 <ewoud> maybe we should increase the priority on that?
14:56:54 <knesenko> ewoud: yes we need ...
14:57:08 <ewoud> I could look into using $employers anti spam cluster where we already have many customers
14:57:08 <dcaro> today I reviewed the infra mailing list, there were  alot of spam messages
14:57:20 <eedri> knesenko, i think there was a request for network testing on jenkins, dcaro can elaborate on that
14:57:59 <eedri> ewoud, there is spamassasing opensource project which was used in last $company
14:58:02 <dcaro> yes, There's a request to create a new jenkins job to run networking functional tests
14:58:20 <ewoud> eedri: $employers cluster is also based on spamassassin
14:58:27 <eedri> ewoud, http://spamassassin.apache.org/
14:58:51 <dcaro> in my last one we also used calmav
14:59:00 <dcaro> www.clamav.net
14:59:31 <eedri> dcaro, +1, also is familiar
14:59:33 <dcaro> for antimalware, combined with spamassassin
14:59:33 <ewoud> thing is, the service is already running and is mostly a case of creating the right records in our database and changing the MX records
14:59:36 <knesenko> and what do we have now
14:59:41 <ewoud> and that's better than nothing
15:01:02 <knesenko> who can handle it ?
15:01:05 <knesenko> dcaro: :) :) :)
15:01:12 * knesenko is looking on dcaro
15:01:29 <dcaro> ewoud: eedri we don't have any spf/ domainkeys records right?
15:01:38 <ewoud> dcaro: not as far as I know
15:01:48 <dcaro> then it's easier xd
15:01:57 <dcaro> knesenko: ok, put me in ;)
15:02:34 <ewoud> dcaro: https://fedorahosted.org/ovirt/ticket/12
15:02:35 <knesenko> dcaro: thanks . I can help ...
15:02:41 <dcaro> btw. should we have spf/dkim in place? Do we send messages from @ovirt.org addresses?
15:03:12 <knesenko> #action dcaro try to push spam solution - https://fedorahosted.org/ovirt/ticket/12
15:05:06 <knesenko> dcaro: what about the network test eedri just mentioned
15:05:25 <knesenko> we have no time for this meeting , so we will finish in few minutes
15:06:04 <dcaro> knesenko: there's an email on the list about it, but they also asked for a way to start only certain jobs with a tag in the commit message
15:06:46 <knesenko> dcaro: will search for this email
15:06:53 <knesenko> ok thanks guys !
15:06:57 <knesenko> have a nice week
15:07:00 <knesenko> #endmeeting