14:01:12 <quaid> #startmeeting
14:01:12 <ovirtbot> Meeting started Tue Aug  7 14:01:12 2012 UTC.  The chair is quaid. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:12 <ovirtbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:01:21 * eedri here
14:01:28 * mburns here
14:01:29 <quaid> #chair eedri RobertM mburns
14:01:29 <ovirtbot> Current chairs: RobertM eedri mburns quaid
14:01:49 <itamar> itamar does changes to gerrit itself. karsten has permissions to system of gerrit for security audit, backups, etc.
14:01:51 <quaid> #meetingname oVirt Infra Team weekly
14:01:51 <ovirtbot> The meeting name has been set to 'ovirt_infra_team_weekly'
14:02:23 <quaid> #topic roll call & agenda
14:02:33 * RobertM here
14:02:36 <quaid> http://wiki.ovirt.org/wiki/Infrastructure_team_meetings#2012-08-07
14:02:54 <quaid> today we have: puppet, jenkins, and hosting ... anything missing?
14:03:20 <RobertM> documentation?
14:03:29 <quaid> ok
14:03:51 * mburns proposes package signing...
14:04:17 <quaid> ok
14:04:45 <ovirtbot> 14[[07Infrastructure team meetings14]]4 !10 02http://wiki.ovirt.org/w/index.php?diff=4072&oldid=4069&rcid=4171 5* 03Quaid 5* (+40) 10/* 2012-08-07 */ adding new agenda items 
14:05:49 <quaid> ok, getting started
14:05:58 <quaid> #topic Puppet
14:06:11 <quaid> Is ewoud available to talk about puppet and/or foreman planning?
14:07:06 <quaid> ok, we can get back to this one if ewoud becomes available later
14:07:16 <RobertM> quaid, +1
14:07:17 <ohadlevy> quaid: can i help ? :)
14:07:21 <ewoud> oh sorry
14:07:29 <ewoud> bit busy atm
14:07:31 <quaid> #action quaid to ask ewoud for a Puppet update for the mailing list
14:07:55 <eedri> ohadlevy, we're talking about installing a puppet/foreman server to manage all infra servers/vms
14:07:55 <ewoud> short summary now is that I haven't had time to prepare something yet
14:08:08 <quaid> ok
14:08:26 <ewoud> in fact, busy deploying foreman here at work
14:08:57 <quaid> ewoud: do you want to take up the discussion on the mailing list for now?
14:09:13 <ewoud> quaid: yes, I expect that next week I will have had some time
14:09:21 <quaid> ok
14:09:24 <eedri> quaid, do we have a server ready for it ? puppet.ovirt.oirg?
14:09:44 <quaid> eedri: right now that is pointing at the www server
14:09:48 <RobertM> eedri, Yes puppet.ovirt.org points at kitchen sink
14:10:11 * eedri suggests to have a test vm to install and play with foreman/puppet
14:10:24 <eedri> ohadlevy, there is a step-by-step guide for it, right?
14:10:24 <ohadlevy> eedri++ please use the installer to setup everything ;)
14:10:51 <RobertM> quaid, eedri Do we have the resources for something like that?
14:11:12 <quaid> yes and no
14:11:21 <quaid> we can continue using EC2 instances, for example
14:11:28 <eedri> quaid, -1 on that..
14:11:34 <quaid> but no pool of VMs, even small ones
14:11:37 <eedri> quaid, we can try using the new slave i added
14:11:40 <ohadlevy> quaid: yes, foreman can create instances in ec2 too
14:12:05 * eedri added a new physical host slave to jenkins,
14:12:16 <RobertM> ec2 is not our favorite provider right now.
14:12:20 <quaid> eedri: is it accessible to anyone on the Infra team?
14:12:24 <ohadlevy> RobertM: ovirt works too :)
14:12:32 <eedri> quaid, no, even jenkins can't access it
14:12:38 <eedri> quaid, it's a headless slave
14:12:41 <quaid> about EC2, is it a problem for Puppet?
14:12:50 <eedri> come to think about it
14:12:54 <eedri> it can't be used
14:13:05 <eedri> sorry, only as a jenkins slave
14:13:09 <quaid> right
14:13:38 <eedri> quaid, we can utilize one of the current rhel62 ec2 vms
14:13:46 <eedri> don't think it's utilized to the max
14:14:00 <eedri> but i don't like the performance we'd probably get
14:14:16 <quaid> ah, that's the question - does the performance matter for puppet & foreman
14:14:43 <eedri> ohadlevy, do you have any minimal req for puppet/foreman installation?
14:14:46 <RobertM> The rhel62 boxes are more used then the slave01 / 02 thanks to jenkins.ekohl.nl
14:15:18 <RobertM> puppet requirements are min.
14:15:28 <RobertM> It could run on linode01
14:15:32 <karmatronic> eedri: i d say a 1G-memory vm will do the job
14:15:42 <RobertM> Forman haven't used
14:15:57 <eedri> we also need to think about the meaning of running a public/foreman instance
14:16:08 <eedri> we might need to harden it, so security concerns
14:16:25 <karmatronic> eedri: thought that depends on how many machines will be handled by puppet
14:16:39 <quaid> ok, since we don't have ewoud for this discussion, I'd like us to continue it on list
14:16:47 <eedri> +1
14:16:56 <RobertM> +1
14:17:01 <RobertM> +1 move to list
14:17:03 <quaid> ok, so let's move on to ...
14:17:12 <ewoud> quaid: if you need my input, I can answer but +1 on moving it to the list
14:17:13 <quaid> #action move Puppet/Foreman discussion to list
14:17:43 <quaid> ewoud: can you start the list discussion sometime today or tomorrow?
14:17:49 <ewoud> quaid: ok
14:17:56 <quaid> thanks
14:18:06 <quaid> moving on to the next topic ...
14:18:14 <quaid> #topic Jenkins
14:18:23 <RobertM> I propuse we move to hosting 1st before jenkins.
14:18:27 <quaid> ok
14:18:37 <eedri> :)
14:18:40 * quaid switches around agenda
14:18:47 <quaid> #topic Hosting
14:18:52 <eedri> jenkins topic has the tendency to suck all meeting time..
14:18:56 <eedri> better leave it to the end
14:19:16 <quaid> so I don't think we got a resolution on the mailing list about hosting
14:19:19 <RobertM> The hosting question can effect the jenkins topic allot so let get that one done 1st.
14:19:41 <quaid> I guess I was looking for us to throw out a few hosting providers as ideas - preferably ones that use KVM :)
14:19:46 <RobertM> I think only 3 people chimed in on it.
14:20:58 <ewoud> I was mostly wondering where the budget is and where it comes from
14:21:02 <RobertM> quaid, Do we have actually ussage info we can reference for both the master and the slaves?
14:21:14 <quaid> RobertM: I don't know where that would be
14:21:22 <mburns> i was more hoping for one of the sponsor companies to say they will host something...
14:21:27 <RobertM> Who manages the EC2 instances
14:21:37 <eedri> RobertM, itamar
14:21:51 <RobertM> itamar, Would have access to that info then
14:22:01 <eedri> RobertM, not sure.
14:22:08 <eedri> RobertM, what info are we looking for?
14:22:09 <quaid> ewoud: I haven't looked for a specific budget amount, but I'm going to ask e.g. itamar for budget solutions
14:22:34 <RobertM> EC2 keeps basic stats on all EC2 instances.  Not going to get micro break down but will get an idea.  Need it for billing purpuses.
14:22:37 <quaid> mburns: +1 but we've been dangling that out there for a few months, no takers yet
14:23:47 <eedri> itamar says gerrit uses extra large amazon instance
14:23:58 <ewoud> I've heard good stories about hetzner
14:24:10 <RobertM> eedri, Looking for bandwidth mainly.  That is the one area where VPS differant themselves
14:24:12 <eedri> he recommends a bare metal server with quad core
14:24:24 <ewoud> looking at their prices they're relative cheap
14:24:32 <eedri> i understood from him that he doesn't have such info
14:24:38 <jboggs> mburns, 2.5.1 tests fine to me
14:24:49 <quaid> #info for Jenkins master, bare metal server with quad core is good minimum
14:24:58 <eedri> RobertM, we can look at jenkins monitoring charts, but that's about it
14:25:01 <quaid> #info Hetzner is one hosting idea
14:25:20 <mburns> jboggs: thanks!
14:25:36 <eedri> quaid, i think itamar talked about a server for gerrit.ovirt.org, but it applies to jenkins master as well
14:25:40 <RobertM> I have ovirt.info those 2 bare metal boxes are 2x quads with 16G of ram.  Sata HD only.
14:26:46 <ewoud> #link http://www.hetzner.de/en/hosting/produktmatrix/rootserver-produktmatrix-ex
14:27:44 <eedri> RobertM, i think we should understand how much is spent today on EC2 vms and derive from that our budets for other hosting vendors
14:28:07 <eedri> RobertM, and with that budget get the best we can put our hands on
14:28:48 <RobertM> +1 for eedri idea.  The question is can be convert those EC2 instances for $$$?
14:29:14 <eedri> RobertM, we're running those instance 24/7.. i think that's a lot of money
14:29:29 <eedri> RobertM, we might be able to lower costs even if moving to a diff vendoer
14:29:32 <ewoud> I think we can find much more power for less $$$
14:29:54 <quaid> eedri: one consideration, the EC2 instances are being pulled from a big pool & not e.g. itamar's budget, afaik
14:30:12 <RobertM> eedri, ewoud Don't disagree.  EC2 runs around 50 a month per instance depending on bandwidth and IO used.
14:30:27 <quaid> I was thinking we could start with US$100 as a monthly cap
14:31:01 <RobertM> But as quaid just the EC2 instances might be donated and not easily converted to money that can be used at another provider.
14:31:14 <RobertM> just=just said
14:31:30 <eedri> what i meant was, maybe we should focus on what is the budget, regadless where it comes from
14:31:44 <eedri> and from there understand what we can get
14:32:00 <eedri> i.e search for offers from hosting provides with that $$$
14:32:09 <RobertM> eedri, True but no one seems to know what that budget is
14:32:43 <quaid> I'll ask for a number from the Red Hat folk involved
14:32:59 <eedri> on arch maybe?
14:33:12 <eedri> so other companies will get a chance to know?
14:33:14 <RobertM> I suspect the number is near zero
14:33:49 <RobertM> and everything is being donated as services not cash other then maybe linode01
14:35:04 <quaid> well, EC2 isn't being donated
14:35:57 <quaid> should we gently prod on board@?
14:36:33 <eedri> quaid, if we ask the board, maybe its best we come with a ready proposal
14:36:38 <RobertM> If the project is paying for EC then we can migrate off and save $$$
14:36:46 <eedri> quaid, i.e $$$ for a certain hosting offer
14:37:58 <RobertM> eedri, quaid I can tell you that my 2 boxes are twice as fast as 5 EC2 VM and the one donated slave.
14:38:28 <quaid> RobertM: there is another department at Red Hat that has the EC2 account, and they are covering us oh so very kindly
14:38:53 <RobertM> So I was right it is a donated service :)
14:39:12 <quaid> RobertM: so from an RHT perpsective, we'll save $$$ if we switch from EC2 and still pay for another hosting provider directly ( as we are for Linode)
14:39:29 <quaid> RobertM: right, just not donated by Amazon :)
14:39:53 <RobertM> Never said it was.  I said the "oVirt" project wasn't paying
14:39:58 <quaid> ok, I think we have enough to go on
14:40:11 <quaid> #action continue list discussion to pick a good host to try
14:40:25 <quaid> #action quaid to find out if there is budget from RHT to use here
14:40:39 <quaid> #action someone asks the board@ for resources/help/hosts
14:40:49 <RobertM> quaid, +1 on RHT
14:41:02 <quaid> any missing actions?
14:41:11 <RobertM> Nope
14:41:44 <quaid> ok, let's give ourselves some time for our favorite topic!
14:41:56 <quaid> well, before that
14:42:12 <quaid> I guess let's tackle the other topics so there is time, & leave Jenkins for a bit longer
14:42:19 <quaid> #topic Infra documentation
14:42:22 <quaid> what's missing?
14:42:30 <quaid> #info list of who has access to what services
14:42:37 <quaid> #undo
14:42:37 <ovirtbot> Removing item from minutes: <MeetBot.items.Info object at 0x8c98e6c>
14:42:43 <quaid> #info list of who has what access to what services
14:42:59 <RobertM> +1
14:43:28 <eedri> +1
14:43:44 <eedri> should we document list of servers? and roles
14:43:47 <RobertM> Robert suggests assign task to me for creating a matrix of services ovirt is offering and who has admin rights to them
14:43:51 <eedri> or it's a security risk
14:44:15 <quaid> #info need to document list of servers and roles
14:44:24 <quaid> eedri: transparency is always a security risk :)
14:44:29 <quaid> but one we bear
14:44:55 <quaid> #action RobertM make a matrix of services and who has admin rights on them (CLI, WebUI, etc.)
14:45:12 <RobertM> eedri, quaid I think we would be better to just list all the services and add contact people to the services.  That exposes less details but expose the info people care about.  who to contact when something breaks
14:45:14 <eedri> well, one aspect of documentation is documenting tasks and rfe's
14:45:17 <quaid> RobertM: whatever we have is in this category:
14:45:19 <quaid> http://ovirt.org/wiki/Category:Infrastructure_documentation
14:45:35 <eedri> i think it will be eaiser for infra team to have a tracking server like jira/redmine for ongoing takss
14:45:50 <quaid> yes, I left out doing that until we begged ourselves to do it :)
14:46:06 <quaid> -1 to Jira but I'd try Redmine or even Teambox
14:46:22 <RobertM> I prefer cerberus myself.
14:46:25 <quaid> the latter we can get hosted on Teambox.com
14:46:27 * eedri uses redmine for his jenkins infra
14:46:31 <RobertM> and I have a 5 user license
14:47:09 <quaid> in general, my recommendation is to keep to open source that we can run ourselves, if we choose - hosted is fine, but I hate getting locked in to a closed solution
14:47:26 <eedri> +1 for using an opensource free solution
14:47:31 <quaid> #action discuss task trackers on the mailing list
14:48:22 <quaid> ok, anything more on docs?
14:48:28 <RobertM> I generally prefer open source but prefer something that fits a workflow more.  Do we know what workflow we want to use?
14:48:43 <karmatronic> quaid: asana is interesting too for you guys
14:48:58 <eedri> RobertM, let's continue this on list, like quid suggested
14:49:04 <RobertM> And that isn't documentation so +1 move to list
14:49:53 <RobertM> #action Open thread on issue tracker for changes to infra.
14:50:43 <eedri> quid, next topic?
14:50:50 <eedri> quaid, next topic?
14:50:56 <quaid> aye
14:51:06 <quaid> #topic package signing
14:51:09 <quaid> was that mburns ?
14:51:15 <mburns> yes
14:51:29 <mburns> i had a bz filed against ovirt-node-iso that some packages in the image aren't signed
14:51:42 <mburns> we had discussed this way back around the 3.0 GA that we needed a solution
14:51:58 <mburns> now that we have a dedicated infra team, it makes sense to bring it up again
14:52:01 <RobertM> mburns, What is required to sign a package.
14:52:45 <mburns> both private and public keys for one thing
14:53:09 <mburns> and it should really be restricted to just a small subset of people that can do it
14:53:10 <RobertM> Do any one have any links to the steps used to sign a package?
14:53:36 <mburns> http://fedoranews.org/tchung/gpg/
14:54:20 <mburns> or we can track down some release engineers from RHT and find out what process they use
14:54:22 <RobertM> It the signing going to require changes to the actual git repo's or can we add it as part of the jenkins jobs?
14:54:36 <mburns> RobertM: i wouldn't do it in jenkins....
14:54:44 <quaid> ok, we as
14:55:08 <mburns> we want it to be a conscious decision for each package we sign
14:55:27 <mburns> for example, nightly wouldn't be signed at all, but when we post a beta, we sign with a beta key
14:55:36 <eedri> mburns, we're talking about offical releases right? no nightlies
14:55:38 <RobertM> mburns, To me that is a requirement.  Since the plan as I understand it is to do releases from Jenkins
14:55:42 <mburns> and then when we move to GA, we'd sign with a release key
14:55:52 <eedri> mburns, got it
14:56:04 <mburns> RobertM: not releases from jenkins, builds from jenkins that will eventually make up a release
14:56:17 <mburns> but i would very strongly object to signing anything automatically
14:56:56 <RobertM> ok so they could be signed as a final step to moving packages to stable
14:57:10 <quaid> +1 to Infra signing, we just need to work out the process
14:57:19 <quaid> how can we tackle this for the 3.1 release?
14:57:28 <RobertM> Who wants to take on the project for getting this done?
14:57:50 <RobertM> It sounds great but it will require time and someone willing to do the work to get it done.
14:57:54 <mburns> quaid: too late for 3.1
14:58:15 <quaid> mburns: do you think this is something you can wrangle together initially, then we find a few key signers to help, etc.
14:58:27 <mburns> package signing needs to happen prior to the node iso build
14:58:53 <mburns> quaid: not likely in the next 3-4 weeks
14:59:17 * mburns would recommend that the release manager and release manager backup lead this effort
14:59:33 <RobertM> 1st step is getting someone willing to take on the task :)
15:00:00 <mburns> i normally would be, but i just don't have time over the next month...
15:00:20 <mburns> i'll talk to Ofer and see if we can coordinate something
15:00:51 <quaid> that helps, thanks
15:01:02 * eedri thinks we should put more effort in draw more people to infra team..
15:01:05 <RobertM> I suggest we table this until someone decides they want to take up the project.
15:01:27 <eedri> maybe ofer free ovirt t-shirts :)
15:02:11 <quaid> #action mburns to see if he and ofer can figure out signing process and details
15:02:18 <RobertM> Personally I like Red Hat swaq myself :)
15:02:29 <eedri> RobertM, my closet is packed with them..
15:02:43 <eedri> RobertM, a swag for every occation
15:03:01 <RobertM> Well eedri next time I am in Irasel I will need to pick some of it up :)
15:03:09 <eedri> RobertM, my treat
15:03:20 <RobertM> Let since I have never been out of the US not likely to happen anytime soon
15:03:29 <eedri> :)
15:03:59 <eedri> quaid, moving on to jenkins topic?
15:04:00 <mburns> i'll talk to Ofer about it, and try to get a plan together
15:04:07 <mburns> one more thing prior to jenkins?
15:04:20 <mburns> more just a "something to think about" topi
15:04:21 <mburns> c
15:04:24 <RobertM> mburns, If you need help let me know but I don't want to be the driver on that one
15:05:00 <mburns> do we want to have a presentation on ovirt infrastructure at the workshop in Barcelona in November?
15:05:18 <quaid> well, we are at the top of the hour
15:05:29 <eedri> i wouldn't mind being there and talking about jenkins infra...
15:05:37 <quaid> no other meeting conflict here, but
15:05:50 <mburns> just something to think about
15:05:51 <eedri> but what would we present?
15:05:57 <quaid> +1 to Infra presentation
15:06:04 * mburns trying to convince my boss to send me too
15:06:10 <quaid> eedri: developer infrastructure is pretty cool topic
15:06:12 <mburns> so i could probably cover it
15:06:28 <mburns> but if we want to have something, we should pitch it soon
15:06:32 <eedri> quaid, so talking about the gerrit process + jenkins + puppet maybe
15:06:45 <eedri> quaid, could be similar to the presentation i did in last jenkins conf
15:06:48 <quaid> eedri: exactly
15:07:12 <quaid> question - do we want to move on to Jenkins for the day, or close the meeting for the week?
15:07:24 <eedri> quaid, i just got action to add
15:07:28 <eedri> quaid, so we won't forget it
15:07:29 * quaid is conscious of not staying to the hour limit very well, each week
15:07:30 <RobertM> +1 Move on to Jenkins
15:07:36 <quaid> #topic Jenkins
15:07:57 * mburns has to leave and do some other work, but ping me if you need me
15:08:46 <eedri> RobertM, has pointed to me that nighly builds don't have proper rpm names
15:09:16 <eedri> RobertM, and they have fixed versions instead of an rpm name with git sha or build#
15:10:14 <eedri> we should change all rpms jobs to create proper rpms names.
15:10:32 <eedri> i belive this could be done with RELEASE_VERSION or similar env while running 'make rpm'
15:10:49 <RobertM> Quick round up.  Jenkins Master is overloaded and we can't really add patch level builds until we do something about it.  We have nightly being moved but the file names will prevent updates because it is missing build number / git keys.
15:11:03 <mburns> eedri: we do that for ovirt-node{,-iso}
15:11:20 <eedri> mburns, great, so it needs to be applied to other projects as well
15:11:22 <mburns> eedri: spec file has entry in RELEASE for %{EXTRA_RELEASE}
15:11:38 <mburns> and we set that in the build process
15:11:42 <RobertM> mburns, ovirt-node-stable/ovirt-node-iso is being moved nightly and it is formated correctly
15:11:55 <eedri> mburns, do you know if it supported on other projects makefile?
15:12:19 <mburns> eedri: i don't know
15:12:34 <RobertM> The files were named correctly at one point.
15:12:46 <mburns> eedri: it's pretty easy to do if they're not set right
15:12:58 <RobertM> I know because I attempted to install this before the name was changed from xxx-0001
15:13:01 <eedri> mburns, will it require change to makefiles? or it can be done from the command line
15:13:30 <mburns> eedri: probably need spec file updates
15:13:35 <mburns> possibly makefile too
15:13:57 <mburns> eedri: might be able to get away without makefile changes, but i'd have to look
15:14:09 <eedri> you think we should raise it on ovirt meeting?
15:14:31 <mburns> probably
15:15:19 <RobertM> # RPM version
15:15:19 <RobertM> APP_VERSION:=$(shell cat pom.xml | grep '<engine.version>' | awk -F\> '{print $$2}' | awk -F\< '{print $$1}')
15:15:19 <RobertM> RPM_VERSION:=$(shell echo $(APP_VERSION) | sed "s/-/_/")
15:15:21 <eedri> quaid, shuld we mark this as action?
15:16:24 <eedri> RobertM, i think the idea is to give the rpm a unique name using the git sha
15:16:32 <eedri> RobertM, or the jenkins build#
15:16:45 <mburns> we use git and date, iirc
15:17:32 <eedri> #action to find out how to create rpm names with git+date for ovirt projects (code change or environment vars)
15:17:48 <RobertM> build # is easier to use in my option since it is easier to see what build someone is using but date is fine.
15:18:00 <quaid> eedri: yes, you can do that
15:18:15 <quaid> actually, anyone can do an #action :)
15:18:22 * quaid is on a call now, lost track of the thread here, sorry
15:19:04 <RobertM> What is everyone though on #ovirt-jenkins keep, continue testing, or kill?
15:19:18 <eedri> RobertM, +1 for keeping alive
15:19:48 <eedri> RobertM, servers a bit as audit log
15:20:09 <mburns> keep alive, but change to fail/stopped failing/still failing in this channel...
15:20:17 <eedri> RobertM, i don't think though users will know about failures from there..
15:20:33 <RobertM> ok
15:20:43 <eedri> +1 on keeping only failures
15:20:45 <quaid> +1 to a logging channel, keep #ovirt from being overwhelmed; it's natural that we are adding bots etc.
15:20:45 <mburns> or maybe setup ovirtbot to handle that part of it
15:20:48 <eedri> too much info right now
15:22:12 <RobertM> I have been tweaking down the chatter some.  But putting failed summary only into #ovirt can be done.
15:22:19 <eedri> +1 for having #ovirt-jenkins-failures and #ovirt-jenkins-log
15:22:35 <eedri> or adding the failures to #ovirt also might work
15:22:57 <mburns> leave ovirt-jenkins as is, but put failures here i think
15:23:19 <RobertM> +1 for keeping ovirt-jenkins for all info and failure to #ovirt
15:23:29 <eedri> +1
15:23:43 <eedri> RobertM, does it suppots coloring text?
15:24:08 <eedri> RobertM, for marking failures in bright red or similar
15:24:12 <RobertM> I don't think so but I can check
15:25:52 <RobertM> So are we at constance to make failure / fixed reports go to #ovirt and keep ovirt-jenkins as is (Check for color coding)?
15:30:01 <RobertM> It looks like Jenkins IRC doesn't support color or reporting to two diff irc channels with diff Notification Strategy's will look deeper.
15:31:33 * eedri have to go
15:33:21 <eedri> RobertM, i think you can end the meeting
15:33:32 <RobertM> before we end the meeting anyone want Jabber notictions on Failures?
15:33:47 <RobertM> Out side of that lets end the meeting
15:35:35 <mburns> ok
15:35:37 <mburns> #endmeeting