20:00:09 <mmcgrath> #startmeeting
20:00:12 * ricky 
20:00:21 <mmcgrath> #topic Infrastructure -- Who's here?
20:00:35 * ricky 
20:00:41 * LinuxCode 
20:00:44 * SmootherFrOgZ is
20:00:45 * lmacken 
20:00:48 * mmcgrath is here
20:00:50 * nirik is lurking around.
20:01:17 <mmcgrath> Ok, well lets get started
20:01:24 <mmcgrath> #topic Infrastructure -- F11 Release
20:01:40 <mmcgrath> .ticket 1395
20:01:43 <zodbot> mmcgrath: #1395 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1395
20:01:51 <mmcgrath> So what went wrong with this release?
20:02:02 <mmcgrath> All in all I think it went very well.
20:02:04 <nirik> very little I think. ;)
20:02:20 <mmcgrath> The early bit flip (thanks f13) provided a 75% mirror rate when the anouncement went out.
20:02:24 * herlo is lurking
20:02:27 <mmcgrath> thats much better then for F10.
20:02:29 <ricky> I didn't expect docs links to change, so we had to do some last-minute updates that made us a bit late
20:02:38 <ricky> We'll make sure to double check that better next time around.
20:02:48 <mmcgrath> Also, as mdomsch has pointed out many times, the torrents were screaming.  The fastest torrent dl's I've ever had during a release.
20:03:01 * sijis is here.
20:03:03 <mmcgrath> 15-20 minutes for 4 isos compared to 8 or 9 hours previously.
20:03:04 * ianweller 
20:03:11 * abadger1999 here
20:03:17 <nirik> yeah, due to them opening early we got a lot more seeders at release time.
20:03:23 <mmcgrath> ricky: should we make that a docs task or an us task?
20:03:45 <ricky> That'll be a websites task, I'll send links to the test page to list in advance.
20:04:29 * johe_ here too, by the way
20:04:29 <mmcgrath> k
20:04:52 <mmcgrath> The other thing that went kind of wacky was the bit leaks we had the thursday or friday before hand.
20:04:57 <mmcgrath> I think f13 is working on those though.
20:05:16 <mmcgrath> Oh, one other thing.
20:05:18 <mmcgrath> Proxy3 was hurting.
20:05:26 <mmcgrath> And I think I know why but I need to do a bit more research.
20:05:30 <mmcgrath> long story short it was doing too much.
20:05:41 <mmcgrath> I think it's just because it's one of our slower boxes in terms of cpu and disk.
20:05:49 <ricky> Did the x86_64 vs. i386 difference also come into play?
20:06:06 <mmcgrath> ricky: not this time.  But proxy1 and proxy2 are kind of odd because they're behind a balancer.
20:06:07 * mmcgrath needs to fix that.
20:06:14 <ricky> Ah
20:06:37 <mmcgrath> But once the release was out the door....  Things went great.
20:06:42 <mmcgrath> Anyone have anything else on this topic?
20:06:42 <rjune_wrk> why are they behind the balancer, but the slowest box has to face the cold cruel world all alone?
20:07:04 <mmcgrath> rjune_wrk: well, we've got 4 sites, only one of which has a balancer.
20:07:19 <mmcgrath> so we've got 5 proxy servers, 2 behind the balancer, the others are all at the various sites.
20:07:23 <rjune_wrk> ah, ok.
20:07:44 <mmcgrath> Ok, anyone have anything else?
20:08:00 <mmcgrath> Solid
20:08:08 <mmcgrath> #topic Infrastructure -- Fedora Community
20:08:12 <lmacken> We deployed our first TurboGears2 application this week :)
20:08:17 <mmcgrath> So fedora community shipped.
20:08:22 <mmcgrath> lmacken: you want to talk a bit about all of it?
20:08:29 <lmacken> Sure
20:08:36 <SmootherFrOgZ> \o/
20:08:37 <lmacken> Software speaking, not much went wrong.
20:08:37 * mmcgrath is talking about https://admin.fedoraproject.org/community/
20:08:40 <lmacken> I hit some insane traceback during staging that was caused by pointing to the unresolvabl
20:08:45 <lmacken> e db1.stg, instead of just db1.  Other than that, everything went very smooth.
20:09:03 <lmacken> It's using both of our new memecached servers, using postgres on db2 for the feed store, and currently load balancing to mod_wsgi on app1 and 2
20:09:22 <lmacken> and it's the snappiest I've ever seen the app as of yet
20:09:39 <lmacken> but anyway, I did another update today to fix a variety of bugs, and we're going to keep cranking on it
20:09:42 <mmcgrath> It also means we can, at our leasure, convert our apps to TG2.
20:09:50 <lmacken> I want to plan a post-mortem when spot gets back from vacation
20:10:02 <lmacken> so we can talk about what happened, and how we can do it better
20:10:20 <mmcgrath> lmacken: is the whole tg2 stack in Fedora yet?  I need to install and start playing with it.
20:10:39 <lmacken> the whole TurboGears2/FedoraCommunity/Moksha stack is in our fedora-infrastructure repo, and a chunk of it is still under review
20:10:50 <ricky> Does our tg2 support mean close-to automatic support for pylons as well?
20:10:57 <lmacken> ricky: yes
20:11:00 <ricky> Awesome
20:11:03 <lmacken> pylons should work fine now
20:11:12 <mmcgrath> excellent.
20:11:15 <mmcgrath> Anyone have any questions for Luke on fedora community?
20:11:20 <lmacken> I will start writing up a TurboGears2 SOP
20:11:45 <abadger1999> I'd like to get together some questions to ask spot wrt licensing.
20:12:14 <ricky> I guess that can be discussed at the post-mortem as soon as spot is back - can that be on IRC?
20:12:15 <abadger1999> That way we can have less round trips to legal.
20:12:24 <abadger1999> IRC +1
20:12:41 <mmcgrath> <nod> I just want to be better educated about it before I copy and paste :)
20:12:49 <lmacken> yes, it will be public, probably on irc or asterisk
20:13:20 <rjune_wrk> IRC+1
20:13:28 <mmcgrath> Anyone have anything else on this?
20:13:45 <abadger1999> asterisk -1; IRC is better for public stuff.
20:13:47 <lmacken> mmcgrath: when do you wanna spread to app3-6?
20:13:50 <mmcgrath> lmacken: good work, thanks for the time you've put in on it.
20:13:51 <lmacken> abadger1999: agreed
20:14:00 <mmcgrath> lmacken: today or tomorrow.  I know you did some updates, were they straight rpm updates?
20:14:22 <lmacken> mmcgrath: thank you.  Thanks for your help as well, couldn't have done it without you guys
20:14:51 <lmacken> mmcgrath: yep, straight rpm updates
20:14:55 <mmcgrath> coolz.
20:14:58 <mmcgrath> ok so next topic
20:15:05 <mmcgrath> #topic Infrastructure -- Memcached
20:15:16 <mmcgrath> we've been slipping down the slippery slope that is memcached.
20:15:21 <mmcgrath> a month ago we were using it for sessions on the wiki.
20:15:41 <mmcgrath> starting today we've got 3 apps using it and dedicated memcached servers.
20:16:02 <mmcgrath> the fas memcached stuff is going to exist just on the fas servers for security reasons.
20:16:09 <mmcgrath> everything else is on memcached[1-2]
20:16:21 <mmcgrath> ricky: do you want to remove memcached from puppet on app[1-2] or do you want me to
20:16:24 <mmcgrath> ?
20:16:29 <ricky> I'll get that after the meeting
20:16:38 <mmcgrath> solid
20:16:47 <mmcgrath> Nothing major here, just mentioning it.  Any questions?
20:16:56 <ricky> Speaking of security, I just want to let everybody know that load access into memcached = arbitrary command execution with python-memcached
20:17:07 <ricky> So be careful about that :-)
20:17:28 <mmcgrath> <nod>
20:17:52 <mmcgrath> k
20:18:01 <mmcgrath> #topic Infrastructure -- fas update
20:18:12 <mmcgrath> So we just got done (well, are still doing) the fas updates.
20:18:14 <mmcgrath> This was a pretty major change as changes go.
20:18:23 <mmcgrath> Especially in how our shell accounts and aliases are generated.
20:18:52 <mmcgrath> So keep your eyes out for oddities.
20:19:10 <mmcgrath> So far things have been going fine, we've been running in staging for a while now without issue.
20:19:30 <mmcgrath> I'm going to be moving our fas-sync to run more often on some of our more public servers
20:19:35 <mmcgrath> fedorapeople, fedorahosted, cvs, etc.
20:19:43 <mmcgrath> Any questions there?
20:19:51 <ricky> Woohoo :-)  My goal is <= 5 minutes sync time
20:20:11 <mmcgrath> :)
20:20:12 <mmcgrath> Ok
20:20:19 <mmcgrath> #topic Infrastructure -- Outage this morning
20:20:26 <mmcgrath> So we had some doom this morning for those of you hanging out in #sysadmin-noc
20:20:35 <mmcgrath> Not sure what happened yet but I opened a bug about it
20:20:36 <mmcgrath> https://bugzilla.redhat.com/505352
20:20:42 <mmcgrath> xen13 up and died.
20:20:46 <mmcgrath> and took db2 with it, twice.
20:20:53 <mmcgrath> I moved db2 to a different host, hasn't happened since.
20:21:10 <mmcgrath> at the moment, nothing new on this so I'll move on.
20:21:19 <mmcgrath> #topic Infrastructure -- Pkgdb update
20:21:27 <mmcgrath> abadger1999: we're on for a pkgdb rollout on Monday?
20:21:32 <abadger1999> that's right.
20:21:47 <mmcgrath> and this is a wsgi conversion?
20:21:50 <mmcgrath> any other changes?
20:21:57 <abadger1999> It's a major update that gets us using wsgi for the packagedb, all the bugfixes that accrued during change freeze,
20:22:11 <mmcgrath> I've still got smolt to convert.
20:22:11 <abadger1999> moving to usernames instead of userids i nthe public API.
20:22:18 <mmcgrath> after that I *think* all of our tg stuff will be good to go.
20:22:24 <ricky> Afdter pkgdb and smolt, will all of our apps be on haproxy + wsgi?
20:22:32 <abadger1999> Switching package status from the orphan owner to the status field.
20:22:38 <abadger1999> Lots of stuff changing.
20:23:35 <lmacken> I'll get a new bodhi ready to go at that time
20:23:44 <lmacken> I should probably get a new one into staging now so we can test bodhi#319
20:23:45 <mmcgrath> ricky: I believe so.
20:24:00 <abadger1999> lmacken: Excellent
20:24:03 <mmcgrath> lmacken: sounds good.
20:24:06 <mmcgrath> Anything else on this?
20:24:23 <abadger1999> Test out admin.stg.fedoraproject.org/pkgdb ! ;-)
20:24:31 <mmcgrath> hehehe
20:24:38 <mmcgrath> #topic Infrastructure -- releng1.stg
20:24:40 <abadger1999> I'm sure there's more bugs since this has major changes.
20:24:43 <ricky> After the pkgdb update, let's go through staging
20:24:44 <mmcgrath> speaking of staging
20:24:47 <mmcgrath> abadger1999: yeah :)
20:25:00 <mmcgrath> releng1.stg is nearing rediness.
20:25:01 <ricky> And make sure to get all the commits we want out of it before restarting the branch.
20:25:25 <mmcgrath> lmacken: I need to find out from you and jwb what is missing from it.  I think it's going to take a close eye to make sure we know what we're putting in staging.
20:25:32 <mmcgrath> cron jobs and things that might fail, that sort of thing.
20:25:47 <lmacken> mmcgrath: readonly mount of /mnt/koji?
20:25:52 <mmcgrath> lmacken: it'll have that.
20:26:10 <lmacken> ok, to test mashing and whatnot, we'll have to tweak some paths
20:26:35 <mmcgrath> lmacken: and I don't think anything mash like is installed yet.
20:26:47 <mmcgrath> but I figure we'll work through it one boy one until it's  usable for you and jwb.
20:27:00 <lmacken> ok, cool
20:27:01 <mmcgrath> I also assume you're going to need some sort of $TEMP_STORAGE
20:27:07 <lmacken> yeah
20:27:14 <mmcgrath> that's fairly large.  we'll figure that out when the time comes.
20:27:19 <mmcgrath> was jwb at FAD?
20:27:24 <lmacken> yep
20:27:33 <mmcgrath> probably traveling / busy
20:27:36 <mmcgrath> we'll get ahold of him.
20:27:38 <mmcgrath> ok, ricky :)
20:27:44 <mmcgrath> #topic Infrastructure -- Staging
20:28:00 <mmcgrath> So we need to refresh the staging branch at some point soon.
20:28:25 <mmcgrath> and make sure all of our staging commits get merged back with master.
20:28:37 <mmcgrath> ricky: what are a rundown of some of the commits left?
20:28:40 <mmcgrath> I know pkgdb had a bunch.
20:29:07 <ricky> Here's the full diff: http://ricky.fedorapeople.org/staging.diff
20:29:25 <ricky> It's mostly pkgdb, I thnk
20:29:27 <ricky> **think
20:29:55 <ricky> It's not that bad, so we might get away with not blowing it away
20:30:07 <ricky> But I guess that's something to look at once pkgdb changes have been merged.
20:30:11 <abadger1999> pkgdb will be synced after Monday, so that's good.
20:30:14 <mmcgrath> Yeah
20:30:20 <ricky> Cool.
20:30:25 <abadger1999> Do we have a good recipe for merging the changes yet?
20:30:32 <ricky> And thanks to mmcgrath for starting to merge often this week
20:30:37 <mmcgrath> All in all, considering we don't really monitor this interaction much... things are in better shape then they otherwise could be.
20:30:56 <mmcgrath> ricky: figured i'd make a habit of it
20:31:42 <ricky> abadger1999: It'll probably consist mostly of finding out what we want to keep or throw away
20:32:08 <ricky> The process of merging isn't that painful at all, just getting updates on half-finished or abandoned work in staging.
20:32:30 <abadger1999> ricky: I'm just wondering, can I copy changes from staging to production, commit, and git will figure it out?  Or do I need to use some git command to pick out specific changesets?
20:33:17 <ricky> It's cleaner to cherry-pick, but I *think* hand-copying will work as well.  I haven't tried doing work that way though, so I'm not sure how well it gets handled.
20:33:27 <ricky> I think the diff is short enough for us to go through it manually though.
20:33:39 <mmcgrath> I've hand-copied with success in the past.
20:33:47 <mmcgrath> but try to use git just so I'm mo'betta with it.
20:35:15 <abadger1999> Okay.  We'll see what it looks like after Monday's merge ;-)
20:35:15 <mmcgrath> Ok, anything else on that?
20:35:18 <mmcgrath> yeah
20:35:31 <mmcgrath> With that, we'll open the floor
20:35:37 <mmcgrath> #topic Infrastructure -- Open Floor
20:35:43 <mmcgrath> Anyone have anything they'd like to discuss?
20:35:46 <LinuxCode> apart from a million questions I have ?
20:35:49 <LinuxCode> lol
20:35:59 <ricky> Ask away :-)
20:36:01 <LinuxCode> I will have to figure out what is done for what purpose the enxt weeks
20:36:20 <LinuxCode> ricky, I ask you guys the next weeks, or we would be here all night
20:36:25 <LinuxCode> lol
20:36:32 <ricky> feel free to ask in #fedora-admin after the meeting as well
20:36:38 <LinuxCode> Id like to figure out how everything fits together
20:36:41 <LinuxCode> sure
20:36:59 <LinuxCode> I have one question though, the fas update was to kill bugs ?
20:37:08 <LinuxCode> it said so in the email I had
20:37:18 <abadger1999> LinuxCode: For some definition of bugs, yes.
20:37:40 <LinuxCode> abadger1999, I take it to make sure that people participate and not just sign up then ?
20:37:41 <abadger1999> Ther were bugs and there was also performance problems that were causing failures to occur.
20:37:46 <LinuxCode> ohh ok
20:37:50 <LinuxCode> i see
20:38:17 <ricky> The extra @fp.o aliases being created was a bug too
20:38:26 <LinuxCode> ricky, thats what I meant
20:38:27 <ricky> Unfortunately, it went unnoticed for a loong time.
20:38:47 <LinuxCode> would have been nice to give people a week though to sort themselves out
20:38:56 <ricky> The policy hasn't changed, it was just a bug that we weren't enforcing it.
20:38:59 <LinuxCode> I had the email today saying about the bug
20:39:04 <LinuxCode> hmm k
20:39:13 <LinuxCode> well it doesnt matter now
20:39:21 <LinuxCode> for me anyway
20:39:23 <LinuxCode> ;-}
20:39:25 <mmcgrath> heheh
20:39:33 <mmcgrath> Anyone have anything else to discuss?  If not we'll close the meeting early.
20:40:04 <mmcgrath> Closing in 30
20:40:34 <mmcgrath> 10
20:40:48 <mmcgrath> #endmeeting