View source for User:NeilK/Worklog/2010-12-14 to 2010-12-20

Projects: Upload Wizard, MediaStorage, Resource Loader

Status: solid

Last week:
* Mostly OOO sick
* Worked on MediaStorage
* Working on UploadWizard patches for next week (Tuesday)
** didn't have a patch ready, instead worked on the concurrency bugs since they indicate a deeper issue

This week:
* UploadWizard bugs
* Able to help with Resource Loader a bit

== Tuesday Dec 14 2010 ==

* [http://wikitech.wikimedia.org/view/Media_server/2011_Media_Storage_plans/Conference_call_2010-12-14 Conference call re: MediaStorage]
* Weekly Features meeting
* Committed some refactors to HTMLForm, along with a way to list and clear all stashed files from UploadStash. This was a feature I was trying to add Friday in an oh-wouldn't-this-be-helpful-while-debugging mood and it took way too long. Upside, at least now I understand MediaWiki's HTMLForm and other HTML classes, since I was trying to do everything the right way. Now, back to Roan's concurrency patch, which was the whole point of this.

== Wednesday Dec 15 2010 ==

* Figured out the concurrency issue and decided that we cannot actually fix this problem at the moment. I fiddled with Roan's idea of using separate memcache items to store everything and it works, although that switches us over to a completely new (and even more fragile) way of storing temporary upload data. That wouldn't be so bad, but we ''also'' rely on existing public API methods for the final steps of publishing, which all assume we are stashing files in the ''old'' way. So: we lose, unless we also rewrite those API methods, which aren't *quite* expired...

The right thing is to rearchitect all of MediaWiki to use a database table for stashed upload data. For many reasons -- reliability, concurrency, cross-colo functionality. Sessions and memcache are for stashing stuff that's slightly expensive to calculate, not for building apps on top of. Also, we can't impose any policy on sessions or memcache, like "must retain this data for as long as it takes to upload the file" -- for all we know the upload could take hours.

Deferring that until later. In the meantime we can do one upload at a time. (We could also rejigger the front end so that no upload was ever started within one or two seconds of each other, but that mitigates the problem, doesn't solve it -- you can't control timing with HTTP APIs).

== Thursday Dec 16 2010 ==

Worked on the highest priority frontend bugs. Fixed Safari 3.2 (and probably a lot of other legacy browsers in the process; there was a javascript backwards incompatibility). Oddly I don't have any environment where I can run Mac Safari 3.2 (Snow Leopard), but I realized it's also available for Windows. I hope the issue was the same.

It would have been nice if Calcey had noted which Safari was actually failing. I couldn't replicate the issue in 5, for obvious reasons. Then I remembered to check their test plan; I forgot they were going to test in a slightly outmoded version of Safari.

Set up a good environment to test IE8, and also got a good handle on why IE8 fails. No fix for the latter yet.

== Friday Dec 17 2010 ==

Bryan Tong Minh noticed my uploadstash-backend branch and said that if I'm rewriting it, it should be kicked out of 1.17. Makes sense, but perhaps I could accelerate writing the db version instead.

Dropped into the MogileFS IRC channel to ask a couple of questions, turned into a bit longer session with main developer (Dormando). Added a lot of info to [http://wikitech.wikimedia.org/view/Media_server/Distributed_File_Storage_choices MediaStorage Distributed Filesystem Choices].

Worked on this IE7/8 issue and tracked it down to the 'file' parameter being curiously missing on the PHP side. Although after installing IE8 dev tools (IE8 development now merely sucks, instead of sucking HARD) it appears to be fully part of the form as far as javascript is concerned on IE's end. For some reason I can't config Wireshark to read incoming network data from my IE8 VM, so I don't *really* know what the hell IE is sending. Lots of fruitless fiddling with config. Maybe should get some help here.

Some people debugging on IE use an HTTP proxy to capture WTH it is sending. Annoying that I can't do this with Wireshark.

== Monday Dec 20 2010 ==

Moved desks.

Added a bit more data to the Distributed FS page.

Consulted Ryan Lane on the Wireshark issue when debugging my IE8 inside VMWare. We tried a number of things (even going to raw tcpdump) and he is stumped too. Somehow VMWare is arranging that Apache gets the data, but we can't figure out how it's getting there. Will try IRC or scouring the internets, if not will give up and ... try running wireshark within windows? ...use proxy?