Archive:Hardware and hosting report: Difference between revisions

From Wikimedia Foundation Governance Wiki
Content deleted Content added
Anthere (talk | contribs)
No edit summary
m wikimedia squids, not wikimania squid; paragraphs
Line 3: Line 3:
===General report===
===General report===


''By Domas Mituzas'' - september 2005
''By Domas Mituzas'' - September 2005


Already in March it was clear that we needed more hardware to solve our main performance bottlenecks, but there was lots of hesitation on what to buy. This somewhat ended in mid-April, when we ordered 20 new application server (Apache) boxes, which were deployed in May. Then again, our main performance bottleneck happened to be our database environment, which was resolved by ordering and deploying two shiny new dual-Opteron boxes with 16GB of RAM each, accompanied by an external Just a Bunch of Disks (JBOD) enclosure. In this configuration we eliminated previous bottlenecks, as disk performance and in-memory caches were critical points. These two boxes have already shown to be capable of handling 5000 queries per second each without any sweating and were of great aid during content rebuilds during the MediaWiki 1.5 upgrade (we could run live site without any significant performance issues).
Already in March it was clear that we needed more hardware to solve our main performance bottlenecks, but there was lots of hesitation on what to buy. This somewhat ended in mid-April, when we ordered 20 new application server (Apache) boxes, which were deployed in May. Then again, our main performance bottleneck happened to be our database environment, which was resolved by ordering and deploying two shiny new dual-Opteron boxes with 16GB of RAM each, accompanied by an external Just a Bunch of Disks (JBOD) enclosure. In this configuration we eliminated previous bottlenecks, as disk performance and in-memory caches were critical points. These two boxes have already shown to be capable of handling 5000 queries per second each without any sweating and were of great aid during content rebuilds during the MediaWiki 1.5 upgrade (we could run live site without any significant performance issues).


Lots of burden was removed from databases by using some more efficient code, disabling really slow functions and, notably, deployment of the new Lucene search. Lucene can run on cheap Apache boxes instead of our jumbo (well, not that really in enterprise scale) DBs, therefore we could scale up quite a lot since December with the same old, poor boxes. Archives (article history) were also placed on cheap Apache boxes thus freeing expensive space on the database servers. Image server overloads were temporarily resolved by distributing content to several servers, but a more modern content storage system is surely required and planned.<br>
Lots of burden was removed from databases by using some more efficient code, disabling really slow functions and, notably, deployment of the new Lucene search. Lucene can run on cheap Apache boxes instead of our jumbo (well, not that really in enterprise scale) DBs, therefore we could scale up quite a lot since December with the same old, poor boxes. Archives (article history) were also placed on cheap Apache boxes thus freeing expensive space on the database servers. Image server overloads were temporarily resolved by distributing content to several servers, but a more modern content storage system is surely required and planned.

There were several downtimes related to Colocation facility power and network issues, of which the longest one was during our move (on wheels!) to a new facility, where we have more light, power, space and fresh air. Anyway, acute withdrawals were cured by working Wikis.
There were several downtimes related to Colocation facility power and network issues, of which the longest one was during our move (on wheels!) to a new facility, where we have more light, power, space and fresh air. Anyway, acute withdrawals were cured by working Wikis.


Line 16: Line 17:
===Multilingual error messages finally implemented !===
===Multilingual error messages finally implemented !===


On the 28 of september, '''Mark Ryan''' announced that multilingual messages had now been implemented on the Wikimania squids. Here is an incomplete list of those in IRC who helped with translations: taw, Mackowaty, WarX, SuiSui, aoineko, Submarine, Rama, Frieda, Quistnix, galwaygirl, Fenix, mnemo and avatar. Particular thanks must go to fuddlemark for extensive Javascript help, and to Jeronim for implementing the new message across the squids. Everyone's help
On the 28 of september, '''Mark Ryan''' announced that multilingual messages had now been implemented on the Wikimedia squids. Here is an incomplete list of those in IRC who helped with translations: taw, Mackowaty, WarX, SuiSui, aoineko, Submarine, Rama, Frieda, Quistnix, galwaygirl, Fenix, mnemo and avatar. Particular thanks must go to fuddlemark for extensive Javascript help, and to Jeronim for implementing the new message across the squids. Everyone's help
has been greatly appreciated. :)
has been greatly appreciated. :)



Revision as of 19:14, 7 October 2005

Q3 report

General report

By Domas Mituzas - September 2005

Already in March it was clear that we needed more hardware to solve our main performance bottlenecks, but there was lots of hesitation on what to buy. This somewhat ended in mid-April, when we ordered 20 new application server (Apache) boxes, which were deployed in May. Then again, our main performance bottleneck happened to be our database environment, which was resolved by ordering and deploying two shiny new dual-Opteron boxes with 16GB of RAM each, accompanied by an external Just a Bunch of Disks (JBOD) enclosure. In this configuration we eliminated previous bottlenecks, as disk performance and in-memory caches were critical points. These two boxes have already shown to be capable of handling 5000 queries per second each without any sweating and were of great aid during content rebuilds during the MediaWiki 1.5 upgrade (we could run live site without any significant performance issues).

Lots of burden was removed from databases by using some more efficient code, disabling really slow functions and, notably, deployment of the new Lucene search. Lucene can run on cheap Apache boxes instead of our jumbo (well, not that really in enterprise scale) DBs, therefore we could scale up quite a lot since December with the same old, poor boxes. Archives (article history) were also placed on cheap Apache boxes thus freeing expensive space on the database servers. Image server overloads were temporarily resolved by distributing content to several servers, but a more modern content storage system is surely required and planned.

There were several downtimes related to Colocation facility power and network issues, of which the longest one was during our move (on wheels!) to a new facility, where we have more light, power, space and fresh air. Anyway, acute withdrawals were cured by working Wikis.

There was some impressive development outside Florida as well. A new datacenter in Amsterdam, generously supplied by Kennisnet, provided us with a capability to cache content for whole Europe and neighboring regions. Moreover, it enabled us to build distributed DNS infrastructure, and preparations are made to serve static content from there in case of emergencies. Various other distribution schemes are researched as well.

Currently there are preparations made to deploy our content in a Korea datacenter provided by Yahoo. There we sure will use our established caching technology, but we might already take one step further and put our master content servers for regional languages there. As well, further expansion of our existing Florida content-processing facility is thought about.

Multilingual error messages finally implemented !

On the 28 of september, Mark Ryan announced that multilingual messages had now been implemented on the Wikimedia squids. Here is an incomplete list of those in IRC who helped with translations: taw, Mackowaty, WarX, SuiSui, aoineko, Submarine, Rama, Frieda, Quistnix, galwaygirl, Fenix, mnemo and avatar. Particular thanks must go to fuddlemark for extensive Javascript help, and to Jeronim for implementing the new message across the squids. Everyone's help has been greatly appreciated.  :)

Now, we just hope not to see these messages too often...

/Archives 2005