r72060 MediaWiki - Code Review archive

Repository:MediaWiki
Revision:r72059‎ | r72060 | r72061 >
Date:22:55, 31 August 2010
Author:catrope
Status:deferred (Comments)
Tags:
Comment:
resourceloader: Finish implementation of file dependency tracking
Modified paths:
  • /branches/resourceloader/phase3/includes/ResourceLoader.php (modified) (history)
  • /branches/resourceloader/phase3/includes/ResourceLoaderModule.php (modified) (history)

Diff [purge]

Index: branches/resourceloader/phase3/includes/ResourceLoader.php
@@ -84,7 +84,7 @@
8585 protected static function filter( $filter, $data ) {
8686 // FIXME: $file is not used by any callers as path rewriting is currently kinda broken
8787 global $wgMemc;
88 - $key = wfMemcKey( 'resourceloader', $filter, md5( $data ) );
 88+ $key = wfMemcKey( 'resourceloader', 'filter', $filter, md5( $data ) );
8989 $cached = $wgMemc->get( $key );
9090 if ( $cached !== false && $cached !== null ) {
9191 return $cached;
Index: branches/resourceloader/phase3/includes/ResourceLoaderModule.php
@@ -119,6 +119,9 @@
120120 private $loaders = array();
121121 private $parameters = array();
122122
 123+ // In-object cache for file dependencies
 124+ private $fileDeps = null;
 125+
123126 /* Public methods */
124127
125128 /**
@@ -304,7 +307,29 @@
305308 }
306309
307310 public function getStyle( $skin ) {
308 - return $this->getPrimaryStyle() . "\n" . $this->getSkinStyle( $skin );
 311+ $style = $this->getPrimaryStyle() . "\n" . $this->getSkinStyle( $skin );
 312+
 313+ // Extract and store the list of referenced files
 314+ $files = CSSMin::getLocalFileReferences( $style );
 315+
 316+ // Only store if modified
 317+ if ( $files !== $this->getFileDependencies( $skin ) ) {
 318+ $encFiles = FormatJson::encode( $files );
 319+ $dbw = wfGetDb( DB_MASTER );
 320+ $dbw->replace( 'module_deps',
 321+ array( array( 'md_module', 'md_skin' ) ), array(
 322+ 'md_module' => $this->getName(),
 323+ 'md_skin' => $skin,
 324+ 'md_deps' => $encFiles,
 325+ )
 326+ );
 327+
 328+ // Save into memcached
 329+ global $wgMemc;
 330+ $key = wfMemcKey( 'resourceloader', 'module_deps', $this->getName(), $skin );
 331+ $wgMemc->set( $key, $encFiles );
 332+ }
 333+ return $style;
309334 }
310335
311336 public function getMessages() {
@@ -341,7 +366,7 @@
342367 (array)self::getSkinFiles( $skin, $this->skinScripts ),
343368 (array)self::getSkinFiles( $skin, $this->skinStyles ),
344369 $this->loaders,
345 - $this->getFileDependencies( $lang, $skin )
 370+ $this->getFileDependencies( $skin )
346371 );
347372 return max( array_map( 'filemtime', $files ) );
348373 }
@@ -426,13 +451,30 @@
427452 * @return array of files
428453 */
429454 protected function getFileDependencies( $skin ) {
430 - $dbr = wfGetDb( DB_SLAVE );
431 - $deps = $dbr->selectField( 'module_deps', 'md_deps', array(
432 - 'md_module' => $this->getName(),
433 - 'md_skin' => $skin,
434 - ), __METHOD__
435 - );
436 - return $deps ? FormatJson::decode( $deps ) : array();
 455+ // Try in-object cache first
 456+ if ( !is_null( $this->fileDeps ) ) {
 457+ return $this->fileDeps;
 458+ }
 459+
 460+ // Now try memcached
 461+ global $wgMemc;
 462+ $key = wfMemcKey( 'resourceloader', 'module_deps', $this->getName(), $skin );
 463+ $deps = $wgMemc->get( $key );
 464+
 465+ if ( !$deps ) {
 466+ $dbr = wfGetDb( DB_SLAVE );
 467+ $deps = $dbr->selectField( 'module_deps', 'md_deps', array(
 468+ 'md_module' => $this->getName(),
 469+ 'md_skin' => $skin,
 470+ ), __METHOD__
 471+ );
 472+ if ( !$deps ) {
 473+ $deps = '[]'; // Empty array so we can do negative caching
 474+ }
 475+ $wgMemc->set( $key, $deps );
 476+ }
 477+ $this->fileDeps = FormatJson::decode( $deps, true );
 478+ return $this->fileDeps;
437479 }
438480
439481 /**

Comments

#Comment by Tim Starling (talk | contribs)   12:50, 17 September 2010

My profiling is telling me that this module_deps code is a piece nice juicy low hanging fruit ready for optimisation.

One issue is that memcached fetches are not especially fast. On Wikimedia they take about 500us real time on average. Loading the startup module calls ResourceLoaderFileModule::getFileDependencies() 70 times, causing 70 memcached fetches, so on Wikimedia we'd be looking at 35ms real time, which is a lot.

The question I have is whether this dependency feature is needed at all. The CSS files all have versions in the URLs:

	/* @embed */
	background-image: url(images/watch-icons.png?1);

If you changed that version, the timestamp on the CSS would be changed and the image file would be regenerated. That scheme might cause problems for third-party users making their own skins, but all we need to do for them is to respect $wgCacheEpoch, which we should be doing anyway, and to make sure it's documented that $wgCacheEpoch needs to be updated when images embedded in CSS change.

#Comment by Catrope (talk | contribs)   13:29, 17 September 2010

We already do automagic versions for image files by appending the mtime to the URL (CSSMin.php line 97). The fact that the ?1's are still there in the CSS files is a glitch.

This means that when the image's mtime changes, the generated CSS has to be updated to reflect that (and to re-embed the data URL, of course), which means we still need to track these images. You're right that the number of memcached fetches is totally ridiculous and should be lowered. This could be done by loading the entries for all requested modules at once.

I'm personally also doubting how useful memcached even is here: it shouldn't be too expensive to grab ~70 rows from the database with a simple, indexed query relative to a memcached fetch, should it?

#Comment by Catrope (talk | contribs)   13:32, 17 September 2010

And yes, we should respect $wgCacheEpoch. Filed this as bug 25201.

Status & tagging log