r83529 MediaWiki - Code Review archive

Repository:MediaWiki
Revision:r83528‎ | r83529 | r83530 >
Date:16:47, 8 March 2011
Author:catrope
Status:resolved (Comments)
Tags:
Comment:
Make updateCollation.php a bit less murderous for WMF databases:
* Don't run a COUNT(*) query on what's potentially the entire categorylinks table on enwiki (hundreds of millions of rows). Put it in a miser mode check
* Wait for DB replication to catch up before processing the next batch. Implemented LoadBalancer::waitAll() for this purpose, which should behave more nicely than wfWaitForSlaves()
Modified paths:
  • /trunk/phase3/includes/db/LoadBalancer.php (modified) (history)
  • /trunk/phase3/maintenance/updateCollation.php (modified) (history)

Diff [purge]

Index: trunk/phase3/maintenance/updateCollation.php
@@ -45,9 +45,16 @@
4646 $this->addOption( 'force', 'Run on all rows, even if the collation is ' .
4747 'supposed to be up-to-date.' );
4848 }
 49+
 50+ public function syncDBs() {
 51+ $lb = wfGetLB();
 52+ $dbw = $lb->getConnection( DB_MASTER );
 53+ $pos = $dbw->getMasterPos();
 54+ $lb->waitForAll( $pos );
 55+ }
4956
5057 public function execute() {
51 - global $wgCategoryCollation;
 58+ global $wgCategoryCollation, $wgMiserMode;
5259
5360 $dbw = wfGetDB( DB_MASTER );
5461 $force = $this->getOption( 'force' );
@@ -55,30 +62,32 @@
5663 $options = array( 'LIMIT' => self::BATCH_SIZE );
5764
5865 if ( $force ) {
59 - $collationConds = array();
6066 $options['ORDER BY'] = 'cl_from, cl_to';
6167 } else {
6268 $collationConds = array( 0 =>
6369 'cl_collation != ' . $dbw->addQuotes( $wgCategoryCollation ) );
6470
65 - $count = $dbw->selectField(
66 - 'categorylinks',
67 - 'COUNT(*)',
68 - $collationConds,
69 - __METHOD__
70 - );
 71+ if ( !$wgMiserMode ) {
 72+ $count = $dbw->selectField(
 73+ 'categorylinks',
 74+ 'COUNT(*)',
 75+ $collationConds,
 76+ __METHOD__
 77+ );
7178
72 - if ( $count == 0 ) {
73 - $this->output( "Collations up-to-date.\n" );
74 - return;
 79+ if ( $count == 0 ) {
 80+ $this->output( "Collations up-to-date.\n" );
 81+ return;
 82+ }
 83+ $this->output( "Fixing collation for $count rows.\n" );
7584 }
76 - $this->output( "Fixing collation for $count rows.\n" );
7785 }
7886
7987 $count = 0;
8088 $row = false;
8189 $batchConds = array();
8290 do {
 91+ $this->output( 'Processing next ' . self::BATCH_SIZE . ' rows... ');
8392 $res = $dbw->select(
8493 array( 'categorylinks', 'page' ),
8594 array( 'cl_from', 'cl_to', 'cl_sortkey_prefix', 'cl_collation',
@@ -140,6 +149,8 @@
141150
142151 $count += $res->numRows();
143152 $this->output( "$count done.\n" );
 153+
 154+ $this->syncDBs();
144155 } while ( $res->numRows() == self::BATCH_SIZE );
145156 }
146157 }
Index: trunk/phase3/includes/db/LoadBalancer.php
@@ -338,6 +338,18 @@
339339 }
340340 wfProfileOut( __METHOD__ );
341341 }
 342+
 343+ /**
 344+ * Set the master wait position and wait for ALL slaves to catch up to it
 345+ */
 346+ public function waitForAll( $pos ) {
 347+ wfProfileIn( __METHOD__ );
 348+ $this->mWaitForPos = $pos;
 349+ for ( $i = 1; $i < count( $this->mServers ); $i++ ) {
 350+ $this->doWait( $i );
 351+ }
 352+ wfProfileOut( __METHOD__ );
 353+ }
342354
343355 /**
344356 * Get any open connection to a given server index, local or foreign

Follow-up revisions

RevisionCommit summaryAuthorDate
r835311.17wmf1: MFT r83529catrope17:21, 8 March 2011
r83558Merge r83555 from 1.17wmf1: fix wait for slaves code from r83529catrope00:24, 9 March 2011
r83874(bug 27975) Fix r83529 (slave catchup in updateCollation.php) to not try to w...catrope09:30, 14 March 2011
r865471.17: MFT r82413, r83529, r83545, r83874, r85544, r86065, r86346, r86477catrope20:24, 20 April 2011

Comments

#Comment by Hashar (talk | contribs)   17:15, 11 March 2011

Fixme per bug 27975

SHOW MASTER STATUS is send even when using a single database layout. This might trigger an user permission error from the database.

Status & tagging log