r101744 MediaWiki - Code Review archive

Repository:MediaWiki
Revision:< r101743‎ | r101744 | r101745 >
Date:23:17, 2 November 2011
Author:krinkle
Status:deferred (Comments)
Tags:
Comment:
[JSTesting] partial rewrite of TestSwarmMWFetcher
* Before rewrite (should've been a separate commit)
-- Removing savedPath/chdir code (unused). We probably shouldn't be changing paths. If need to execute commands on paths outside our current working directory we can provide relative or absolute paths to those functions.
-- Removing getFirstRev, using minRev directly.
-- Fix bug in __construct (using || instead of && to verify required options)
-- Fix bug in getNextFollowingRevId, distinguish between invalid id and no id at all. Return null if there is no next id (given id is >= HEAD)
-- Fix bug in log(); Call fclose() on the handle, not the file path. Pass handle as first argument to fwrite(), instead of file path as second argument.

* During/after ewrite:
-- Creating separate class for the install process so we have a context for stuff like paths and revision ids (instead of passing around everything all the time).
-- Removing getLogFile(). Instead putting the full path directly in getPaths()
-- Changing log path from /log/r123/debug.log to /logs/r123.log
-- Renaming variables and functions to make a clearer distinction between the local checkout and the remote repo (i.e. "getNextRev")
-- Changing order of functions. Defining functions in order of usage (and before they are used - where possible).

* Follows-up r101636

--

Right now it successfully does the checkout and installing of the wiki.
@todo:
- Append settings template to LocalSettings
- Append require() for global settings to LocalSettings
- TestSwarm job submitter (needs config for testswarm instance location)
- After running preprod-mode, got error:

Location: r101591/">http://localhost/mw/trunk/tools/testswarm/scripts/_checkouts_symlink/r101591/

A database error has occurred. Did you forget to run maintenance/update.php after upgrading? See: http://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: DELETE FROM msg_resource
Function: MessageBlobStore::clear
Error: 8 attempt to write a readonly database
Modified paths:
  • /trunk/tools/testswarm/scripts/testswarm-mw-fetcher-run.php (modified) (history)
  • /trunk/tools/testswarm/scripts/testswarm-mw-fetcher.php (modified) (history)

Diff [purge]

Index: trunk/tools/testswarm/scripts/testswarm-mw-fetcher-run.php
——@@ -1,15 +1,17 @@
22 <?php
33 /**
4 - * Testswarm fetcher example.
 4+ * Example for running the TestSwarm MediaWiki fetcher.
55 *
66 * Licensed under GPL version 2
77 *
8 - * @author Antoine "hashar" Musso © 2011
 8+ * @author Antoine "hashar" Musso, 2011
 9+ * @author Timo Tijhof, 2011
910 */
10 -require_once( "testswarm-mw-fetcher.php" );
1111
 12+date_default_timezone_set( 'UTC' );
 13+
1214 // Choose a mode below and the switch structure will forge options for you!
13 -#$mode = 'dev';
 15+$mode = 'dev';
1416 $mode = 'preprod';
1517 #$mode = 'prod';
1618
——@@ -17,26 +19,31 @@
1820 # Magic stuff for lazy people
1921 switch( $mode ) {
2022 # Options for local debuggings
21 - case 'dev': $options = array(
22 - 'debug' => true,
23 - 'root' => '/tmp/tsmw',
24 - 'url' => 'http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/testswarm/scripts',
25 - 'minrevision' => 88439, # will not fetch anything before that rev
26 - ); break;
 23+ case 'dev':
 24+ $options = array(
 25+ 'debug' => true,
 26+ 'root' => '/tmp/tsmw-trunk-dev',
 27+ 'svnUrl' => 'http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/testswarm/scripts',
 28+ 'minRev' => 88439, # will not fetch anything before that rev
 29+ );
 30+ break;
2731
2832 # Options fetching from phase3. Debug on.
2933 case 'preprod':
3034 $options = array(
3135 'debug' => true,
32 - 'root' => '/tmp/tsmw-trunk',
33 - 'url' => 'http://svn.wikimedia.org/svnroot/mediawiki/trunk/phase3',
34 - 'minrevision' => 101591,
35 - ); break;
 36+ 'root' => '/tmp/tsmw-trunk-preprod',
 37+ 'svnUrl' => 'http://svn.wikimedia.org/svnroot/mediawiki/trunk/phase3',
 38+ 'minRev' => 101591,
 39+ );
 40+ break;
3641
3742 default:
3843 print "Mode $mode unimplemented. Please edit ".__FILE__."\n";
3944 exit( 1 );
4045 }
4146
42 -$fetcher = new TestSwarmMWFetcher( $options );
43 -$fetcher->tryInstallNextRev();
 47+require_once( 'testswarm-mw-fetcher.php' );
 48+
 49+$main = new TestSwarmMWMain( $options );
 50+$main->tryFetchNextRev();
Index: trunk/tools/testswarm/scripts/testswarm-mw-fetcher.php
——@@ -5,10 +5,9 @@
66 * As of November 2nd 2011, this is still a work in progress.
77 *
88 * Latest version can be found in the Mediawiki repository under
9 - * /trunk/tools/testswarm/
 9+ * /trunk/tools/testswarm/
1010 *
1111 * Based on http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/testswarm/scripts/testswarm-mediawiki-svn.php?revision=94359&view=markup
12 - * (which only did a static dump of /resources and /tests/qunit).
1312 *
1413 * @author Timo Tijhof, 2011
1514 * @author Antoine "hashar" Musso, 2011
——@@ -21,277 +20,260 @@
2221 * to install any PECL extension.
2322 *
2423 * @todo We might want to abstract svn commands to later use git
25 - * @todo FIXME: Get the classes/function implied from MediaWiki somehow.
 24+ * @todo Create some kind of locking system (either inside this script or outside of it),
 25+ * to prevent this script from running if it is already running (since checking out & installing MediaWiki
 26+ * can easily take over 5 minutes).
2627 *
2728 * @example:
2829 * @code
2930 * $options = array(
30 - * 'root' => '',
31 - * 'url' => 'http://svn.wikimedia.org/svnroot/mediawiki/trunk/phase3',
 31+ * 'root' => '/tmp/testswarm-mw',
 32+ * 'svnUrl' => 'http://svn.wikimedia.org/svnroot/mediawiki/trunk/phase3',
3233 * );
33 - * $fetcher = new TestSwarmMWFetcher( $options );
34 - * $fetcher->tryInstallNextRev();
 34+ * $main = new TestSwarmMWMain( $options );
 35+ * $main->tryFetchNextRev();
3536 * @endcode
3637 */
37 -class TestSwarmMWFetcher {
 38+class TestSwarmMWMain {
3839
3940 /** Base path to run into */
4041 protected $root;
 42+
4143 /** URL to a subversion repository as supported by the Subversion cli */
42 - protected $url;
 44+ protected $svnUrl;
 45+
4346 /** subversion command line utility */
4447 protected $svnCmd = '/usr/bin/svn';
45 - /** whether to enable debugging */
46 - protected $debugEnabled = false;
47 - /** hold previous path when chdir() to a checkout directory */
48 - protected $savedPath = null;
 48+
 49+ /** Whether to enable debugging */
 50+ protected $debugMode = false;
 51+
4952 /** Minimum revision to start with. At least 1 */
50 - protected $minRevision = 1;
 53+ protected $minRev = 1;
5154
 55+ /** Path to log file */
 56+ protected $logPath;
 57+
 58+
 59+ /** GETTERS **/
 60+
 61+ public function getSvnCmd() { return $this->svnCmd; }
 62+ public function getSvnUrl() { return $this->svnUrl; }
 63+ public function getLogPath() { return $this->logPath; }
 64+
 65+
 66+ /** SETTERS **/
 67+
 68+ public function setLogPath( $path ) {
 69+ $this->logPath = $path;
 70+ return true;
 71+ }
 72+
 73+
 74+ /** INIT **/
 75+
5276 /**
5377 * Init the testswarm fetcher.
5478 *
55 - * @param @options Array: Required options are:
 79+ * @param @options array: Required options are:
5680 * 'root' => root path where all stuff happens
57 - * 'url' => URL for the repository
 81+ * 'svnUrl' => URL to the repository (or a subdirectory of it)
5882 * Other options:
5983 * 'svnCmd' => path/to/svn (default: /usr/bin/svn)
60 - * 'debug' => true/false
61 - * 'minrevision' => int (revision to start at)
 84+ * 'debug' => (default: false)
 85+ * 'minRev' => int (default: 1)
6286 */
63 - function __construct( $options = array() ) {
 87+ public function __construct( $options = array() ) {
6488
6589 // Verify we have been given required options
66 - if( !isset( $options['root'] )
67 - && !isset( $options['url'] )
68 - ) {
69 - throw new Exception( __METHOD__ . ": " . __CLASS__ . " constructor must be passed 'root' and 'url' options\n" );
 90+ if ( !isset( $options['root'] ) || !isset( $options['svnUrl'] ) ) {
 91+ throw new Exception( __METHOD__ . ": Required options 'root' and/or 'svnUrl' missing" );
7092 }
71 - $this->root = $options['root'];
72 - $this->url = $options['url'];
7393
74 - // and now the optional options
75 - if( isset($options['svnCmd'] ) ) {
 94+ $this->root = $options['root'];
 95+ $this->svnUrl = $options['svnUrl'];
 96+
 97+ // Default log file
 98+ $this->setLogPath( "{$options['root']}/logs/default.log" );
 99+
 100+ // Optional options
 101+ if ( isset( $options['svnCmd'] ) ) {
76102 $this->svnCmd = $options['svnCmd'];
77103 }
78 - if( isset($options['debug'] ) ) {
79 - $this->debugEnabled = true;
 104+
 105+ if ( isset( $options['debug'] ) ) {
 106+ $this->debugMode = true;
80107 }
81 - if( isset($options['minrevision'] ) ) {
82 - if( $options['minrevision'] < 1 ) {
83 - # minrevision = 0 will just screw any assumption made in this script.
 108+
 109+ if ( isset( $options['minRev'] ) ) {
 110+ if ( $options['minRev'] < 1 ) {
 111+ # minRev = 0 will just screw any assumption made in this script.
84112 # so we really do not want it.
85 - throw new Exception( __METHOD__ . ": " . __CLASS__ . " option 'minrevision' must be >= 1 \n" );
 113+ throw new Exception( __METHOD__ . ": Option 'minRev' must be >= 1 " );
86114 }
87 - $this->minRevision = $options['minrevision'];
 115+ $this->minRev = $options['minRev'];
88116 }
 117+
 118+ return $this;
89119 }
90120
91121 /**
92 - * Try to install the next revision after our latest install.
 122+ * Try to fetch the next revision (relative latest checkout in the checkouts directory).
93123 * This is the main entry point after construction.
94124 */
95 - function tryInstallNextRev() {
96 - // Setup checkouts dir if it does not exist yet (happens on initial run).
97 - $checkouts = "{$this->root}/checkouts";
98 - if( !file_exists( $checkouts ) ) {
99 - $this->mkdir( $checkouts );
100 - }
 125+ public function tryFetchNextRev() {
 126+ $this->prepareRootDirs();
101127
102128 // Now find out the next revision in the remote repository
103 - $next = $this->getNextRevisionId();
 129+ $next = $this->getNextCheckoutRevId();
104130 if ( !$next ) {
105 - $this->debug( __METHOD__ . " no next revision." );
 131+ $this->debug( 'No next revision', __METHOD__ );
106132 return false;
107133 } else {
108134 // And install it
109 - return $this->doInstallById( $next );
 135+ $fetcher = new TestSwarmMWFetcher( &$this, $next );
 136+ return $fetcher->run();
110137 }
111138 }
112139
113 - /**
114 - * This is the main function doing checkout and installation for
115 - * a given rev.
116 - *
117 - * @param $id integer: Revision id to install
118 - * @return
119 - */
120 - function doInstallById( $id ) {
121 - if( !is_int( $id ) ) {
122 - throw new Exception( __METHOD__ . " passed a non integer revision number\n" );
123 - }
124140
125 - $this->doCheckout( $id );
126 - $this->doInstall( $id );
127 - $this->doAppendSettings( $id );
 141+ /** SVN REVISION HELPERS **/
128142
129 - # TODO:
130 - // get list of tests (see the current file in svn/trunk/tools)
131 - // request to testswarm to add jobs to run these tests
132 - // --> 'api' POST request to TestSwarm/addjob.php (with login/auth token)
133 - }
134 -
135143 /**
136 - * Checkout a given revision in our specific tree.
137 - * Throw an exception if anything got wrong.
138 - * @todo Output is not logged.
139 - *
140 - * @param $id integer: Revision id to checkout.
 144+ * Get latest revision fetched in the working copy.
 145+ * @return integer
141146 */
142 - function doCheckout( $id ){
143 - $this->msg( "Checking out r{$id}" );
 147+ public function getLastCheckoutRevId() {
 148+ $paths = $this->getPathsForRev( 0 );
 149+ $checkoutPath = dirname( $paths['mw'] );
144150
145 - // create checkout directory
146 - $revPath = self::getPath( 'mw', $id );
147 - $this->mkdir( $revPath );
 151+ // scandir() sorts in descending order if given a nonzero value as second argument.
 152+ // PHP 5.4 accepts constant SCANDIR_SORT_DESCENDING
 153+ $subdirs = scandir( $checkoutPath, 1 );
 154+ $this->debug( "Scan of '{$checkoutPath}' returned:\n - /" . implode("\n - /", $subdirs ) );
148155
149 - # TODO FIXME : we might want to log the output of svn commands
150 - $cmd = "{$this->svnCmd} checkout {$this->url}@r{$id} {$revPath}";
151 - $this->exec( $cmd, $retval );
152 - if( $retval !== 0 ) {
153 - throw new Exception(__METHOD__." error running subversion checkout.\n" );
 156+ // Verify the directory is like 'r123' (it could be '.', '..' or even something completely different)
 157+ if ( $subdirs[0][0] === 'r' ) {
 158+ return substr( $subdirs[0], 1 );
 159+ } else {
 160+ return null;
154161 }
155 -
156 - // TODO handle errors for above commands.
157 - // $this->getPath( 'log' );
158162 }
159163
160164 /**
161 - * Install a given revision.
162 - *
163 - * @param $id integer: Revision id to run the installer for.
 165+ * Get the first revision after the given revision in the remote repository.
 166+ * @param $id integer
 167+ * @return null|integer: Null if there is no next, other wise integer id of next revision.
164168 */
165 - function doInstall( $id ) {
166 - $this->msg( "Installing r{$id}" );
 169+ public function getNextFollowingRevId( $id ) {
167170
168 - # Create database directory (needed on initial run)
169 - $this->mkdir(
170 - $this->getPath( 'db', $id )
171 - );
 171+ /**
 172+ * @todo FIXME: Takes a loooooooongg time to look up for "1:HEAD"
 173+ *
 174+ * @example:
 175+ * $ svn log -q -r101656:HEAD --limit 2 http://svn.wikimedia.org/svnroot/mediawiki/trunk/phase3
 176+ * ------------------------------------------------------------------------
 177+ * r101656 | aaron | 2011-11-02 19:47:04 +0100 (Wed, 02 Nov 2011)
 178+ * ------------------------------------------------------------------------
 179+ * r101666 | brion | 2011-11-02 20:36:49 +0100 (Wed, 02 Nov 2011)
 180+ * ------------------------------------------------------------------------
 181+ */
 182+ $nextRev = $id + 1;
 183+ $cmd = "{$this->svnCmd} log -q -r{$nextRev}:HEAD --limit 1 {$this->svnUrl}";
172184
173 - # Erase MW_INSTALL_PATH which would interact with the install script
174 - putenv( "MW_INSTALL_PATH" );
175 -
176 - // Now simply run the CLI installer:
177 - $cmd = "php {$this->getPath( 'mw', $id )}/maintenance/install.php \
178 - --dbname=testwarm_mw_r{$id} \
179 - --dbtype=sqlite \
180 - --dbpath={$this->getPath( 'db', $id )} \
181 - --pass=testpass \
182 - --showexceptions=true \
183 - --confpath={$this->getPath( 'mw', $id )} \
184 - WIKINAME \
185 - ADMINNAME
186 - ";
 185+ $retval = null;
187186 $output = $this->exec( $cmd, $retval );
188187
189 - print "Installer output for r{$id}:\n";
190 - print $output;
 188+ if ( $retval !== 0 ) {
 189+ throw new Exception(__METHOD__. ': Error running subversion log' );
 190+ }
191191
192 - if( $retval !== 0 ) {
193 - throw new Exception(__METHOD__." error running MediaWiki installer.\n" );
 192+ preg_match( "/r(\d+)/m", $output, $m );
 193+
 194+ if ( !isset( $m[1] ) ) {
 195+ // No next revision, given id is already >= HEAD
 196+ return null;
194197 }
 198+
 199+ $followingRev = (int)$m[1];
 200+ if ( $followingRev === 0 ) {
 201+ throw new Exception( __METHOD__ . " Remote returned a invalid revision id: '{$m[1]}'" );
 202+ }
 203+ return $followingRev;
195204 }
196205
197206 /**
198 - * TODO: implement :-)
199 - * @param $id integer: Revision id to append settings to.
 207+ * Get next changed revision for a given checkout
 208+ * @return String|Boolean: false if nothing changed, else the upstream revision just after.
200209 */
201 - function doAppendSettings( $id ) {
202 - $this->msg( "Appending settings for r{$id} installation (not implemented)" );
203 - return true;
 210+ public function getNextCheckoutRevId() {
 211+ $cur = $this->getLastCheckoutRevId();
 212+ if ( is_null ( $cur ) ) {
 213+ $this->debug( 'Checkouts dir empty? Looking up remote repo...', __METHOD__ );
 214+ $next = $this->minRev;
 215+ } else {
 216+ $next = (int)$cur;//$this->getNextFollowingRevId( $cur );
 217+ }
204218
205 - # Notes for later implementation:
 219+ $this->debug( __METHOD__ . ": Going to use r{$next}" );
 220+ return $next;
 221+ }
206222
207 - // append to mwPath/LocalSettings.php
208 - // -- contents of LocalSettings.tpl.php
209 - // -- require_once( '{$this->getPath('globalsettings')}' );"
210223
211 - /**
212 - * Possible additional common settings to append to LocalSettings after install:
213 - * See gerrit integration/jenkins.git :
214 - * https://gerrit.wikimedia.org/r/gitweb?p=integration/jenkins.git;a=tree;f=jobs/MediaWiki-phpunit;hb=HEAD
215 - *
216 - * $wgShowExceptionDetails = true;
217 - * $wgShowSQLErrors = true;
218 - * #$wgDebugLogFile = dirname( __FILE__ ) . '/build/debug.log';
219 - * $wgDebugDumpSql = true;
220 - */
221 - }
 224+ /** DIRECTORY STRUCTURE **/
222225
223 - /**
224 - * Utility function to log a message for a given id
225 - *
226 - * @param $msg String message to log. Will be prefixed with date("c")
227 - * @param $id Integer Revision id.
228 - */
229 - function log( $msg, $id ) {
230 - $file = $this->getLogFile( $id );
231 - // append stuff to logfile
232 - fopen( $file, "w+" );
233 - fwrite( date("c").$msg, $file );
234 - fclose( $file );
 226+ public function getRootDirs() {
 227+ return array(
 228+ "{$this->root}/dbs",
 229+ "{$this->root}/checkouts",
 230+ "{$this->root}/conf",
 231+ "{$this->root}/logs",
 232+ );
235233 }
236234
237 - /**
238 - * Utility function to output a dated message
239 - *
240 - * @param $msg String message to show. Will be prefixed with date("c")
241 - */
242 - function msg( $msg ) {
243 - print date("c") . " $msg\n";
 235+ public function prepareRootDirs() {
 236+ foreach( $this->getRootDirs() as $dir ) {
 237+ if ( !file_exists( $dir ) ) {
 238+ $this->mkdir( $dir );
 239+ }
 240+ }
244241 }
245242
246243 /**
247 - * Print a message to STDOUT when debug mode is enabled
248 - * Messages are prefixed with "DEBUG> "
249 - * Multi lines messages will be prefixed as well.
 244+ * This function is where most of the directory layout is kept.
 245+ * All other methods should use this array to determine where to look or save.
250246 *
251 - * @param $msg string Message to print
 247+ * @param $id integer: Revision number.
 248+ * @return Array of paths relevant for an install.
252249 */
253 - function debug( $msg ) {
254 - if( !$this->debugEnabled ) {
255 - return;
 250+ public function getPathsForRev( $id ) {
 251+ if ( !is_int( $id ) ) {
 252+ throw new Exception( __METHOD__ . ': Given non numerical revision' );
256253 }
257 - foreach( explode( "\n", $msg ) as $line ) {
258 - print "DEBUG> $line\n";
259 - }
 254+
 255+ return array(
 256+ 'db' => "{$this->root}/dbs",
 257+ 'mw' => "{$this->root}/checkouts/r{$id}",
 258+ 'globalsettings' => "{$this->root}/conf/GlobalSettings.php",
 259+ 'localsettingstpl' => "{$this->root}/conf/LocalSettings.tpl.php",
 260+ 'log' => "{$this->root}/logs/r{$id}.log",
 261+ );
260262 }
261263
262 - /** unused / unneeded? @author Antoine Musso */
263 - function changePath( $path ) {
264 - if( $this->savedPath !== null ) {
265 - throw new Exception( __METHOD__ . " called but a saved path exist '" . $this->savedPath ."' Did you forget to call restorePath()?\n");
266 - }
267 - $this->debug( "chdir( $path )" );
268264
269 - $oldPath = getcwd();
270 - if( chdir( $path ) ) {
271 - $this->savedPath = $oldPath;
272 - } else {
273 - throw new Exception( __METHOD__ . " failed to chdir() to $path\n" );
274 - }
275 - }
 265+ /** UTILITY FUNCTIONS **/
276266
277 - /** unused / unneeded? @author Antoine Musso */
278 - function restorePath() {
279 - if( $this->savedPath === null ) {
280 - return false;
281 - }
282 - chdir( $this->savedPath );
283 - }
284 -
285267 /**
286 - * Execute a command!
287 - * Ripped partially from wfExec()
288 - * throw an exception if anything goes wrong.
 268+ * Execute a shell command!
 269+ * Ripped partially from wfShellExec()
 270+ * Throws an exception if anything goes wrong.
289271 *
290 - * @param $cmd string Command which will be passed as is (no escaping FIXME)
291 - * @param &$retval reference Will be given the command exit level
292 - * @return Command output.
 272+ * @param $cmd string: Command which will be passed as is (no escaping FIXME)
 273+ * @param &$retval reference: Will be given the command exit level
 274+ * @return mixed: Command output.
293275 */
294 - function exec( $cmd, &$retval = 0 ) {
295 - $this->debug( __METHOD__ . " $cmd" );
 276+ public function exec( $cmd, &$retval = 0 ) {
 277+ $this->debug( "Executing '$cmd'", __METHOD__ );
296278
297279 // Pass command to shell and use ob to fetch the output
298280 ob_start();
——@@ -299,10 +281,12 @@
300282 $output = ob_get_contents();
301283 ob_end_clean();
302284
303 - if( $retval == 127 ) {
304 - throw new Exception( __METHOD__ . "probably missing executable. Check env.\n" );
 285+ if ( $retval == 127 ) {
 286+ throw new Exception( __METHOD__ . ': Probably missing executable. Check env.' );
305287 }
306288
 289+ $this->debug( "Done executing '$cmd'", __METHOD__ );
 290+
307291 return $output;
308292 }
309293
——@@ -311,152 +295,187 @@
312296 *
313297 * @param $path String Path to create ex: /tmp/my/foo/bar
314298 */
315 - function mkdir( $path ) {
316 - $this->debug( __METHOD__ . " mkdir( $path )" );
317 - if(!file_exists( $path ) ) {
318 - if( mkdir( $path, 0777, true ) ) {
319 - $this->debug( __METHOD__ . ": Created directory $path" );
 299+ public function mkdir( $path ) {
 300+ $this->debug( "Attempting to create directory '$path'...", __METHOD__ );
 301+ if ( !file_exists( $path ) ) {
 302+ if ( mkdir( $path, 0777, true ) ) {
 303+ $this->debug( "Created directory '$path'", __METHOD__ );
320304 } else {
321 - throw new Exception( __METHOD__ . " Error creating directory $path\n" );
 305+ throw new Exception( __METHOD__ . ": Failed to create directory '$path'" );
322306 }
323307 } else {
324 - $this->debug( __METHOD__ . " mkdir( $path ). Path already exist." );
 308+ $this->debug( "Creating directory '$path' aborted. Directory already exist", __METHOD__ );
325309 }
326310 }
327311
328 - /** unused / unneeded? @author Antoine Musso */
329 - function getLogFile( $id ) {
330 - return $logfile = $this->getPath( 'log', $id ). "/debug.log";
331 - }
332312
333 - /** UTILITY FUNCTIONS **/
 313+ /** LOGGING **/
334314
335315 /**
336 - * Get next changed revision for a given checkout
337 - * @return String|Boolean: false if nothing changed, else the upstream revision just after.
 316+ * Utility function to log a message for a given id.
 317+ *
 318+ * @param $msg string: message to log. Will be prefixed with date("c")
 319+ * @param $callee string: Callee function to be logged as origin.
338320 */
339 - function getNextRevisionId() {
340 - $cur = $this->getCurrentRevisionId();
341 - if( $cur === null ) {
342 - $this->debug( __METHOD__ . " checkouts dir empty? Looking up remote repo." );
343 - $next = $this->getFirstRevision();
344 - } else {
345 - $next = $this->getRevFollowing( $cur );
346 - }
 321+ public function log( $msg, $callee = '', $prefix = '' ) {
 322+ $msg = $prefix . ( $callee !== '' ? "$callee: " : '' ) . $msg;
 323+ $file = $this->getLogPath();
347324
348 - $this->debug( __METHOD__ . " returns $next" );
349 - return $next;
 325+ echo "$msg\n";
 326+
 327+ // Append to logfile
 328+ $fhandle = fopen( $file, "w+" );
 329+ fwrite( $fhandle, '[' . date( 'r' ) . '] ' . $msg );
 330+ fclose( $fhandle );
350331 }
351332
352 - function getFirstRevision() {
353 - $start = $this->minRevision - 1;
354 - $firstRevision = $this->getRevFollowing( $start );
355 - $this->debug( __METHOD__ . " using first revision '$firstRevision'" );
356 - return $firstRevision;
 333+ /**
 334+ * Log a debug message. Only logged in debug mode.
 335+ * Messages are prefixed with "DEBUG> ".
 336+ * Multiline messages will be split up.
 337+ *
 338+ * @param $msg string: Message to print.
 339+ * @param $callee string.
 340+ */
 341+ public function debug( $msg, $callee = '', $prefix = '' ) {
 342+ if ( !$this->debugMode ) {
 343+ return;
 344+ }
 345+ foreach( explode( "\n", $msg ) as $line ) {
 346+ $line = $prefix . ( $callee !== '' ? "$callee: " : '' ) . $line;
 347+ echo "DEBUG> $line\n";
 348+ }
357349 }
 350+}
358351
359 - function getRevFollowing( $id ) {
360 - $nextRev = $id + 1;
361 - // FIXME looking up for 1:HEAD takes a loooooooongg time
362 - $cmd = "{$this->svnCmd} log -q -r{$nextRev}:HEAD --limit 1 {$this->url}";
363 - $output = $this->exec( $cmd, $retval );
 352+class TestSwarmMWFetcher {
364353
365 - if( $retval !== 0 ) {
366 - throw new Exception(__METHOD__." error running subversion log.\n" );
 354+ /** Instance of TestSwarmMWMain */
 355+ private $main;
 356+
 357+ /** MediaWiki revision id being fetched */
 358+ protected $svnRevId;
 359+
 360+ /** Array as created by TestSwarmMWMain::getPathsForRev */
 361+ protected $paths;
 362+
 363+ public function __construct( TestSwarmMWMain $main, $svnRevId ) {
 364+ // Basic validation
 365+ if ( !is_int( $svnRevId ) ) {
 366+ throw new Exception( __METHOD__ . ": Invalid argument. svnRevId must be an integer" );
367367 }
368368
369 - preg_match( "/r(\d+)/m", $output, $m );
370 - $followingRev = (int) $m[1];
371 - if( $followingRev === 0 ) {
372 - throw new Exception( __METHOD__ . " remote gave us non integer revision: '{$m[1]}'\n" );
373 - }
374 - return $followingRev;
 369+ $this->paths = $main->getPathsForRev( $svnRevId );
 370+ $main->setLogPath( $this->paths['log'] );
 371+
 372+ $this->main = $main;
 373+ $this->svnRevId = $svnRevId;
375374 }
 375+
376376 /**
377 - * Get latest revision fetched in the working copy.
378 - * @return integer
 377+ * This is the main function doing checkout and installation for
 378+ * a given rev.
 379+ *
 380+ * @param $id integer: Revision id to install
 381+ * @return
379382 */
380 - function getCurrentRevisionId() {
381 - $checkoutsDir = dirname( $this->getPath( 'mw', 0 ) );
382 - // scandir sort in descending order if passing a nonzero value
383 - // PHP 5.4 accepts constant SCANDIR_SORT_DESCENDING
384 - $dirs = scandir( $checkoutsDir, 1 );
385 - $this->debug( "From '$checkoutsDir' Got directories:\n".implode("\n", $dirs ) );
386 - // On first run, we will have to take care of that.
387 - if ( $dirs[0][0] === 'r' ) {
388 - return substr( $dirs[0], 1 );
389 - } else {
390 - return null;
391 - }
 383+ public function run() {
 384+ $this->main->log( "Run for r{$this->svnRevId} started", __METHOD__ );
 385+
 386+ $this->doCheckout();
 387+ $this->doInstall();
 388+ $this->doAppendSettings();
 389+
 390+ /**
 391+ * @todo FIXME:
 392+ * - Get list of tests (see old file for how)
 393+ * - Make POST request to TestSwarm install to add jobs for these test runs
 394+ * (CURL addjob.php with login/auth token)
 395+ */
392396 }
393397
394398 /**
395 - * This function is where most of the directory layout is kept
396 - * All other methods should use getPath whenever they are looking for a path
 399+ * Checkout a given revision in our specific tree.
 400+ * Throw an exception if anything got wrong.
397401 *
398 - * @param $type string: Resource to fetch:
399 - * 'db': path to DB dir to put testwarm_mw_r123.sqlite file in
400 - * 'mw': path to MW dir
401 - * 'globalsettings': path to global settings file
402 - * 'localsettingstpl': path to LocalSettings.php template file
403 - * 'log': path to log file
404 - * @param $id integer: Revision number.
405 - * @return Full path to ressource or 'false'
 402+ * @todo Output is not logged.
406403 */
407 - function getPath( $type, $id ) {
408 - if ( !in_array( $type, array( 'globalsettings', 'localsettingstpl' ) )
409 - && !is_int( $id ) ) {
410 - throw new Exception( __METHOD__ . "given non numerical revision" );
 404+ public function doCheckout(){
 405+ $this->main->log( 'Checking out...', __METHOD__ );
 406+
 407+ // Create checkout directory for this revision
 408+ $this->main->mkdir( $this->paths['mw'] );
 409+
 410+ // @todo FIXME: We might want to log the output of svn commands
 411+ $cmd = "{$this->main->getSvnCmd()} checkout {$this->main->getSvnUrl()}@r{$this->svnRevId} {$this->paths['mw']}";
 412+
 413+ $retval = null;
 414+ $this->main->exec( $cmd, $retval );
 415+ if ( $retval !== 0 ) {
 416+ throw new Exception(__METHOD__ . ": Error running subversion checkout" );
411417 }
412418
413 - switch ( $type ) {
 419+ // @todo: Handle errors for above commands.
 420+ }
414421
415 - case 'db':
416 - return "{$this->root}/dbs/";
 422+ /**
 423+ * Install the fresly checked out MediaWiki version.
 424+ */
 425+ public function doInstall() {
 426+ $this->main->log( 'Installing...', __METHOD__ );
417427
418 - case 'mw':
419 - return "{$this->root}/checkouts/r{$id}";
 428+ // Erase MW_INSTALL_PATH which would interact with the install script
 429+ putenv( 'MW_INSTALL_PATH' );
420430
421 - case 'globalsettings':
422 - return "{$this->root}/conf/GlobalSettings.php";
 431+ // If admin access is needed, shell dev should run maintenance/changePassword.php,
 432+ // we don't need to know this password.
 433+ $randomAdminPass = substr( sha1( $this->svnRevId . serialize( $this->paths ) . rand( 100, 999 ) ), 0, 32 );
 434+ // For convenience, put it in debug (not in saved log)
 435+ $this->main->debug( "Generated wikiadmin pass: {$randomAdminPass}", __METHOD__ );
423436
424 - case 'localsettingstpl':
425 - return "{$this->root}/conf/LocalSettings.tpl.php";
 437+ // Now simply run the CLI installer:
 438+ $cmd = "php {$this->paths['mw']}/maintenance/install.php \
 439+ --dbname=testwarm_mw_r{$this->svnRevId} \
 440+ --dbtype=sqlite \
 441+ --dbpath={$this->paths['db']} \
 442+ --showexceptions=true \
 443+ --confpath={$this->paths['mw']} \
 444+ --pass={$randomAdminPass} \
 445+ TrunkWikiR{$this->svnRevId} \
 446+ WikiSysop
 447+ ";
426448
427 - case 'log':
428 - return "{$this->root}/log/r{$id}";
 449+ $retval = null;
 450+ $output = $this->main->exec( $cmd, $retval );
429451
430 - default:
431 - return false;
 452+ $this->main->log( "-- MediaWiki installer output: \n$output\n-- End of MediaWiki installer output", __METHOD__ );
 453+
 454+ if ( $retval !== 0 ) {
 455+ throw new Exception(__METHOD__ . ": Error running MediaWiki installer" );
432456 }
433457 }
434 -}
435458
436 -# Remaning notes from design session. Leave them for now unless you are Timo.
437 -/** GENERAL:
438 -format: php
439 -Directory structure:
440 -dbs/
441 -testswarm_mw_r123.sqlite
442 -conf/
443 -GlobalSettings.php
444 -// global conf, empty in most cases. Could be used to globally do something important
445 -LocalSettingsTemplate.php
446 -// copied/appended to LocalSettings.php that install.php makes
447 -checkouts/
448 -// publicly available through Apache; symlinked to /var/www/testswarm-mw
449 -r123/
450 - */
 459+ /**
 460+ * @todo FIXME: Implement :-)
 461+ * @param $id integer: Revision id to append settings to.
 462+ */
 463+ public function doAppendSettings() {
 464+ $this->log( 'Appending settings... *TODO!*', __METHOD__ );
 465+ return true;
451466
452 -/** INIT:
453 -get latest svn revision number for trunk/phase3
454 -you want the next changed revision. Not the latest.
455 -svn log -r BASE:HEAD -l 2 -q
456 -I didn't even know that was an option, even better (no risk of missing a rev if > 1 commit between runs). Awesome!
457 -If BASE is at HEAD, you will only get one line though. So need to verify!
458 -Getting the latest checked out directory is all about ls -1 | tail -1
459 -check: already checked out ? Abort otherwise
460 -(file_exists(checkouts/r...)
461 -svn checkout (or export) into the checkouts/r..
462 - */
 467+ // append to mwPath/LocalSettings.php
 468+ // -- contents of LocalSettings.tpl.php
 469+ // -- require_once( '{$this->getPath('globalsettings')}' );"
463470
 471+ /**
 472+ * Possible additional common settings to append to LocalSettings after install:
 473+ * See gerrit integration/jenkins.git:
 474+ * https://gerrit.wikimedia.org/r/gitweb?p=integration/jenkins.git;a=tree;f=jobs/MediaWiki-phpunit;hb=HEAD
 475+ *
 476+ * $wgShowExceptionDetails = true;
 477+ * $wgShowSQLErrors = true;
 478+ * #$wgDebugLogFile = dirname( __FILE__ ) . '/build/debug.log';
 479+ * $wgDebugDumpSql = true;
 480+ */
 481+ }
 482+}
\ No newline at end of file

Follow-up revisions

RevisionCommit summaryAuthorDate
r101745[JSTesting] fix require_once path for testswarm-mw-fetcher.php...krinkle23:21, 2 November 2011
r101750[JSTesting] fix function documentation...krinkle23:29, 2 November 2011
r101754[JSTesting] TestSwarmMWFetcher...krinkle00:04, 3 November 2011

Past revisions this follows-up on

RevisionCommit summaryAuthorDate
r101591export specified range of revisions (as stubs)ariel07:58, 2 November 2011
r101636MW fetcher and installer for TestSwarm...hashar17:13, 2 November 2011

Comments

#Comment by Hashar (talk | contribs)   18:49, 3 November 2011

TIP TIP TIP for small projects use git as a front end to svn :-)

Example: 

git svn clone <somerepo> <localpath>
cd <localpath>
... Do a ton of changes like above ...
git add -p  # pick your hunks
git commit  # with a nice message
git add -p  # pick your hunks
git commit  # with a nice message
...
git svn dcommit

Et voila. This way you will be able to submit your messy working copy in separate commits :-D

#Comment by Hashar (talk | contribs)   19:08, 3 November 2011

I have it roughly reviewed. Thanks for splitting the classes!

I am marking this revision as deferred since there will be surely have a lot of bugs still. Anyway it is good enough for our first tests.

Now need to package it up in the testswarm package. I will followup with demon on this.

Status & tagging log