r81536 MediaWiki - Code Review archive

Repository:MediaWiki
Revision:r81535‎ | r81536 | r81537 >
Date:20:54, 4 February 2011
Author:btongminh
Status:resolved (Comments)
Tags:
Comment:
(bug 19751) Filesystem is now checked during image undeletion
* FSRepo::storeBatch() now does an sha1 check unless SKIP_VALIDATION flag is set
* Introduced Status::$success in addition to Status::$successcount
** FSRepo::storeBatch() now logs success/failure in this variable
* LocalFileRestoreBatch now aborts on failure in FSRepo::storeBatch() and cleans up the already copied files
** Introduced FSRepo::cleanupBatch() for this purpose
* SpecialUndelete now aborts if LocalFile::restore() gives a fatal
Modified paths:
  • /trunk/phase3/RELEASE-NOTES (modified) (history)
  • /trunk/phase3/includes/Status.php (modified) (history)
  • /trunk/phase3/includes/filerepo/FSRepo.php (modified) (history)
  • /trunk/phase3/includes/filerepo/FileRepo.php (modified) (history)
  • /trunk/phase3/includes/filerepo/LocalFile.php (modified) (history)
  • /trunk/phase3/includes/specials/SpecialUndelete.php (modified) (history)

Diff [purge]

Index: trunk/phase3/includes/Status.php
@@ -17,7 +17,9 @@
1818 var $value;
1919
2020 /** Counters for batch operations */
21 - var $successCount = 0, $failCount = 0;
 21+ public $successCount = 0, $failCount = 0;
 22+ /** Array to indicate which items of the batch operations failed */
 23+ public $success = array();
2224
2325 /*semi-private*/ var $errors = array();
2426 /*semi-private*/ var $cleanCallback = false;
Index: trunk/phase3/includes/filerepo/LocalFile.php
@@ -1197,7 +1197,7 @@
11981198
11991199 $status = $batch->execute();
12001200
1201 - if ( !$status->ok ) {
 1201+ if ( !$status->isGood() ) {
12021202 return $status;
12031203 }
12041204
@@ -1807,9 +1807,11 @@
18081808 $storeStatus = $this->file->repo->storeBatch( $storeBatch, FileRepo::OVERWRITE_SAME );
18091809 $status->merge( $storeStatus );
18101810
1811 - if ( !$status->ok ) {
1812 - // Store batch returned a critical error -- this usually means nothing was stored
1813 - // Stop now and return an error
 1811+ if ( !$status->isGood() ) {
 1812+ // Even if some files could be copied, fail entirely as that is the
 1813+ // easiest thing to do without data loss
 1814+ $this->cleanupFailedBatch( $storeStatus, $storeBatch );
 1815+ $status->ok = false;
18141816 $this->file->unlock();
18151817
18161818 return $status;
@@ -1914,6 +1916,17 @@
19151917
19161918 return $status;
19171919 }
 1920+
 1921+ function cleanupFailedBatch( $storeStatus, $storeBatch ) {
 1922+ $cleanupBatch = array();
 1923+
 1924+ foreach ( $storeStatus->success as $i => $success ) {
 1925+ if ( $success ) {
 1926+ $cleanupBatch[] = array( $storeBatch[$i][1], $storeBatch[$i][1] );
 1927+ }
 1928+ }
 1929+ $this->file->repo->cleanupBatch( $cleanupBatch );
 1930+ }
19181931 }
19191932
19201933 # ------------------------------------------------------------------------------
Index: trunk/phase3/includes/filerepo/FSRepo.php
@@ -146,16 +146,23 @@
147147 * same contents as the source
148148 */
149149 function storeBatch( $triplets, $flags = 0 ) {
 150+ wfDebug( __METHOD__ . ': Storing ' . count( $triplets ) .
 151+ " triplets; flags: {$flags}\n" );
 152+
 153+ // Try creating directories
150154 if ( !wfMkdirParents( $this->directory ) ) {
151155 return $this->newFatal( 'upload_directory_missing', $this->directory );
152156 }
153157 if ( !is_writable( $this->directory ) ) {
154158 return $this->newFatal( 'upload_directory_read_only', $this->directory );
155159 }
 160+
 161+ // Validate each triplet
156162 $status = $this->newGood();
157163 foreach ( $triplets as $i => $triplet ) {
158164 list( $srcPath, $dstZone, $dstRel ) = $triplet;
159165
 166+ // Resolve destination path
160167 $root = $this->getZonePath( $dstZone );
161168 if ( !$root ) {
162169 throw new MWException( "Invalid zone: $dstZone" );
@@ -166,6 +173,7 @@
167174 $dstPath = "$root/$dstRel";
168175 $dstDir = dirname( $dstPath );
169176
 177+ // Create destination directories for this triplet
170178 if ( !is_dir( $dstDir ) ) {
171179 if ( !wfMkdirParents( $dstDir ) ) {
172180 return $this->newFatal( 'directorycreateerror', $dstDir );
@@ -175,6 +183,7 @@
176184 }
177185 }
178186
 187+ // Resolve source
179188 if ( self::isVirtualUrl( $srcPath ) ) {
180189 $srcPath = $triplets[$i][0] = $this->resolveVirtualUrl( $srcPath );
181190 }
@@ -183,6 +192,8 @@
184193 $status->fatal( 'filenotfound', $srcPath );
185194 continue;
186195 }
 196+
 197+ // Check overwriting
187198 if ( !( $flags & self::OVERWRITE ) && file_exists( $dstPath ) ) {
188199 if ( $flags & self::OVERWRITE_SAME ) {
189200 $hashSource = sha1_file( $srcPath );
@@ -196,6 +207,7 @@
197208 }
198209 }
199210
 211+ // Windows does not support moving over existing files, so explicitly delete them
200212 $deleteDest = wfIsWindows() && ( $flags & self::OVERWRITE );
201213
202214 // Abort now on failure
@@ -203,7 +215,8 @@
204216 return $status;
205217 }
206218
207 - foreach ( $triplets as $triplet ) {
 219+ // Execute the store operation for each triplet
 220+ foreach ( $triplets as $i => $triplet ) {
208221 list( $srcPath, $dstZone, $dstRel ) = $triplet;
209222 $root = $this->getZonePath( $dstZone );
210223 $dstPath = "$root/$dstRel";
@@ -222,6 +235,20 @@
223236 $status->error( 'filecopyerror', $srcPath, $dstPath );
224237 $good = false;
225238 }
 239+ if ( !( $flags & self::SKIP_VALIDATION ) ) {
 240+ wfSuppressWarnings();
 241+ $hashSource = sha1_file( $srcPath );
 242+ $hashDest = sha1_file( $dstPath );
 243+ wfRestoreWarnings();
 244+
 245+ if ( $hashDest === false || $hashSource !== $hashDest ) {
 246+ wfDebug( __METHOD__ . ': File copy validation failed: ' .
 247+ "$srcPath ($hashSource) to $dstPath ($hashDest)\n" );
 248+
 249+ $status->error( 'filecopyerror', $srcPath, $dstPath );
 250+ $good = false;
 251+ }
 252+ }
226253 }
227254 if ( $good ) {
228255 $this->chmod( $dstPath );
@@ -229,9 +256,28 @@
230257 } else {
231258 $status->failCount++;
232259 }
 260+ $status->success[$i] = $good;
233261 }
234262 return $status;
235263 }
 264+
 265+ /**
 266+ * Deletes a batch of (zone, rel) pairs. It will try to delete each pair,
 267+ * but ignores any errors doing so.
 268+ *
 269+ * @param $pairs array Pair of (zone, rel) pairs to delete
 270+ */
 271+ function cleanupBatch( $pairs ) {
 272+ foreach ( $pairs as $i => $pair ) {
 273+ list( $zone, $rel ) = $pair;
 274+ $root = $this->getZonePath( $zone );
 275+ $path = "$root/$rel";
 276+
 277+ wfSuppressWarnings();
 278+ unlink( $path );
 279+ wfRestoreWarnings();
 280+ }
 281+ }
236282
237283 function append( $srcPath, $toAppendPath, $flags = 0 ) {
238284 $status = $this->newGood();
Index: trunk/phase3/includes/filerepo/FileRepo.php
@@ -17,6 +17,7 @@
1818 const DELETE_SOURCE = 1;
1919 const OVERWRITE = 2;
2020 const OVERWRITE_SAME = 4;
 21+ const SKIP_VALIDATION = 8;
2122
2223 var $thumbScriptUrl, $transformVia404;
2324 var $descBaseUrl, $scriptDirUrl, $scriptExtension, $articleUrl;
Index: trunk/phase3/includes/specials/SpecialUndelete.php
@@ -336,6 +336,9 @@
337337 if( $restoreFiles && $this->title->getNamespace() == NS_FILE ) {
338338 $img = wfLocalFile( $this->title );
339339 $this->fileStatus = $img->restore( $fileVersions, $unsuppress );
 340+ if ( !$this->fileStatus->isOk() ) {
 341+ return false;
 342+ }
340343 $filesRestored = $this->fileStatus->successCount;
341344 } else {
342345 $filesRestored = 0;
Index: trunk/phase3/RELEASE-NOTES
@@ -112,6 +112,7 @@
113113 * (bug 26809) Uploading files with multiple extensions where one of the extensions
114114 is blacklisted now gives the proper extension in the error message.
115115 * (bug 26961) Hide anon edits in watchlist preference now actually works.
 116+* (bug 19751) Filesystem is now checked during image undeletion
116117
117118 === API changes in 1.18 ===
118119 * (bug 26339) Throw warning when truncating an overlarge API result

Follow-up revisions

RevisionCommit summaryAuthorDate
r88911Follow-up r81536: Fix misleading comment in Status and add a comment to Local...btongminh18:41, 26 May 2011
r91188Follow-up r81536: Properly extract ($dstZone, $dstRel) from the triplet.btongminh17:43, 30 June 2011

Comments

#Comment by Platonides (talk | contribs)   09:43, 14 May 2011

An array called $success to indicate which items of the batch operations failed?

Also, you're storing it whether it's actually good or not.

#Comment by Bryan (talk | contribs)   10:28, 14 May 2011

Well, something is wrong here.

#Comment by Bryan (talk | contribs)   18:41, 26 May 2011

The issue was actually a misleading comment, fixed in r88911.

#Comment by Hashar (talk | contribs)   16:38, 6 June 2011

tagging for 1.17 inclusion since it fixes a possible data lost

#Comment by RobLa-WMF (talk | contribs)   23:41, 7 June 2011

This is a fix for a two year old bug, and doesn't appear to be running in production. Doesn't seem like a 1.17 tarball blocker to me.

#Comment by Catrope (talk | contribs)   11:39, 8 June 2011

Alright, untagging 1.17 then. If this bug was present in 1.16 as well, we can afford to delay the fix till 1.18.

Besides, per my comment below I'm fairly sure this fix doesn't even work.

#Comment by Catrope (talk | contribs)   16:09, 7 June 2011
+				$cleanupBatch[] = array( $storeBatch[$i][1], $storeBatch[$i][1] );

This looks wrong to me. Won't this end up as something like array( array( 'public', 'public' ), array( 'public', 'public' ), ... ) ? Shouldn't the second element be $storeBatch[$i][0] instead?

#Comment by Bryan (talk | contribs)   17:43, 30 June 2011

Thanks for the review. Fixed in r91188.

Status & tagging log