r60773 MediaWiki - Code Review archive

Repository:MediaWiki
Revision:r60772‎ | r60773 | r60774 >
Date:07:15, 7 January 2010
Author:robla
Status:deferred (Comments)
Tags:todo 
Comment:
Decouple page titles from MIME type, adding img_file_ext field to db (bug 4421
Modified paths:
  • /branches/extensionless-files/includes/DefaultSettings.php (modified) (history)
  • /branches/extensionless-files/includes/Title.php (modified) (history)
  • /branches/extensionless-files/includes/filerepo/ArchivedFile.php (modified) (history)
  • /branches/extensionless-files/includes/filerepo/FSRepo.php (modified) (history)
  • /branches/extensionless-files/includes/filerepo/File.php (modified) (history)
  • /branches/extensionless-files/includes/filerepo/FileRepo.php (modified) (history)
  • /branches/extensionless-files/includes/filerepo/LocalFile.php (modified) (history)
  • /branches/extensionless-files/includes/filerepo/OldLocalFile.php (modified) (history)
  • /branches/extensionless-files/includes/upload/UploadBase.php (modified) (history)
  • /branches/extensionless-files/languages/messages/MessagesEn.php (modified) (history)
  • /branches/extensionless-files/maintenance/tables.sql (modified) (history)
  • /branches/extensionless-files/maintenance/updaters.inc (modified) (history)

Diff [purge]

Index: branches/extensionless-files/maintenance/updaters.inc
@@ -169,6 +169,7 @@
170170 // A field changed name mid-release cycle, so fix it for anyone using
171171 // trunk
172172 array( 'rename_eu_wiki_id' ),
 173+ array( 'add_field', 'image', 'img_file_ext', 'patch-img_file_ext.sql'),
173174 ),
174175
175176 'sqlite' => array(
@@ -194,6 +195,7 @@
195196 array( 'add_index', 'change_tag', 'change_tag_rc_tag', 'patch-change_tag-indexes.sql' ),
196197 array( 'add_field', 'redirect', 'rd_interwiki', 'patch-rd_interwiki.sql' ),
197198 array( 'do_update_transcache_field' ),
 199+ array( 'add_field', 'image', 'img_file_ext', 'patch-img_file_ext.sql'),
198200
199201 // version-independent searchindex setup, added in 1.16
200202 array( 'sqlite_setup_searchindex' ),
Index: branches/extensionless-files/maintenance/tables.sql
@@ -778,7 +778,12 @@
779779 img_timestamp varbinary(14) NOT NULL default '',
780780
781781 -- SHA-1 content hash in base-36
782 - img_sha1 varbinary(32) NOT NULL default ''
 782+ img_sha1 varbinary(32) NOT NULL default '',
 783+
 784+ -- File extension, appended to the on-disk file in cases where the
 785+ -- extension derived from img_name doesn't match the media type of
 786+ -- the file. See bug #4421.
 787+ img_file_ext varchar(32) binary NOT NULL default ''
783788 ) /*$wgDBTableOptions*/;
784789
785790 CREATE INDEX /*i*/img_usertext_timestamp ON /*_*/image (img_user_text,img_timestamp);
@@ -819,6 +824,7 @@
820825 oi_minor_mime varbinary(32) NOT NULL default "unknown",
821826 oi_deleted tinyint unsigned NOT NULL default 0,
822827 oi_sha1 varbinary(32) NOT NULL default ''
 828+ oi_file_ext varchar(32) binary NOT NULL default ''
823829 ) /*$wgDBTableOptions*/;
824830
825831 CREATE INDEX /*i*/oi_usertext_timestamp ON /*_*/oldimage (oi_user_text,oi_timestamp);
@@ -871,6 +877,7 @@
872878 fa_user int unsigned default 0,
873879 fa_user_text varchar(255) binary,
874880 fa_timestamp binary(14) default '',
 881+ fa_file_ext varchar(32) binary NOT NULL default ''
875882
876883 -- Visibility of deleted revisions, bitfield
877884 fa_deleted tinyint unsigned NOT NULL default 0
Index: branches/extensionless-files/includes/upload/UploadBase.php
@@ -311,8 +311,8 @@
312312 $warnings['badfilename'] = $filename;
313313
314314 // Check whether the file extension is on the unwanted list
315 - global $wgCheckFileExtensions, $wgFileExtensions;
316 - if ( $wgCheckFileExtensions ) {
 315+ global $wgCheckTitleFileExtensions, $wgFileExtensions;
 316+ if ( $wgCheckTitleFileExtensions ) {
317317 if ( !$this->checkFileExtension( $this->mFinalExtension, $wgFileExtensions ) )
318318 $warnings['filetype-unwanted-type'] = $this->mFinalExtension;
319319 }
@@ -408,16 +408,18 @@
409409 }
410410
411411 /* Don't allow users to override the blacklist (check file extension) */
412 - global $wgCheckFileExtensions, $wgStrictFileExtensions;
 412+ global $wgCheckTitleFileExtensions, $wgStrictFileExtensions;
413413 global $wgFileExtensions, $wgFileBlacklist;
414 - if ( $this->mFinalExtension == '' ) {
415 - $this->mTitleError = self::FILETYPE_MISSING;
416 - return $this->mTitle = null;
417 - } elseif ( $this->checkFileExtensionList( $ext, $wgFileBlacklist ) ||
418 - ( $wgCheckFileExtensions && $wgStrictFileExtensions &&
419 - !$this->checkFileExtension( $this->mFinalExtension, $wgFileExtensions ) ) ) {
420 - $this->mTitleError = self::FILETYPE_BADTYPE;
421 - return $this->mTitle = null;
 414+ if ( $wgCheckTitleFileExtensions ) {
 415+ if ( $this->mFinalExtension == '' ) {
 416+ $this->mTitleError = self::FILETYPE_MISSING;
 417+ return $this->mTitle = null;
 418+ } elseif ( $this->checkFileExtensionList( $ext, $wgFileBlacklist ) ||
 419+ ( $wgStrictFileExtensions &&
 420+ !$this->checkFileExtension( $this->mFinalExtension, $wgFileExtensions ) ) ) {
 421+ $this->mTitleError = self::FILETYPE_BADTYPE;
 422+ return $this->mTitle = null;
 423+ }
422424 }
423425
424426 # If there was more than one "extension", reassemble the base
Index: branches/extensionless-files/includes/filerepo/OldLocalFile.php
@@ -97,10 +97,18 @@
9898
9999 /**
100100 * This function tacks on file extension to archive_name, if needed.
101 - * Stub function pending full implementation of bug 4421.
102101 */
103102 public function getArchiveFilename() {
104 - return $this->getArchiveName();
 103+ if ( !isset( $this->archive_filename ) ) {
 104+ $archiveName = $this->getArchiveName();
 105+ if ( $archiveName == '') {
 106+ throw new MWException( "Blank return value from getArchiveName in ".__METHOD__ );
 107+ }
 108+ $this->archive_filename = $this->getArchiveName()
 109+ . $this->getAddedFileExt();
 110+ wfDebug(__METHOD__.": archive_name: {$this->archive_name} archive_filename: {$this->archive_filename}\n");
 111+ }
 112+ return $this->archive_filename;
105113 }
106114
107115 function isOld() {
Index: branches/extensionless-files/includes/filerepo/LocalFile.php
@@ -50,7 +50,8 @@
5151 $upgraded, # Whether the row was upgraded on load
5252 $locked, # True if the image row is locked
5353 $missing, # True if file is not present in file system. Not to be cached in memcached
54 - $deleted; # Bitfield akin to rev_deleted
 54+ $deleted, # Bitfield akin to rev_deleted
 55+ $file_ext; # File extension
5556
5657 /**#@-*/
5758
@@ -114,6 +115,7 @@
115116 'img_user_text',
116117 'img_timestamp',
117118 'img_sha1',
 119+ 'img_file_ext',
118120 );
119121 }
120122
@@ -205,7 +207,8 @@
206208
207209 function getCacheFields( $prefix = 'img_' ) {
208210 static $fields = array( 'size', 'width', 'height', 'bits', 'media_type',
209 - 'major_mime', 'minor_mime', 'metadata', 'timestamp', 'sha1', 'user', 'user_text', 'description' );
 211+ 'major_mime', 'minor_mime', 'metadata', 'timestamp', 'sha1',
 212+ 'user', 'user_text', 'description', 'file_ext' );
210213 static $results = array();
211214 if ( $prefix == '' ) {
212215 return $fields;
@@ -497,6 +500,21 @@
498501 return $this->media_type;
499502 }
500503
 504+ /**
 505+ * Return the additional file extension as needed to make the mime type
 506+ * and filename match. Pull from DB if it's loaded from there; otherwise
 507+ * generate it. Includes the leading dot.
 508+ */
 509+ function getAddedFileExt() {
 510+ $this->load();
 511+ if( isset($this->file_ext) ) {
 512+ return $this->file_ext;
 513+ }
 514+ else {
 515+ return parent::getAddedFileExt();
 516+ }
 517+ }
 518+
501519 /** canRender inherited */
502520 /** mustRender inherited */
503521 /** allowInlineDisplay inherited */
@@ -761,6 +779,7 @@
762780 */
763781 function upload( $srcPath, $comment, $pageText, $flags = 0, $props = false, $timestamp = false, $user = null ) {
764782 $this->lock();
 783+
765784 $status = $this->publish( $srcPath, $flags );
766785 if ( $status->ok ) {
767786 if ( !$this->recordUpload2( $status->value, $comment, $pageText, $props, $timestamp, $user ) ) {
@@ -846,7 +865,8 @@
847866 'img_user' => $user->getId(),
848867 'img_user_text' => $user->getName(),
849868 'img_metadata' => $this->metadata,
850 - 'img_sha1' => $this->sha1
 869+ 'img_sha1' => $this->sha1,
 870+ 'img_file_ext' => $this->getAddedFileExt()
851871 ),
852872 __METHOD__,
853873 'IGNORE'
@@ -873,7 +893,8 @@
874894 'oi_media_type' => 'img_media_type',
875895 'oi_major_mime' => 'img_major_mime',
876896 'oi_minor_mime' => 'img_minor_mime',
877 - 'oi_sha1' => 'img_sha1'
 897+ 'oi_sha1' => 'img_sha1',
 898+ 'oi_file_ext' => 'img_file_ext'
878899 ), array( 'img_name' => $this->getName() ), __METHOD__
879900 );
880901
@@ -892,7 +913,8 @@
893914 'img_user' => $user->getId(),
894915 'img_user_text' => $user->getName(),
895916 'img_metadata' => $this->metadata,
896 - 'img_sha1' => $this->sha1
 917+ 'img_sha1' => $this->sha1,
 918+ 'img_file_ext' => $this->getAddedFileExt()
897919 ), array( /* WHERE */
898920 'img_name' => $this->getName()
899921 ), __METHOD__
@@ -970,11 +992,43 @@
971993 */
972994 function publish( $srcPath, $flags = 0 ) {
973995 $this->lock();
 996+
 997+ if( $this->fileExists ) {
 998+ // First construct name for $archiveRel, using the cached filename
 999+ $this->setProps( $this->repo->getFileProps( $this->getVirtualUrl() ) );
 1000+
 1001+ $archiveDate = gmdate( 'YmdHis' );
 1002+ $archiveName = $archiveDate . '!'. $this->getName();
 1003+ $archiveFilename = $archiveDate . '!'. $this->getFilename();
 1004+ $archiveRel = 'archive/' . $this->getHashPath() . $archiveFilename;
 1005+
 1006+ $currentRel = $this->getRel();
 1007+ }
 1008+ else {
 1009+ $archiveRel = null;
 1010+ $currentRel = null;
 1011+ }
 1012+
 1013+ // Now regenerate the filename with the new MIME type.
 1014+ // First, clear cached file_ext, filename, and extension
 1015+ unset($this->file_ext);
 1016+ unset($this->filename);
 1017+ unset($this->extension);
 1018+
 1019+ // Set the props (MIME type) for this object to match srcPath
 1020+ if ( $this->repo->isVirtualUrl ( $srcPath ) ) {
 1021+ $props = $this->repo->getFileProps ( $srcPath );
 1022+ }
 1023+ else {
 1024+ $props = File::getPropsFromPath( $srcPath );
 1025+ }
 1026+ $this->setProps( $props );
 1027+ // generate new dstRel with extension to match new MIME type
9741028 $dstRel = $this->getRel();
975 - $archiveName = gmdate( 'YmdHis' ) . '!'. $this->getName();
976 - $archiveRel = 'archive/' . $this->getHashPath() . $archiveName;
 1029+
9771030 $flags = $flags & File::DELETE_SOURCE ? LocalRepo::DELETE_SOURCE : 0;
978 - $status = $this->repo->publish( $srcPath, $dstRel, $archiveRel, $flags );
 1031+ wfDebugLog("upload", __METHOD__.": srcPath: {$srcPath} dstRel: {$dstRel} currentRel: {$currentRel} archiveRel: {$archiveRel} flags: {$flags}");
 1032+ $status = $this->repo->publish( $srcPath, $dstRel, $currentRel, $archiveRel, $flags );
9791033 if ( $status->value == 'new' ) {
9801034 $status->value = '';
9811035 } else {
@@ -1235,7 +1289,10 @@
12361290 }
12371291
12381292 function addOld( $oldName ) {
1239 - $this->srcRels[$oldName] = $this->file->getArchiveRel( $oldName );
 1293+ $oldFile = OldLocalFile::newFromArchiveName( $this->file->title,
 1294+ $this->file->repo,
 1295+ $oldName );
 1296+ $this->srcRels[$oldName] = $this->file->getArchiveRel( $oldFile->getArchiveFilename() );
12401297 $this->archiveUrls[] = $this->file->getArchiveUrl( $oldName );
12411298 }
12421299
@@ -1345,7 +1402,8 @@
13461403 'fa_description' => 'img_description',
13471404 'fa_user' => 'img_user',
13481405 'fa_user_text' => 'img_user_text',
1349 - 'fa_timestamp' => 'img_timestamp'
 1406+ 'fa_timestamp' => 'img_timestamp',
 1407+ 'fa_file_ext' => 'img_file_ext'
13501408 ), $where, __METHOD__ );
13511409 }
13521410
@@ -1377,7 +1435,8 @@
13781436 'fa_user' => 'oi_user',
13791437 'fa_user_text' => 'oi_user_text',
13801438 'fa_timestamp' => 'oi_timestamp',
1381 - 'fa_deleted' => $bitfield
 1439+ 'fa_deleted' => $bitfield,
 1440+ 'fa_file_ext' => 'oi_file_ext'
13821441 ), $where, __METHOD__ );
13831442 }
13841443 }
@@ -1494,6 +1553,9 @@
14951554 foreach( $batch as $batchItem )
14961555 if( $result[$batchItem[0]] )
14971556 $newBatch[] = $batchItem;
 1557+ else
 1558+ wfDebugLog( 'filedelete',__METHOD__.": Skipping ".$batchItem[0] );
 1559+
14981560 return $newBatch;
14991561 }
15001562 }
@@ -1609,12 +1671,14 @@
16101672 'minor_mime' => $row->fa_minor_mime,
16111673 'major_mime' => $row->fa_major_mime,
16121674 'media_type' => $row->fa_media_type,
1613 - 'metadata' => $row->fa_metadata
 1675+ 'metadata' => $row->fa_metadata,
 1676+ 'file_ext' => $row->fa_file_ext
16141677 );
16151678 }
16161679
16171680 if ( $first && !$exists ) {
16181681 // This revision will be published as the new current version
 1682+ $this->file->setProps( $props );
16191683 $destRel = $this->file->getRel();
16201684 $insertCurrent = array(
16211685 'img_name' => $row->fa_name,
@@ -1630,7 +1694,8 @@
16311695 'img_user' => $row->fa_user,
16321696 'img_user_text' => $row->fa_user_text,
16331697 'img_timestamp' => $row->fa_timestamp,
1634 - 'img_sha1' => $sha1
 1698+ 'img_sha1' => $sha1,
 1699+ 'img_file_ext' => $props['file_ext']
16351700 );
16361701 // The live (current) version cannot be hidden!
16371702 if( !$this->unsuppress && $row->fa_deleted ) {
@@ -1650,7 +1715,8 @@
16511716 } while ( isset( $archiveNames[$archiveName] ) );
16521717 }
16531718 $archiveNames[$archiveName] = true;
1654 - $destRel = $this->file->getArchiveRel( $archiveName );
 1719+ $destRel = $this->file->getArchiveRel( $archiveName.$props['file_ext'] );
 1720+ wfDebugLog( 'fileundelete', __METHOD__.": Restoring archive destRel: {$destRel}" );
16551721 $insertBatch[] = array(
16561722 'oi_name' => $row->fa_name,
16571723 'oi_archive_name' => $archiveName,
@@ -1667,7 +1733,8 @@
16681734 'oi_major_mime' => $props['major_mime'],
16691735 'oi_minor_mime' => $props['minor_mime'],
16701736 'oi_deleted' => $this->unsuppress ? 0 : $row->fa_deleted,
1671 - 'oi_sha1' => $sha1 );
 1737+ 'oi_sha1' => $sha1,
 1738+ 'oi_file_ext' => $props['file_ext'] );
16721739 }
16731740
16741741 $deleteIds[] = $row->fa_id;
@@ -1793,7 +1860,7 @@
17941861 * @ingroup FileRepo
17951862 */
17961863 class LocalFileMoveBatch {
1797 - var $file, $cur, $olds, $oldCount, $archive, $target, $db;
 1864+ var $file, $cur, $olds, $oldCount, $archive, $target, $db, $mime;
17981865
17991866 function __construct( File $file, Title $target ) {
18001867 $this->file = $file;
@@ -1801,9 +1868,16 @@
18021869 $this->oldHash = $this->file->repo->getHashPath( $this->file->getName() );
18031870 $this->newHash = $this->file->repo->getHashPath( $this->target->getDBkey() );
18041871 $this->oldName = $this->file->getName();
 1872+ $this->oldFilename = $this->file->getFilename();
18051873 $this->newName = $this->file->repo->getNameFromTitle( $this->target );
1806 - $this->oldRel = $this->oldHash . $this->oldName;
1807 - $this->newRel = $this->newHash . $this->newName;
 1874+ $this->oldRel = $this->oldHash . $this->file->getFilename();
 1875+ $this->mime = $this->file->getMimeType();
 1876+ $this->newRel = $this->newHash . $this->file->repo->getFilenameFromTitle( $this->target,
 1877+ $this->mime );
 1878+ $this->oldExt = $this->file->getAddedFileExt();
 1879+ $this->newExt = $this->file->repo->getAddedExtensionFromTitle( $this->target,
 1880+ $this->mime );
 1881+
18081882 $this->db = $file->repo->getMasterDb();
18091883 }
18101884
@@ -1823,20 +1897,30 @@
18241898 $this->oldCount = 0;
18251899
18261900 $result = $this->db->select( 'oldimage',
1827 - array( 'oi_archive_name', 'oi_deleted' ),
 1901+ array( 'oi_archive_name', 'oi_deleted',
 1902+ 'oi_major_mime', 'oi_minor_mime' ),
18281903 array( 'oi_name' => $this->oldName ),
18291904 __METHOD__
18301905 );
 1906+ $mimeMagic = MimeMagic::singleton();
18311907 while( $row = $this->db->fetchObject( $result ) ) {
 1908+ $mime = $row->oi_major_mime . "/" . $row->oi_minor_mime;
 1909+
18321910 $oldName = $row->oi_archive_name;
 1911+ $extension = File::getNormalizedExtensionFromName( $oldName );
 1912+ if ( !$mimeMagic->isMatchingExtension( $extension, $mime ) ) {
 1913+ $oldName .= "." .
 1914+ $mimeMagic->getPreferredExtensionForType( $mime );
 1915+ }
 1916+
18331917 $bits = explode( '!', $oldName, 2 );
18341918 if( count( $bits ) != 2 ) {
18351919 wfDebug( "Invalid old file name: $oldName \n" );
18361920 continue;
18371921 }
18381922 list( $timestamp, $filename ) = $bits;
1839 - if( $this->oldName != $filename ) {
1840 - wfDebug( "Invalid old file name: $oldName \n" );
 1923+ if( $this->oldFilename != $filename ) {
 1924+ wfDebug( "Mismatched file name: $oldName (vs. {$this->oldFilename})\n" );
18411925 continue;
18421926 }
18431927 $this->oldCount++;
@@ -1844,9 +1928,16 @@
18451929 if( $row->oi_deleted & File::DELETED_FILE ) {
18461930 continue;
18471931 }
 1932+
 1933+ $newNameFixed = $this->newName;
 1934+ $extension = File::getNormalizedExtensionFromName( $newNameFixed );
 1935+ if ( !$mimeMagic->isMatchingExtension( $extension, $mime ) ) {
 1936+ $newNameFixed .= "." .
 1937+ $mimeMagic->getPreferredExtensionForType( $mime );
 1938+ }
18481939 $this->olds[] = array(
18491940 "{$archiveBase}/{$this->oldHash}{$oldName}",
1850 - "{$archiveBase}/{$this->newHash}{$timestamp}!{$this->newName}"
 1941+ "{$archiveBase}/{$this->newHash}{$timestamp}!{$newNameFixed}"
18511942 );
18521943 }
18531944 $this->db->freeResult( $result );
@@ -1887,7 +1978,7 @@
18881979 // Update current image
18891980 $dbw->update(
18901981 'image',
1891 - array( 'img_name' => $this->newName ),
 1982+ array( 'img_name' => $this->newName, 'img_file_ext' => $this->newExt ),
18921983 array( 'img_name' => $this->oldName ),
18931984 __METHOD__
18941985 );
@@ -1912,9 +2003,62 @@
19132004 $status->successCount += $affected;
19142005 $status->failCount += $total - $affected;
19152006
 2007+ $this->fixAddedOldFileExtensionsInDB();
 2008+
19162009 return $status;
19172010 }
19182011
 2012+ /**
 2013+ * Iterate through each old file in oldimage and reevaluate whether
 2014+ * oi_file_ext should be empty or should get a new value based on mime type
 2015+ */
 2016+ function fixAddedOldFileExtensionsInDB() {
 2017+ $dbw = $this->db;
 2018+
 2019+ // grab a list of the distinct MIME types for all old files
 2020+ $result = $dbw->select( 'oldimage',
 2021+ array( 'oi_major_mime', 'oi_minor_mime' ),
 2022+ array( 'oi_name' => $this->newName ),
 2023+ __METHOD__,
 2024+ 'DISTINCT'
 2025+ );
 2026+ $mimetypes=array();
 2027+
 2028+ // build a $mimetypes array
 2029+ while( $row = $dbw->fetchObject( $result ) ) {
 2030+ $mimetypes[]=array($row->oi_major_mime, $row->oi_minor_mime);
 2031+ }
 2032+
 2033+ // for each of the $mimetypes, check the type against the page title
 2034+ // extension. If there's a mismatch, populate oi_file_ext with a
 2035+ // matching file extension. If not, leave it blank.
 2036+ $mimeMagic = MimeMagic::singleton();
 2037+ foreach ($mimetypes as $mimepart) {
 2038+ $mime = $mimepart[0]."/".$mimepart[1];
 2039+ $extension = File::getNormalizedExtensionFromName( $this->newName );
 2040+ wfDebugLog( 'imagemove', "newName: {$this->newName} Mime type: {$mime} Extension: {$extension}" );
 2041+
 2042+ if ( $mimeMagic->isMatchingExtension( $extension, $mime ) ) {
 2043+ $addedExt = '';
 2044+ }
 2045+ else {
 2046+ $addedExt = "." .
 2047+ $mimeMagic->getPreferredExtensionForType( $mime );
 2048+ }
 2049+
 2050+ $dbw->update(
 2051+ 'oldimage',
 2052+ array(
 2053+ 'oi_file_ext' => $addedExt,
 2054+ ),
 2055+ array( 'oi_major_mime' => $mimepart[0],
 2056+ 'oi_minor_mime' => $mimepart[1],
 2057+ 'oi_name' => $this->newName ),
 2058+ __METHOD__
 2059+ );
 2060+ }
 2061+ }
 2062+
19192063 /*
19202064 * Generate triplets for FSRepo::storeBatch().
19212065 */
Index: branches/extensionless-files/includes/filerepo/FSRepo.php
@@ -305,12 +305,56 @@
306306 }
307307
308308 /**
 309+ * Validate and stage a given relative path (e.g. create parent directories)
 310+ * @param string $targetRel Relative path to check/stage
 311+ * @return FileRepoStatus
 312+ */
 313+ function prepTarget( $targetRel ) {
 314+ global $wgCheckFileExtensions;
 315+ global $wgCheckFileExtensions, $wgStrictFileExtensions;
 316+ global $wgFileExtensions, $wgFileBlacklist;
 317+ $status = $this->newGood();
 318+ if ( !$this->validateFilename( $targetRel ) ) {
 319+ throw new MWException( 'Validation error in $targetRel' );
 320+ }
 321+ $mimeMagic = MimeMagic::singleton();
 322+ $ext = File::getNormalizedExtensionFromName( $targetRel );
 323+ $mime = $mimeMagic->guessTypesForExtension( $ext );
 324+
 325+ if ( $ext == '' ) {
 326+ throw new MWException( 'MIME type detection error for $targetRel' );
 327+ }
 328+
 329+ /* Don't allow users to override the blacklist (check file extension) */
 330+ if ( in_array( $ext, $wgFileBlacklist ) ) {
 331+ // Since the file extension uploaded by the user may be
 332+ // different than the extension generated from the mime type,
 333+ // display the mime type instead.
 334+ $status->fatal( 'filetype-badmime', $mime );
 335+ return $status;
 336+ }
 337+ if ( $wgCheckFileExtensions
 338+ && $wgStrictFileExtensions
 339+ && !in_array( $ext, $wgFileExtensions ) )
 340+ {
 341+ $status->fatal( 'filetype-badmime', $mime );
 342+ return $status;
 343+ }
 344+ $targetPath = "{$this->directory}/$targetRel";
 345+ $targetDir = dirname( $targetPath );
 346+ if ( !is_dir( $targetDir ) && !wfMkdirParents( $targetDir ) ) {
 347+ $status->fatal( 'directorycreateerror', $targetDir );
 348+ }
 349+ return $status;
 350+ }
 351+
 352+ /**
309353 * Publish a batch of files
310 - * @param array $triplets (source,dest,archive) triplets as per publish()
 354+ * @param array $tuples (source,dest,current,archive) tuples as per publish()
311355 * @param integer $flags Bitfield, may be FileRepo::DELETE_SOURCE to indicate
312356 * that the source files should be deleted if possible
313357 */
314 - function publishBatch( $triplets, $flags = 0 ) {
 358+ function publishBatch( $tuples, $flags = 0 ) {
315359 // Perform initial checks
316360 if ( !wfMkdirParents( $this->directory ) ) {
317361 return $this->newFatal( 'upload_directory_missing', $this->directory );
@@ -319,30 +363,34 @@
320364 return $this->newFatal( 'upload_directory_read_only', $this->directory );
321365 }
322366 $status = $this->newGood( array() );
323 - foreach ( $triplets as $i => $triplet ) {
324 - list( $srcPath, $dstRel, $archiveRel ) = $triplet;
 367+ foreach ( $tuples as $i => $tuple ) {
 368+ list( $srcPath, $dstRel, $currentRel, $archiveRel ) = $tuple;
325369
326 - if ( substr( $srcPath, 0, 9 ) == 'mwrepo://' ) {
327 - $triplets[$i][0] = $srcPath = $this->resolveVirtualUrl( $srcPath );
 370+ if ( self::isVirtualUrl( $srcPath ) ) {
 371+ $tuples[$i][0] = $srcPath = $this->resolveVirtualUrl( $srcPath );
328372 }
329 - if ( !$this->validateFilename( $dstRel ) ) {
330 - throw new MWException( 'Validation error in $dstRel' );
 373+ wfDebug("upload", __METHOD__.": preparing target dstRel: $dstRel");
 374+ $prepstatus = $this->prepTarget( $dstRel );
 375+ // prepTarget is mainly about directory creation. Abort immediately
 376+ // on directory creation errors since they're likely to be repetitive
 377+ if( !$prepstatus->isOK() ) {
 378+ return $prepstatus;
331379 }
332 - if ( !$this->validateFilename( $archiveRel ) ) {
333 - throw new MWException( 'Validation error in $archiveRel' );
334 - }
335 - $dstPath = "{$this->directory}/$dstRel";
336 - $archivePath = "{$this->directory}/$archiveRel";
337380
338 - $dstDir = dirname( $dstPath );
339 - $archiveDir = dirname( $archivePath );
340 - // Abort immediately on directory creation errors since they're likely to be repetitive
341 - if ( !is_dir( $dstDir ) && !wfMkdirParents( $dstDir ) ) {
342 - return $this->newFatal( 'directorycreateerror', $dstDir );
 381+ $currentPath = "{$this->directory}/$currentRel";
 382+ // only validate archiveRel if we plan to use it, which depends on
 383+ // whether or not currentPath exists
 384+ if ( is_file ( $currentPath ) ) {
 385+ wfDebug("upload", __METHOD__.": preparing target archiveRel: $archiveRel");
 386+ $prepstatus = $this->prepTarget( $archiveRel );
 387+ if( $prepstatus->hasMessage( 'filetype-badmime' ) ) {
 388+ $ext = File::getNormalizedExtensionFromName( $archiveRel );
 389+ $prepstatus=$this->newFatal( 'archivetype-badmime', $currentPath, $ext );
 390+ }
 391+ if( !$prepstatus->isOK() ) {
 392+ return $prepstatus;
 393+ }
343394 }
344 - if ( !is_dir( $archiveDir ) && !wfMkdirParents( $archiveDir ) ) {
345 - return $this->newFatal( 'directorycreateerror', $archiveDir );
346 - }
347395 if ( !is_file( $srcPath ) ) {
348396 // Make a list of files that don't exist for return to the caller
349397 $status->fatal( 'filenotfound', $srcPath );
@@ -353,13 +401,13 @@
354402 return $status;
355403 }
356404
357 - foreach ( $triplets as $i => $triplet ) {
358 - list( $srcPath, $dstRel, $archiveRel ) = $triplet;
 405+ foreach ( $tuples as $i => $tuple ) {
 406+ list( $srcPath, $dstRel, $currentRel, $archiveRel ) = $tuple;
359407 $dstPath = "{$this->directory}/$dstRel";
360 - $archivePath = "{$this->directory}/$archiveRel";
361408
362 - // Archive destination file if it exists
363 - if( is_file( $dstPath ) ) {
 409+ // Archive current file if it exists
 410+ if( is_file( $currentPath ) ) {
 411+ $archivePath = "{$this->directory}/$archiveRel";
364412 // Check if the archive file exists
365413 // This is a sanity check to avoid data loss. In UNIX, the rename primitive
366414 // unlinks the destination file if it exists. DB-based synchronisation in
@@ -369,16 +417,16 @@
370418 $success = false;
371419 } else {
372420 wfSuppressWarnings();
373 - $success = rename( $dstPath, $archivePath );
 421+ $success = rename( $currentPath, $archivePath );
374422 wfRestoreWarnings();
375423 }
376424
377425 if( !$success ) {
378 - $status->error( 'filerenameerror',$dstPath, $archivePath );
 426+ $status->error( 'filerenameerror',$currentPath, $archivePath );
379427 $status->failCount++;
380428 continue;
381429 } else {
382 - wfDebug(__METHOD__.": moved file $dstPath to $archivePath\n");
 430+ wfDebugLog("upload", __METHOD__.": moved file $currentPath to $archivePath");
383431 }
384432 $status->value[$i] = 'archived';
385433 } else {
@@ -402,7 +450,7 @@
403451
404452 if ( $good ) {
405453 $status->successCount++;
406 - wfDebug(__METHOD__.": wrote tempfile $srcPath to $dstPath\n");
 454+ wfDebugLog("upload", __METHOD__.": wrote tempfile $srcPath to $dstPath");
407455 // Thread-safe override for umask
408456 $this->chmod( $dstPath );
409457 } else {
Index: branches/extensionless-files/includes/filerepo/FileRepo.php
@@ -271,13 +271,36 @@
272272
273273 /**
274274 * Get the file name of an image from its title object, possibly with a
275 - * generated extension.
276 - * Stub function pending full implementation of bug 4421.
 275+ * generated extension
277276 */
278277 function getFilenameFromTitle( $title , $mime = NULL ) {
279 - return $this->getNameFromTitle( $title );
 278+ $name = $this->getNameFromTitle( $title );
 279+ $ext = $this->getAddedExtensionFromTitle ( $title, $mime );
 280+ return $name.$ext;
280281 }
 282+
 283+ /**
 284+ * Get an added extension to a filename to make the media type line up with
 285+ * the file name, returning a blank string if no additional extension is
 286+ * needed. Includes the leading dot.
 287+ */
 288+ function getAddedExtensionFromTitle( $title, $mime = NULL ) {
 289+ $name = $this->getNameFromTitle( $title );
281290
 291+ // tack on an extension corresponding to the MIME type if the MIME type
 292+ // is passed in and we figure out we need it.
 293+ $mimeMagic = MimeMagic::singleton();
 294+ $ext = File::getNormalizedExtensionFromName( $name );
 295+ if ( isset($mime) && !$mimeMagic->isMatchingExtension( $ext, $mime ) ) {
 296+ $addext = ".".$mimeMagic->getPreferredExtensionForType( $mime );
 297+ }
 298+ else {
 299+ $addext = '';
 300+ }
 301+
 302+ return $addext;
 303+ }
 304+
282305 static function getHashPathForLevel( $name, $levels ) {
283306 if ( $levels == 0 ) {
284307 return '';
@@ -426,13 +449,19 @@
427450 *
428451 * @param string $srcPath The source path or URL
429452 * @param string $dstRel The destination relative path
 453+ * @param string $currentRel The current relative path to existing file.
 454+ * Usually the same as dstRel, but may be different if the MIME type
 455+ * (and thus file extension) changes.
430456 * @param string $archiveRel The relative path where the existing file is to
431457 * be archived, if there is one. Relative to the public zone root.
432458 * @param integer $flags Bitfield, may be FileRepo::DELETE_SOURCE to indicate
433459 * that the source file should be deleted if possible
434460 */
435 - function publish( $srcPath, $dstRel, $archiveRel, $flags = 0 ) {
436 - $status = $this->publishBatch( array( array( $srcPath, $dstRel, $archiveRel ) ), $flags );
 461+ function publish( $srcPath, $dstRel, $currentRel, $archiveRel, $flags = 0 ) {
 462+ $status = $this->publishBatch( array( array( $srcPath,
 463+ $dstRel,
 464+ $currentRel,
 465+ $archiveRel ) ), $flags );
437466 if ( $status->successCount == 0 ) {
438467 $status->ok = false;
439468 }
Index: branches/extensionless-files/includes/filerepo/File.php
@@ -153,10 +153,13 @@
154154
155155 /**
156156 * Return the file name of this file
157 - * Stub function pending full implementation of bug 4421.
158157 */
159158 public function getFilename() {
160 - return $this->getName();
 159+ if ( !isset( $this->filename ) ) {
 160+ $this->filename = $this->getName()
 161+ . $this->getAddedFileExt();
 162+ }
 163+ return $this->filename;
161164 }
162165
163166 /**
@@ -164,14 +167,38 @@
165168 */
166169 function getExtension() {
167170 if ( !isset( $this->extension ) ) {
168 - $n = strrpos( $this->getName(), '.' );
169 - $this->extension = self::normalizeExtension(
170 - $n ? substr( $this->getName(), $n + 1 ) : '' );
 171+ $name = $this->getName();
 172+ $mime = $this->getMimeType();
 173+ $mimeMagic = MimeMagic::singleton();
 174+ $this->extension = self::getNormalizedExtensionFromName( $name );
 175+ if ( !$mimeMagic->isMatchingExtension( $this->extension, $mime ) ) {
 176+ $this->extension = $mimeMagic->getPreferredExtensionForType( $mime );
 177+ }
171178 }
172179 return $this->extension;
173180 }
174181
175182 /**
 183+ * Return the additional file extension as needed to make the mime type
 184+ * and filename match. Includes the leading dot.
 185+ */
 186+ function getAddedFileExt() {
 187+ if ( !isset($this->file_ext) ) {
 188+ $name = $this->getName();
 189+ $mime = $this->getMimeType();
 190+ $mimeMagic = MimeMagic::singleton();
 191+ $ext = self::getNormalizedExtensionFromName( $name );
 192+ if ( !$mimeMagic->isMatchingExtension( $ext, $mime ) ) {
 193+ $this->file_ext = "." . $this->getExtension();
 194+ }
 195+ else {
 196+ $this->file_ext = "";
 197+ }
 198+ }
 199+ return $this->file_ext;
 200+ }
 201+
 202+ /**
176203 * Return the associated title object
177204 */
178205 public function getTitle() { return $this->title; }
Index: branches/extensionless-files/includes/filerepo/ArchivedFile.php
@@ -108,7 +108,8 @@
109109 'fa_user',
110110 'fa_user_text',
111111 'fa_timestamp',
112 - 'fa_deleted' ),
 112+ 'fa_deleted',
 113+ 'fa_file_ext' ),
113114 $conds,
114115 __METHOD__,
115116 array( 'ORDER BY' => 'fa_timestamp DESC' ) );
@@ -138,6 +139,7 @@
139140 $this->user_text = $row->fa_user_text;
140141 $this->timestamp = $row->fa_timestamp;
141142 $this->deleted = $row->fa_deleted;
 143+ $this->file_ext = $row->fa_file_ext;
142144 } else {
143145 throw new MWException( 'This title does not correspond to an image page.' );
144146 return;
@@ -172,6 +174,7 @@
173175 $file->user_text = $row->fa_user_text;
174176 $file->timestamp = $row->fa_timestamp;
175177 $file->deleted = $row->fa_deleted;
 178+ $file->file_ext = $row->fa_file_ext;
176179
177180 return $file;
178181 }
Index: branches/extensionless-files/includes/Title.php
@@ -2671,13 +2671,15 @@
26722672 if( $this->getNamespace() == NS_FILE ) {
26732673 $file = wfLocalFile( $this );
26742674 if( $file->exists() ) {
 2675+ global $wgCheckTitleFileExtensions;
26752676 if( $nt->getNamespace() != NS_FILE ) {
26762677 $errors[] = array('imagenocrossnamespace');
26772678 }
26782679 if( $nt->getText() != wfStripIllegalFilenameChars( $nt->getText() ) ) {
26792680 $errors[] = array('imageinvalidfilename');
26802681 }
2681 - if( !File::checkExtensionCompatibility( $file, $nt->getDBkey() ) ) {
 2682+ if( $wgCheckTitleFileExtensions &&
 2683+ !File::checkExtensionCompatibility( $file, $nt->getDBkey() ) ) {
26822684 $errors[] = array('imagetypemismatch');
26832685 }
26842686 }
Index: branches/extensionless-files/includes/DefaultSettings.php
@@ -2172,9 +2172,21 @@
21732173 'application/zip',
21742174 );
21752175
2176 -/** This is a flag to determine whether or not to check file extensions on upload. */
 2176+/**
 2177+ * This is a flag to determine whether or not to check file extensions of
 2178+ * resulting uploaded files from uploading or moving a file. This acts as a
 2179+ * proxy for checking MIME types, and ensures files placed in $wgUploadDirectory
 2180+ * have approved file extensions.
 2181+ */
21772182 $wgCheckFileExtensions = true;
21782183
 2184+/**
 2185+ * This is a flag to determine whether or not to enforce matching of page title
 2186+ * with file extensions of uploaded files (e.g if true, disallow a JPEG called
 2187+ * "File:Foo", but allow "File:Foo.jpg").
 2188+ */
 2189+$wgCheckTitleFileExtensions = true;
 2190+
21792191 /**
21802192 * If this is turned off, users may override the warning for files not covered
21812193 * by $wgFileExtensions.
Index: branches/extensionless-files/languages/messages/MessagesEn.php
@@ -2030,6 +2030,7 @@
20312031 Please rename the file and try uploading it again.',
20322032 'badfilename' => 'File name has been changed to "$1".',
20332033 'filetype-badmime' => 'Files of the MIME type "$1" are not allowed to be uploaded.',
 2034+'archivetype-badmime' => 'Previously upload file ("$1") has disallowed extension ("$2"). Please delete this file before continuing.',
20342035 'filetype-bad-ie-mime' => 'Cannot upload this file because Internet Explorer would detect it as "$1", which is a disallowed and potentially dangerous file type.',
20352036 'filetype-unwanted-type' => "'''\".\$1\"''' is an unwanted file type.
20362037 Preferred {{PLURAL:\$3|file type is|file types are}} \$2.",

Follow-up revisions

RevisionCommit summaryAuthorDate
r60778Some extra file deletion debug logging (probably superfluous)...robla07:32, 7 January 2010
r60779Adding forgotten patch-img_file_ext.sql. Critical for bug 4421 fixrobla07:41, 7 January 2010
r60818Followup to r60773. Getting rid of $wgCheckFileExtensions as in its current...robla01:08, 8 January 2010
r60819followup to r60773. Renaming $wgCheckTitleFileExtensions to the now unused $...robla01:29, 8 January 2010

Past revisions this follows-up on

RevisionCommit summaryAuthorDate
r60762Development branch for bug 4421 (Image file extension should not be part of t...robla03:14, 7 January 2010
r60772Adding new functions in preparation for decoupling page title from MIME type ...robla07:13, 7 January 2010

Comments

#Comment by Bryan (talk | contribs)   14:56, 7 January 2010

In UploadBase you changed the file type check from $wgCheckFileExtensions to $wgCheckTitleFileExtensions, yet in SpecialUpload the extensions message is shown based on $wgCheckFileExtensions.

#Comment by RobLa (talk | contribs)   21:18, 7 January 2010

That was on purpose (assuming we're looking at the same lines of code). There's a subtle difference between $wgCheckFileExtensions and $wgCheckTitleFileExtensions. wgCheckFileExtensions is the key that says "do all of the MIME type checking logic for files on disk", whereas wgCheckTitleFileExtensions is "do all of the MIME type checking logic for extensions in page titles".

That said, it appears as though $wgCheckFileExtensions is largely superfluous now. $wgStrictFileExtensions is already there for anyone who wants to ratchet down MIME checking to the provided whitelist. Blacklist checking is unconditional (even when $wgCheckFileExtensions==false), and most of the other cases have been subsumed by $wgCheckTitleFileExtensions

So, there seems to be a couple of different options for moving forward: a. Delete $wgCheckFileExtensions b. Co-opt $wgCheckFileExtensions, renaming the new $wgCheckTitleFileExtensions to $wgCheckFileExtensions.

Any preference there?

#Comment by Bryan (talk | contribs)   21:52, 7 January 2010

b) seems more backwards compatible to me.

#Comment by RobLa (talk | contribs)   01:05, 8 January 2010

Sounds like a plan. I'll get rid of the old version of $wgCheckFileExtensions first, then rename $wgCheckTitleFileExtensions in the next commit.

#Comment by RobLa (talk | contribs)   01:30, 8 January 2010

Done

Status & tagging log