r14141 MediaWiki - Code Review archive

Repository:MediaWiki
Revision:< r14140‎ | r14141 | r14142 >
Date:11:35, 9 May 2006
Author:mark
Status:old
Tags:
Comment:
squid-2.5.STABLE13-8wm:
- Include the HIT conversion into MISS during heavy disk I/O patch by
Adrian Chadd
Modified paths:
  • /trunk/rpms/SOURCES/squid-2.5.STABLE13-diskload.patch (added) (history)
  • /trunk/rpms/SPECS/squid.spec (modified) (history)

Diff [purge]

Index: trunk/rpms/SOURCES/squid-2.5.STABLE13-diskload.patch
——@@ -0,0 +1,418 @@
 2+#
 3+#
 4+# add_file "doc/README.diskload"
 5+# content [a508e639e8882f32ac170fb695a3550052212abe]
 6+#
 7+# patch "src/enums.h"
 8+# from [d3225c7b17f9312bef8509d7dd1284df1f0fcf98]
 9+# to [955437e924a040325c5940579961596c655549ca]
 10+#
 11+# patch "src/fs/aufs/store_dir_aufs.c"
 12+# from [99f2f1eebf5dd0b8e29deb65369dcd0c44bb0cd6]
 13+# to [434e3a5f048cd6111395e88fb1397144b4f2cc14]
 14+#
 15+# patch "src/fs/coss/store_dir_coss.c"
 16+# from [19d4f9ab9f40836bd89d4b34d6a707fb07315f80]
 17+# to [371e6c070f6001f5e22de7286ec60b880529ff4b]
 18+#
 19+# patch "src/fs/diskd/store_dir_diskd.c"
 20+# from [b338483de8d86584e890188328fb92711677fc7f]
 21+# to [29790dc96bd0f60cad3f36d201be8a32655cf28d]
 22+#
 23+# patch "src/fs/null/store_null.c"
 24+# from [fa5f237c80376f490212dc9b5431584958991925]
 25+# to [eda38ac409bcb678d4d853ce97e8eecb25ee57e7]
 26+#
 27+# patch "src/fs/ufs/store_dir_ufs.c"
 28+# from [4793d2c314a1b2a0085408592b65bb858ab30058]
 29+# to [7fea6eabaa15ef88ff1cbbc3b2a8f95da3419aa6]
 30+#
 31+# patch "src/store_dir.c"
 32+# from [e795049d82fdee67d27f34a53aed344c9c60f488]
 33+# to [299ef0c831c2b8f781cfe9992ca0912e5baa5792]
 34+#
 35+# patch "src/store_io.c"
 36+# from [b3189954bb7b33712f066ccf62783d2b0506f710]
 37+# to [8a31eb00c0678314dbf38b0fa41d2bbd3f96f3b9]
 38+#
 39+# patch "src/structs.h"
 40+# from [09fd7df0ee0cc66a8be6be822cf34b2203aac563]
 41+# to [2cc0257c851aa9f3899204538a4037888a0465cf]
 42+#
 43+# patch "src/typedefs.h"
 44+# from [0a1930b4adf4cddddda7f44324e88fb53311162f]
 45+# to [af3aabd123fce77fbe2ce6bfc45c942e516c8640]
 46+#
 47+============================================================
 48+--- doc/README.diskload a508e639e8882f32ac170fb695a3550052212abe
 49+@@ -0,0 +1,37 @@
 50++
 51++The disk load calculation code has been modified slightly - it now
 52++attempts to shed load during both periods of extraordinarily busy
 53++disk activity.
 54++
 55++Each filesystem implements slightly different semantics for knowing
 56++when its busy:
 57++
 58++* aufs: doesn't have a concept of 'per-storedir' IO as the thread
 59++ pool is shared between all aufs storedir; loadav is the number
 60++ of currently busy threads out of the number of maximum threads.
 61++
 62++* ufs: since it is synchronous the "busy" heuristic is simply the
 63++ number of currently open files in the storedir. This allows for
 64++ the usage of the storedirs to be relatively even but it doesn't
 65++ help manage disk load.
 66++
 67++* diskd: each storedir has its own SYSV SHM/MSG queue; so the load
 68++ average is calculated based on the number of currently queued
 69++ messages compared to magic1 and magic2.
 70++
 71++* coss: each coss storedir has its own async-io queue (currently
 72++ 128 entries deep); load average the percentage of these in use
 73++
 74++* null: the cache is always busy.
 75++
 76++The main problem with this; Squid will invalidate cached objects
 77++if there's a problem swapping them in; so heavy disk load can cause
 78++both your hit rate to drop and your cache to start losing objects.
 79++
 80++This isn't easily solvable! We'd need to be able to compare what
 81++we've retrieved against what we have (kind of like an IMS) to see
 82++whether we've fetched something "fresher". If we have then we should
 83++invalidate what we have in the cache. If the object we've fetched is
 84++just as fresh as what we have in the cache then we should hold onto
 85++our cached object. This will require a little more thought.
 86++
 87+============================================================
 88+--- src/enums.h d3225c7b17f9312bef8509d7dd1284df1f0fcf98
 89+@@ -744,4 +744,10 @@
 90+
 91+ #endif
 92+
 93++typedef enum {
 94++ ST_OP_NONE,
 95++ ST_OP_OPEN,
 96++ ST_OP_CREATE
 97++} store_op_t;
 98++
 99+ #endif /* SQUID_ENUMS_H */
 100+============================================================
 101+--- src/fs/aufs/store_dir_aufs.c 99f2f1eebf5dd0b8e29deb65369dcd0c44bb0cd6
 102+@@ -106,6 +106,7 @@
 103+ static STDUMP storeAufsDirDump;
 104+ static STMAINTAINFS storeAufsDirMaintain;
 105+ static STCHECKOBJ storeAufsDirCheckObj;
 106++static STCHECKLOADAV storeAufsDirCheckLoadAv;
 107+ static STREFOBJ storeAufsDirRefObj;
 108+ static STUNREFOBJ storeAufsDirUnrefObj;
 109+ static QS rev_int_sort;
 110+@@ -1591,26 +1592,23 @@
 111+ * object is able to be stored on this filesystem. AUFS filesystems will
 112+ * happily store anything as long as the LRU time isn't too small.
 113+ */
 114+-int
 115++char
 116+ storeAufsDirCheckObj(SwapDir * SD, const StoreEntry * e)
 117+ {
 118+- int loadav;
 119+- int ql;
 120++ return 1;
 121++}
 122+
 123+-#if OLD_UNUSED_CODE
 124+- if (storeAufsDirExpiredReferenceAge(SD) < 300) {
 125+- debug(47, 3) ("storeAufsDirCheckObj: NO: LRU Age = %d\n",
 126+- storeAufsDirExpiredReferenceAge(SD));
 127+- /* store_check_cachable_hist.no.lru_age_too_low++; */
 128+- return -1;
 129+- }
 130+-#endif
 131+- ql = aioQueueSize();
 132+- if (ql == 0)
 133+- loadav = 0;
 134+- loadav = ql * 1000 / MAGIC1;
 135+- debug(47, 9) ("storeAufsDirCheckObj: load=%d\n", loadav);
 136+- return loadav;
 137++int
 138++storeAufsDirCheckLoadAv(SwapDir *SD, store_op_t op)
 139++{
 140++ int loadav, ql;
 141++
 142++ ql = aioQueueSize();
 143++ if (ql == 0) {
 144++ return 1;
 145++ }
 146++ loadav = ql * 1000 / MAGIC1;
 147++ return loadav;
 148+ }
 149+
 150+ /*
 151+@@ -1896,6 +1894,7 @@
 152+ sd->statfs = storeAufsDirStats;
 153+ sd->maintainfs = storeAufsDirMaintain;
 154+ sd->checkobj = storeAufsDirCheckObj;
 155++ sd->checkload = storeAufsDirCheckLoadAv;
 156+ sd->refobj = storeAufsDirRefObj;
 157+ sd->unrefobj = storeAufsDirUnrefObj;
 158+ sd->callback = aioCheckCallbacks;
 159+============================================================
 160+--- src/fs/coss/store_dir_coss.c 19d4f9ab9f40836bd89d4b34d6a707fb07315f80
 161+@@ -691,17 +691,21 @@
 162+ * not store everything. We don't check for maxobjsize here since its
 163+ * done by the upper layers.
 164+ */
 165+-int
 166++char
 167+ storeCossDirCheckObj(SwapDir * SD, const StoreEntry * e)
 168+ {
 169+- CossInfo *cs = (CossInfo *) SD->fsdata;
 170+- int loadav;
 171+-
 172+ /* Check if the object is a special object, we can't cache these */
 173+ if (EBIT_TEST(e->flags, ENTRY_SPECIAL))
 174+- return -1;
 175++ return 0;
 176++ return 1;
 177++}
 178+
 179+- /* Otherwise, we're ok */
 180++int
 181++storeCossDirCheckLoadAv(SwapDir *SD, store_op_t op)
 182++{
 183++ CossInfo *cs = (CossInfo *) SD->fsdata;
 184++ int loadav;
 185++
 186+ /* Return load, cs->aq.aq_numpending out of MAX_ASYNCOP */
 187+ loadav = cs->aq.aq_numpending * 1000 / MAX_ASYNCOP;
 188+ return loadav;
 189+@@ -780,6 +784,7 @@
 190+ sd->statfs = storeCossDirStats;
 191+ sd->maintainfs = NULL;
 192+ sd->checkobj = storeCossDirCheckObj;
 193++ sd->checkload = storeCossDirCheckLoadAv;
 194+ sd->refobj = NULL; /* LRU is done in storeCossRead */
 195+ sd->unrefobj = NULL;
 196+ sd->callback = storeCossDirCallback;
 197+============================================================
 198+--- src/fs/diskd/store_dir_diskd.c b338483de8d86584e890188328fb92711677fc7f
 199+@@ -110,6 +110,7 @@
 200+ static STDUMP storeDiskdDirDump;
 201+ static STMAINTAINFS storeDiskdDirMaintain;
 202+ static STCHECKOBJ storeDiskdDirCheckObj;
 203++static STCHECKLOADAV storeDiskdDirCheckLoadAv;
 204+ static STREFOBJ storeDiskdDirRefObj;
 205+ static STUNREFOBJ storeDiskdDirUnrefObj;
 206+ static QS rev_int_sort;
 207+@@ -1819,15 +1820,24 @@
 208+ * object is able to be stored on this filesystem. DISKD filesystems will
 209+ * happily store anything as long as the LRU time isn't too small.
 210+ */
 211+-int
 212++char
 213+ storeDiskdDirCheckObj(SwapDir * SD, const StoreEntry * e)
 214+ {
 215+ diskdinfo_t *diskdinfo = SD->fsdata;
 216+ /* Check the queue length */
 217+ if (diskdinfo->away >= diskdinfo->magic1)
 218+- return -1;
 219++ return 0;
 220++ return 1;
 221++}
 222++
 223++int
 224++storeDiskdDirCheckLoadAv(SwapDir *SD, store_op_t op)
 225++{
 226++ diskdinfo_t *diskdinfo = SD->fsdata;
 227+ /* Calculate the storedir load relative to magic2 on a scale of 0 .. 1000 */
 228+ /* the parse function guarantees magic2 is positivie */
 229++ if (diskdinfo->away >= diskdinfo->magic1)
 230++ return -1;
 231+ return diskdinfo->away * 1000 / diskdinfo->magic2;
 232+ }
 233+
 234+@@ -2199,6 +2209,7 @@
 235+ sd->statfs = storeDiskdDirStats;
 236+ sd->maintainfs = storeDiskdDirMaintain;
 237+ sd->checkobj = storeDiskdDirCheckObj;
 238++ sd->checkload = storeDiskdDirCheckLoadAv;
 239+ sd->refobj = storeDiskdDirRefObj;
 240+ sd->unrefobj = storeDiskdDirUnrefObj;
 241+ sd->callback = storeDiskdDirCallback;
 242+============================================================
 243+--- src/fs/null/store_null.c fa5f237c80376f490212dc9b5431584958991925
 244+@@ -87,9 +87,18 @@
 245+ storeRebuildComplete(&counts);
 246+ }
 247+
 248+-static int
 249++static char
 250+ storeNullDirCheckObj(SwapDir * SD, const StoreEntry * e)
 251+ {
 252++ return 0;
 253++}
 254++
 255++/*
 256++ * We should never, in theory, see an open/create, but just in case..
 257++ */
 258++static int
 259++storeNullDirCheckLoadAv(SwapDir *SD, store_op_t op)
 260++{
 261+ return -1;
 262+ }
 263+
 264+@@ -113,6 +122,7 @@
 265+ sd->statfs = storeNullDirStats;
 266+ sd->init = storeNullDirInit;
 267+ sd->checkobj = storeNullDirCheckObj;
 268++ sd->checkload = storeNullDirCheckLoadAv;
 269+ sd->log.clean.start = storeNullDirWriteCleanStart;
 270+ sd->log.clean.done = storeNullDirWriteCleanDone;
 271+ parse_cachedir_options(sd, NULL, 0);
 272+============================================================
 273+--- src/fs/ufs/store_dir_ufs.c 4793d2c314a1b2a0085408592b65bb858ab30058
 274+@@ -104,6 +104,7 @@
 275+ static STDUMP storeUfsDirDump;
 276+ static STMAINTAINFS storeUfsDirMaintain;
 277+ static STCHECKOBJ storeUfsDirCheckObj;
 278++static STCHECKLOADAV storeUfsDirCheckLoadAv;
 279+ static STREFOBJ storeUfsDirRefObj;
 280+ static STUNREFOBJ storeUfsDirUnrefObj;
 281+ static QS rev_int_sort;
 282+@@ -1613,9 +1614,20 @@
 283+ * object is able to be stored on this filesystem. UFS filesystems will
 284+ * happily store anything as long as the LRU time isn't too small.
 285+ */
 286+-int
 287++char
 288+ storeUfsDirCheckObj(SwapDir * SD, const StoreEntry * e)
 289+ {
 290++ return 1;
 291++}
 292++
 293++/*
 294++ * storeUfsDirCheckLoadAv
 295++ *
 296++ * Return load average from 0 to 1000.
 297++ */
 298++int
 299++storeUfsDirCheckLoadAv(SwapDir *SD, store_op_t op)
 300++{
 301+ ufsinfo_t *ufsinfo = SD->fsdata;
 302+ return 500 + ufsinfo->open_files / 2;
 303+ }
 304+@@ -1906,6 +1918,7 @@
 305+ sd->statfs = storeUfsDirStats;
 306+ sd->maintainfs = storeUfsDirMaintain;
 307+ sd->checkobj = storeUfsDirCheckObj;
 308++ sd->checkload = storeUfsDirCheckLoadAv;
 309+ sd->refobj = storeUfsDirRefObj;
 310+ sd->unrefobj = storeUfsDirUnrefObj;
 311+ sd->callback = NULL;
 312+============================================================
 313+--- src/store_dir.c e795049d82fdee67d27f34a53aed344c9c60f488
 314+@@ -156,7 +156,10 @@
 315+ if (!storeDirValidSwapDirSize(dirn, objsize))
 316+ continue;
 317+ /* check for error or overload condition */
 318+- load = sd->checkobj(sd, e);
 319++ if (sd->checkobj(sd, e) == 0) {
 320++ continue;
 321++ }
 322++ load = sd->checkload(sd, ST_OP_CREATE);
 323+ if (load < 0 || load > 1000) {
 324+ continue;
 325+ }
 326+@@ -197,7 +200,10 @@
 327+ for (i = 0; i < Config.cacheSwap.n_configured; i++) {
 328+ SD = &Config.cacheSwap.swapDirs[i];
 329+ SD->flags.selected = 0;
 330+- load = SD->checkobj(SD, e);
 331++ if (SD->checkobj(SD, e) == 0) {
 332++ continue;
 333++ }
 334++ load = SD->checkload(SD, ST_OP_CREATE);
 335+ if (load < 0 || load > 1000) {
 336+ continue;
 337+ }
 338+============================================================
 339+--- src/store_io.c b3189954bb7b33712f066ccf62783d2b0506f710
 340+@@ -7,6 +7,12 @@
 341+ int create_fail;
 342+ int success;
 343+ } create;
 344++ struct {
 345++ int calls;
 346++ int success;
 347++ int open_fail;
 348++ int loadav_fail;
 349++ } open;
 350+ } store_io_stats;
 351+
 352+ OBJH storeIOStats;
 353+@@ -61,8 +67,23 @@
 354+ storeOpen(StoreEntry * e, STFNCB * file_callback, STIOCB * callback,
 355+ void *callback_data)
 356+ {
 357++ int load;
 358++ storeIOState *sio;
 359++
 360+ SwapDir *SD = &Config.cacheSwap.swapDirs[e->swap_dirn];
 361+- return SD->obj.open(SD, e, file_callback, callback, callback_data);
 362++ store_io_stats.open.calls++;
 363++ load = SD->checkload(SD, ST_OP_OPEN);
 364++ if (load < 0 || load > 1000) {
 365++ store_io_stats.open.loadav_fail++;
 366++ return NULL;
 367++ }
 368++ sio = SD->obj.open(SD, e, file_callback, callback, callback_data);
 369++ if (sio == NULL) {
 370++ store_io_stats.open.open_fail++;
 371++ } else {
 372++ store_io_stats.open.success++;
 373++ }
 374++ return sio;
 375+ }
 376+
 377+ void
 378+@@ -114,4 +135,8 @@
 379+ storeAppendPrintf(sentry, "create.select_fail %d\n", store_io_stats.create.select_fail);
 380+ storeAppendPrintf(sentry, "create.create_fail %d\n", store_io_stats.create.create_fail);
 381+ storeAppendPrintf(sentry, "create.success %d\n", store_io_stats.create.success);
 382++ storeAppendPrintf(sentry, "open.calls %d\n", store_io_stats.open.calls);
 383++ storeAppendPrintf(sentry, "open.success %d\n", store_io_stats.open.success);
 384++ storeAppendPrintf(sentry, "open.loadav_fail %d\n", store_io_stats.open.loadav_fail);
 385++ storeAppendPrintf(sentry, "open.open_fail %d\n", store_io_stats.open.open_fail);
 386+ }
 387+============================================================
 388+--- src/structs.h 09fd7df0ee0cc66a8be6be822cf34b2203aac563
 389+@@ -1616,6 +1616,7 @@
 390+ STSTATFS *statfs; /* Dump fs statistics */
 391+ STMAINTAINFS *maintainfs; /* Replacement maintainence */
 392+ STCHECKOBJ *checkobj; /* Check if the fs will store an object */
 393++ STCHECKLOADAV *checkload; /* Check if the fs is getting overloaded .. */
 394+ /* These two are notifications */
 395+ STREFOBJ *refobj; /* Reference this object */
 396+ STUNREFOBJ *unrefobj; /* Unreference this object */
 397+============================================================
 398+--- src/typedefs.h 0a1930b4adf4cddddda7f44324e88fb53311162f
 399+@@ -288,7 +288,8 @@
 400+ typedef int STDBLCHECK(SwapDir *, StoreEntry *);
 401+ typedef void STSTATFS(SwapDir *, StoreEntry *);
 402+ typedef void STMAINTAINFS(SwapDir *);
 403+-typedef int STCHECKOBJ(SwapDir *, const StoreEntry *);
 404++typedef int STCHECKLOADAV(SwapDir *, store_op_t op);
 405++typedef char STCHECKOBJ(SwapDir *, const StoreEntry *);
 406+ typedef void STREFOBJ(SwapDir *, StoreEntry *);
 407+ typedef void STUNREFOBJ(SwapDir *, StoreEntry *);
 408+ typedef void STSETUP(storefs_entry_t *);
Index: trunk/rpms/SPECS/squid.spec
——@@ -5,7 +5,7 @@
66 Summary: The Squid proxy caching server.
77 Name: squid
88 Version: 2.5.STABLE13
9 -Release: 7wm
 9+Release: 8wm
1010 Epoch: 7
1111 License: GPL
1212 Group: System Environment/Daemons
——@@ -38,6 +38,7 @@
3939 Patch252: squid-2.5.STABLE13-errors.patch
4040 Patch253: squid-2.5.STABLE13-nomanglerequestheaders.patch
4141 Patch254: squid-2.5.STABLE13-htcp2.patch
 42+Patch255: squid-2.5.STABLE13-diskload.patch
4243
4344 BuildRoot: %{_tmppath}/%{name}-%{version}-root
4445 Prereq: /sbin/chkconfig logrotate shadow-utils
——@@ -74,6 +75,7 @@
7576 %patch252 -p0 -b .errors
7677 %patch253 -p0 -b .nomanglerequestheaders
7778 %patch254 -p0 -b .htcp2
 79+%patch255 -p0
7880
7981 # Fetch the Wikimedia error page from SVN
8082 cp -a errors/English errors/Wikimedia
——@@ -230,6 +232,10 @@
231233 chgrp squid /var/cache/samba/winbindd_privileged > /dev/null 2>& 1 || true
232234
233235 %changelog
 236+* Tue May 9 2006 Mark Bergsma <mark@nedworks.org> 7:2.5.STABLE13-8.WM
 237+- Include the HIT conversion into MISS during heavy disk I/O patch by
 238+ Adrian Chadd
 239+
234240 * Wed May 3 2006 Mark Bergsma <mark@nedworks.org> 7:2.5.STABLE13-7.WM
235241 - Include a fix by Tim Starling that removes all dynamic malloc() style
236242 calls in htcp.c, and hopefully fixes the high user CPU problems and

Status & tagging log