{
  "schema": "auturnal.deep-research-review.v2",
  "slug": "benchmark-causal-validity",
  "title": "When a Benchmark Supports a Claim: Experimental Units, Workload Mixtures, and Causal Validity",
  "category": "AI systems and measurement",
  "dek": "A benchmark can be repeatable and still answer the wrong question. Experimental units, workload composition and intervention design determine what a measured difference means.",
  "abstract": "Computer-systems benchmarks turn observations into comparative claims, but the validity of that step depends on experimental design. This review connects work on measurement bias, rigorous performance evaluation and randomized execution layouts to distinguish repeated observations from independent experimental units. A hierarchical variance model shows why more iterations within one run cannot eliminate between-run uncertainty. A constructed workload-mixture example shows how a system that is faster on each workload can appear slower when the compared mixtures differ. Both figures are analytical demonstrations, not measurements of a compiler or service. The executable companion checks the variance calculation, the mixture reversal and unit invariance of a geometric-mean speedup. A proposed protocol makes the intervention, population, unit of replication, exclusion rules and practical effect size explicit. The objective is a claim whose scope can be defended, rather than a small confidence interval attached to an ill-defined comparison.",
  "english": "Computer-systems benchmarks turn observations into comparative claims, but the validity of that step depends on experimental design. This review connects work on measurement bias, rigorous performance evaluation and randomized execution layouts to distinguish repeated observations from independent experimental units. A hierarchical variance model shows why more iterations within one run cannot eliminate between-run uncertainty. A constructed workload-mixture example shows how a system that is faster on each workload can appear slower when the compared mixtures differ. Both figures are analytical demonstrations, not measurements of a compiler or service. The executable companion checks the variance calculation, the mixture reversal and unit invariance of a geometric-mean speedup. A proposed protocol makes the intervention, population, unit of replication, exclusion rules and practical effect size explicit. The objective is a claim whose scope can be defended, rather than a small confidence interval attached to an ill-defined comparison.",
  "refs": [
    "cs-20-1",
    "cs-20-2",
    "cs-20-3"
  ],
  "sections": [
    [
      "1. The claim comes before the timer",
      "A benchmark should begin with a question about a defined population of executions. Is the claim about one binary on one machine, about future deployments across machines, or about a family of workloads? Each target requires different replication and sampling. The intervention must also be clear. Changing a compiler flag while also changing memory layout, input ordering and process lifetime does not isolate the flag's effect. Precise timing cannot repair an ambiguous intervention. State the estimand in ordinary language before choosing a summary statistic: for example, the change in completed-request latency under a fixed arrival distribution and resource budget. This makes it possible to judge whether the collected observations actually address the intended question.",
      []
    ],
    [
      "2. What the primary studies warn about",
      "Mytkowicz and colleagues show that apparently innocuous experimental conditions can bias computer-systems measurements. Kalibera and Jones develop a framework for handling performance variability and experimental repetition at relevant levels. Stabilizer uses randomized execution layouts to address layout-dependent effects in performance evaluation. These contributions challenge different shortcuts: assuming harmless setup choices, treating all repetitions as equivalent, and inferring causality from one favorable layout. They do not imply that every benchmark is invalid or that randomization solves every confounder. Their shared methodological value is to make sources of variation part of the design. The bibliography also records that the accessible Kalibera-Jones author record identifies a corrected manuscript, so version provenance remains explicit.",
      [
        "cs-20-1",
        "cs-20-2",
        "cs-20-3"
      ]
    ],
    [
      "3. Repetition has a hierarchy",
      "Suppose run r has a run-specific offset and each iteration within that run has additional noise. Averaging more iterations reduces the within-run component, but it does not create new independent run offsets. Under a simple random-effects model, the variance of the grand mean contains a between-run term divided by the number of runs and a within-run term divided by both runs and iterations. If only one process launch is used, thousands of inner-loop measurements may still share the same layout, thermal history and runtime state. Treating them as thousands of independent deployments understates uncertainty about deployment-to-deployment behavior. The appropriate unit of replication follows the claim, not the timer's sampling frequency.",
      []
    ],
    [
      "4. A visible floor on precision",
      "The first figure fixes a between-run standard deviation of two arbitrary latency units and a within-run standard deviation of ten. It then increases iterations for five or twenty independent runs. The standard error approaches a positive floor determined by the between-run term. These numbers are deliberately constructed; they are not estimates from a real system. Their purpose is to show why an experiment can benefit more from additional independent launches than from many more iterations in one launch. An actual study should estimate variance components or use a suitable cluster-aware analysis, while checking whether stationarity and independence across runs are plausible. A narrow interval computed under the wrong hierarchy does not establish precision about the desired population.",
      []
    ],
    [
      "5. Workload mixtures can reverse the headline",
      "Consider two workload classes. System A takes one unit on the easy class and ten on the hard class; system B takes two and twelve. A is faster within both classes. If A is measured with ninety percent hard work, its average is 9.1. If B is measured with ten percent hard work, its average is 3.0. The aggregate comparison makes A look slower because the workloads differ. With the same fifty-fifty mixture, the averages are 5.5 and 7.0. This is a constructed confounding example, not evidence about a product. It demonstrates why a benchmark must report workload composition and standardize or stratify the comparison before interpreting an aggregate difference as an implementation effect.",
      []
    ],
    [
      "6. Aggregation defines the question",
      "An arithmetic mean of latency answers a different question from a geometric mean of per-workload speedups. The latter gives equal weight to log ratios when each workload receives equal weight, while a ratio of total runtimes gives more influence to longer workloads. Neither is automatically correct. The choice depends on whether the target is an equally weighted suite, a deployment workload distribution, total compute cost or a latency service objective. Report the component results so that readers can reconstruct alternative weightings. Changing units should not change a dimensionless speedup; changing workload weights legitimately can. The executable companion checks the former property and the article makes the latter choice visible rather than hiding it inside one score.",
      []
    ],
    [
      "7. Randomization and pairing answer specific problems",
      "Randomizing execution order can reduce association between treatment and time-varying conditions. Pairing nearby or otherwise matched runs can reduce noise when the pairing variable is relevant and specified in advance. Neither method guarantees independence or eliminates all carryover effects. A warmed cache, compiled runtime, thermal state or shared background workload can persist across treatments. A design should describe reset procedures and test for order effects. Randomized layouts address one class of layout dependence but do not replace workload selection or correct timing boundaries. The analysis must follow the assignment and grouping structure used in the experiment, rather than applying an independent-sample formula merely because it is convenient.",
      []
    ],
    [
      "8. Failed runs are observations too",
      "A service that times out under a difficult request cannot be evaluated only on its successful completions without describing that selection. Excluding failures may make an unstable system appear fast. Similarly, discarding slow runs after looking at the results can remove the very interference that a deployment would encounter. Predefine exclusion criteria based on observable setup failures, and publish both the rule and the number of excluded observations. If retries are part of the service contract, include their resource work and end-to-end delay. A benchmark should report whether its claim concerns successful operations conditional on success or the full offered workload. These are different estimands and should not share an unlabeled latency number.",
      []
    ],
    [
      "9. What the companion checks, and what it cannot",
      "The appendix constructs a covariance matrix for the declared hierarchical model and computes the variance of the average directly, then compares it with the closed form. It checks the two-class workload reversal and confirms that geometric-mean speedup is unchanged when both systems' times are rescaled from one unit to another. These deterministic checks can reveal algebraic or aggregation errors. They do not estimate confidence intervals for real benchmark data, validate a causal model of hardware, or establish the effectiveness of Stabilizer. A real evaluation needs independent data and an analysis appropriate to its design. The figures serve as inspectable counterexamples to common reasoning errors, not as performance results.",
      []
    ],
    [
      "10. Reporting that permits a skeptical reading",
      "A defensible report publishes the workload generator or trace, system configuration, timing boundary, run hierarchy, randomization, exclusions and analysis code. It states the practical importance of an effect as well as its uncertainty. Where many configurations were explored, it distinguishes exploratory selection from a confirmatory comparison on fresh runs. The conclusion names the population to which it applies and the conditions likely to break it. Such reporting can make a modest result more valuable than a dramatic but fragile speedup. The standard is not that every source of variation disappears. It is that the reader can identify what was changed, what was held fixed, how uncertainty was assessed and which alternative explanations remain.",
      []
    ]
  ],
  "section_kinds": [
    "author_analysis_or_verification_proposal",
    "source_based_review",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal"
  ],
  "equations": [
    {
      "tex": "Y_{rj}=\\mu+U_r+\\varepsilon_{rj}",
      "explanation": "Independent run effects U with variance sigma_b squared and independent within-run errors with variance sigma_w squared; a declared model, not fitted data."
    },
    {
      "tex": "\\operatorname{Var}(\\overline{Y})=\\frac{\\sigma_b^2}{R}+\\frac{\\sigma_w^2}{RJ}",
      "explanation": "R independent runs with J iterations each in the stated balanced hierarchy."
    },
    {
      "tex": "\\overline{L}_A(w)=1+9w,\\qquad\\overline{L}_B(w)=2+10w",
      "explanation": "Constructed easy/hard latency values; w is the hard-workload fraction and must be matched for a direct comparison."
    },
    {
      "tex": "G=\\exp\\left(\\frac1m\\sum_{i=1}^{m}\\log\\frac{T_{B,i}}{T_{A,i}}\\right)",
      "explanation": "Equal-weight geometric mean of positive per-workload speedups; a different estimand from the ratio of total times."
    }
  ],
  "table": {
    "headers": [
      "Design choice",
      "Question it resolves",
      "Failure mode if omitted"
    ],
    "rows": [
      [
        "Experimental unit",
        "What counts as independent replication",
        "Pseudoreplication"
      ],
      [
        "Workload weights",
        "Which population the aggregate represents",
        "Mixture confounding"
      ],
      [
        "Timing boundary",
        "What work the outcome includes",
        "Hidden preparation or retries"
      ],
      [
        "Predeclared exclusions",
        "Which observations enter the analysis",
        "Selection after seeing outcomes"
      ]
    ]
  },
  "protocol": [
    "Write the estimand and practical effect threshold before measuring.",
    "Specify interventions, workload weights and timing boundaries.",
    "Choose independent run units and record nested iteration structure.",
    "Randomize or pair appropriately and document resets and carryover.",
    "Retain failures and apply only predeclared exclusion criteria.",
    "Publish component results, uncertainty method, raw data and sensitivity to workload weighting."
  ],
  "limitations": [
    "No real system was benchmarked and no empirical confidence interval is claimed.",
    "The balanced random-effects model excludes several possible dependencies and nonstationarity.",
    "The two-workload reversal is a deliberately constructed example.",
    "The focused references do not replace a full experimental-design or causal-inference treatment."
  ],
  "evidence": [
    {
      "source": "cs-20-1",
      "result": "Seemingly harmless setup choices can bias performance measurements.",
      "boundary": "The result motivates explicit controls rather than a claim that every benchmark is wrong.",
      "locator": "ASPLOS paper and IBM publication record."
    },
    {
      "source": "cs-20-2",
      "result": "Performance evaluation should account for variability at relevant repetition levels.",
      "boundary": "The model used here is a simplified analytical construction, not a reproduction of the paper evaluation.",
      "locator": "Corrected author manuscript record at the University of Kent."
    },
    {
      "source": "cs-20-3",
      "result": "Randomized execution layouts can address layout-dependent measurement effects.",
      "boundary": "Layout randomization does not remove workload confounding or every source of variation.",
      "locator": "Stabilizer, mechanism and experimental framing."
    }
  ],
  "research_questions": [
    "Which variance components dominate deployment-to-deployment performance in modern heterogeneous systems?",
    "How can benchmark suites publish workload weights that reflect multiple legitimate user populations?",
    "What reporting format makes exploratory tuning distinguishable from confirmatory evaluation?"
  ],
  "kind": "ai_assisted_research_guide",
  "author": "Codex (AI)",
  "original_language": "en",
  "created_date": "2026-10-22",
  "published_date": "2026-10-22T13:02:56+00:00",
  "publication_status": "published_commissioned_review",
  "version": 5,
  "previous_version": "/guides/benchmark-causal-validity/v4/",
  "analytic": "benchmark-causal-validity",
  "peer_reviewed": false,
  "journal_publication": false,
  "disclosure": {
    "authorship": "Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.",
    "figures": "Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.",
    "review": "This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems."
  },
  "review_scope": "A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.",
  "revision_note": "Initial local English manuscript; no server upload or publication action.",
  "source_slug": "benchmark-causal-validity",
  "source_revision": "41dc5ea7bf1db9868e2ee5b121f7021c823cc018",
  "source_article_sha256": "02aa750d1f095bede208dd1d560f4971633a6fed1ca75c4314d2bb15615ed2d7",
  "sources": [
    {
      "id": "cs-20-1",
      "key": "Bias",
      "title": "Producing wrong data without doing anything obviously wrong!",
      "authors": [
        {
          "given": "Todd",
          "family": "Mytkowicz"
        },
        {
          "given": "Amer",
          "family": "Diwan"
        },
        {
          "given": "Matthias",
          "family": "Hauswirth"
        },
        {
          "given": "Peter F.",
          "family": "Sweeney"
        }
      ],
      "journal": "Proceedings of the 14th international conference on Architectural support for programming languages and operating systems",
      "year": 2009,
      "doi": "10.1145/1508244.1508275",
      "url": "https://research.ibm.com/publications/producing-wrong-data-without-doing-anything-obviously-wrong",
      "verified_date": "2026-10-22",
      "metadata_source": "Crossref deposit, title checked against primary record",
      "author_list_complete": true,
      "csl_type": "paper-conference",
      "access_scope": "Primary record, abstract and selected accessible text; not an exhaustive full-text or supplementary-material audit.",
      "article_slug": "benchmark-causal-validity",
      "source_article_slug": "benchmark-causal-validity"
    },
    {
      "id": "cs-20-2",
      "key": "Stats",
      "title": "Rigorous benchmarking in reasonable time",
      "authors": [
        {
          "given": "Tomas",
          "family": "Kalibera"
        },
        {
          "given": "Richard",
          "family": "Jones"
        }
      ],
      "journal": "Proceedings of the 2013 international symposium on memory management",
      "year": 2013,
      "doi": "10.1145/2464157.2464160",
      "url": "https://kar.kent.ac.uk/33611/",
      "verified_date": "2026-10-22",
      "metadata_source": "Crossref deposit, title checked against primary record",
      "author_list_complete": true,
      "version_note": "The linked University of Kent record explicitly identifies a corrected author manuscript; use that version for methodological details.",
      "csl_type": "paper-conference",
      "access_scope": "Primary record, abstract and selected accessible text; not an exhaustive full-text or supplementary-material audit.",
      "article_slug": "benchmark-causal-validity",
      "source_article_slug": "benchmark-causal-validity"
    },
    {
      "id": "cs-20-3",
      "key": "Stabilizer",
      "title": "STABILIZER: statistically sound performance evaluation",
      "authors": [
        {
          "given": "Charlie",
          "family": "Curtsinger"
        },
        {
          "given": "Emery D.",
          "family": "Berger"
        }
      ],
      "journal": "Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems",
      "year": 2013,
      "doi": "10.1145/2451116.2451141",
      "url": "https://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf",
      "verified_date": "2026-10-22",
      "metadata_source": "Crossref deposit, title checked against primary record",
      "author_list_complete": true,
      "csl_type": "paper-conference",
      "access_scope": "Primary record, abstract and selected accessible text; not an exhaustive full-text or supplementary-material audit.",
      "article_slug": "benchmark-causal-validity",
      "source_article_slug": "benchmark-causal-validity"
    }
  ],
  "original_source_url": "/guides/assets/v9-computer-science-articles-benchmark-causal-validity-json-02aa750d1f095bed.json",
  "pdf": {
    "document_id": "dbce1ca3-f25a-5a79-b248-a7b77eba34b4",
    "version_id": "62446921-07d7-5205-a8e9-d9d8e8964aee",
    "media_type": "application/pdf",
    "authentication_required": true,
    "download_audited": true
  },
  "presentation_change": "Complete version-bound translated reading views and selectable authenticated original/translated PDFs; original scientific text and evidence retained.",
  "source_locale": "en",
  "source_version_id": "2676c848-7cc8-5ace-8405-e316c87bc983",
  "translations": [
    {
      "locale": "zh",
      "translation_id": "57f127a2-590e-5b65-be6d-c9912b6298c2",
      "translated_content_sha256": "1a7207ae8fee3552c6964581f8c5bbe4b1939348748fce7885fd1974a154d317",
      "sha256": "ca91a74f3fbe76d321611fab1b833e03f09fa223c79666ad26e9298cb276f50d",
      "byte_length": 854827
    },
    {
      "locale": "es",
      "translation_id": "a96ae801-20d7-5611-95c5-29c6199967fa",
      "translated_content_sha256": "77ab4726164310aebe757e2a5793197db01486581a4506982c5f7ad618df8ff4",
      "sha256": "e60e90a593e045286ca47e8d6c7b157bb5691416c33c50c7bb40357bb2aa4285",
      "byte_length": 624649
    },
    {
      "locale": "ja",
      "translation_id": "e7989a67-8211-550b-bb59-eae212b648be",
      "translated_content_sha256": "565bf606195ef35d8a1879fa4b7191f4ef995eadbf799b4c3d5e531ce3c1fdcc",
      "sha256": "21345db1caa8c5f6c4c78e1652781ad76fd5e4fcafcb9c275de0244d2d12525f",
      "byte_length": 817514
    },
    {
      "locale": "ar",
      "translation_id": "894f5363-38b2-5a61-ac7c-12eb958d2830",
      "translated_content_sha256": "0d6ec770e17cd084537d88148acc797ce37acb4c368eb58f7898e9c6291c857a",
      "sha256": "8eacd4dee2bdea91a4417e27fa21d8b986a369aa3bd993e894568e542bc9b53d",
      "byte_length": 612297
    }
  ]
}