Eleven ACM papers as discourse graphs, positive and negative

The edge rule, translated for computer science

The key above is written for laboratory and clinical papers, where a negative counts only with a passing positive control and a stated sensitivity. Computer-science papers have equivalents rather than the same things, and the extraction used them. In a security paper the hypothesis of an attack is the property the target is assumed to have, so a demonstrated attack with a measured effect opposes it; a defence is drawn like any other method, against its own claim. In a user study or a field study the control is a control condition or a comparison group, and the sensitivity statement is the reported test with its sample size and interval. In a benchmark the control is a baseline that was tuned or reproduced, not a number copied from another paper, and the sensitivity statement is variance over runs or seeds, an interval, or a significance test. A null or worse result is drawn as opposing the hypothesis only when those hold; otherwise it is drawn as merely consistent with it. For a paper whose point is a null or negative result, the hypothesis is the expectation it set out to test, so that the null opposes it. The brief the extraction followed is kept with the JSON for each graph.

How the eleven were chosen

From ACM’s catalogue as OpenAlex indexes it, articles whose version of record is CC BY, which in practice means the journals and the Proceedings of the ACM series rather than conference volumes. Within that, a deliberate mix rather than a citation ranking: the most-cited CC BY papers at ACM are surveys, perspectives and graphics methods, and a page meant to show negatives alongside positives needs papers of both kinds. The negatives were found by searching titles and abstracts for the words papers use when a result goes the wrong way; the positives are highly cited empirical papers from the same venues. Each licence was checked against Crossref’s version-of-record field, and every text carries ACM’s own CC BY line. The three security papers were added a day later by a different route: Elisa Bertino’s author record in OpenAlex, filtered to ACM and to CC BY on the version of record, gives five articles and one survey; the three articles with an experiment that could have gone either way were taken, the survey and the two remaining articles (a network-measurement index, a privacy-preserving crowdsourcing optimiser) were not. For one of the three, Crossref’s licence field still points at ACM’s copyright policy while the version of record itself carries the CC BY line; the page goes by the version of record. Citation counts are forward citations (works citing the paper, OpenAlex, 19 October 2026).

The eleven

VM warmup rarely reaches a steady state of peak performance (Proc. ACM Program. Lang. OOPSLA, 2017)

Programming languages · Proc. ACM Program. Lang. 1, OOPSLA, 2017

A negative result: the assumption that virtual machines warm up to a steady state, tested across VMs and benchmarks.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 0 support it, 4 oppose it, 5 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Section 1 Introduction): “H1 Small, deterministic programs reach a steady state of peak performance.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS Cause of many non-warmup process executions is unknown: GC and JIT events explain only some on HotSpot and PyPy, and other VMs lack usable instrumentation. (open-question · Section 5.2)
• ISS Which of Krun's many controls matter most for a reliable experiment is open; untangling the interlocking combinations is estimated to need many months of running time. (open-question · Section 10 Conclusions)
• ISS Whether some property of binary trees, fasta and Richards makes bad inconsistency more likely is unresolved; they share no obvious characteristic. (open-question · Section 5 Results)
• ISS Untried: more distinct hardware (non-x86), systematic tests of whether rebuilding VMs affects warmup, and sampling the ASLR space with Stabilizer, which could not be built. (untried · Section 7 Threats to Validity)

Extraction notes: Text appears to be the version of record: Proc. ACM Program. Lang. Vol. 1, OOPSLA, Article 52, October 2017, CC BY 4.0, with a footnote that updates live at arXiv 1602.00602. The appendices the paper refers to (Appendix A DaCapo/Octane, Appendix C curated plots, Tables 8 and 9) are NOT in the text file, which ends at the references, so nothing from them is used. Table 1 in the extraction lost the classifier icons; the row order (flat, warmup, slowdown, no steady state, good inconsistent, bad inconsistent) is inferred from the caption key order and checked against the prose figures (43.4%/72.4% Linux1240v5, 43.5%/74.7% Linux4790, 30.0%/86.1% OpenBSD4790, and 6.5%/6.5%/43.5% for slowdown/no steady state/bad inconsistent on Linux1240v5), which match; the specific slowdown and no-steady-state percentages for Linux4790 and OpenBSD4790 and the per-class process-execution percentages are read from that inferred order. Table 2 is heavily garbled by the PDF extraction (columns interleaved across benchmarks); the JRuby+Truffle binary trees (1082.0) and fannkuch redux (1016.5) and HotSpot binary trees (7.0) steady-iter values are read from rows whose labels are adjacent in the text and are moderate confidence only; the inconsistent-pair counts such as (27,3) could not be attributed to classes because the icons are lost. The paper says 'seven VMs' but lists eight language implementations: GCC 4.9.3 (C, not a VM) plus Graal 0.18, HHVM 3.15.3, JRuby+Truffle 9.1.2.0, HotSpot 8u112b15, LuaJIT 2.0.4, PyPy 5.6.0, V8 5.4.500.43. Benchmarks: binary trees, spectralnorm, n-body, fasta, fannkuch redux (CLBG) and Richards. Skipped: Graal, HHVM, JRuby+Truffle on OpenBSD; fasta on JRuby+Truffle (crashes); Richards on HHVM (too slow). The number of VM-benchmark pairs per machine is not stated explicitly in the text; only the totals 3660 process executions and 7,320,000 in-process iterations are given. The outcomeClass 'positive' on hardware-os-comparison refers to the paper's own H2 (moderately different hardware/OS has little effect), which the authors call 'a somewhat clear validation'; its valence against H1 is my reading (informs). The outcomeClass 'positive' on georges-heuristic-comparison means the comparison showed what the authors set out to show (that the prior heuristic misreports steady states); it is not a result about H1 and its 'informs' valence is my reading. The 'uninformative' tag on the H3 finding is my reading of the authors' statement that they feel unable to validate or invalidate H3 because of few results. Valence 'opposes' on main-experiment findings is justified because the baseline is the hypothesis itself, measured with 30 process executions per pair, outlier handling, and bootstrapped 99% confidence intervals, in a controlled environment designed to favour H1. Minor inconsistencies: Figure 10 caption says 'TruffleRuby' where the text says JRuby+Truffle; Figures 9 and 11 captions refer to 'Linux1240v5's results (seen in Figure 2)', which appears to mean Table 2. Startup-time results (Section 6, 200 process executions per VM, LuaJIT about 5x and JRuby+Truffle about 3500x slower than C) were omitted as not bearing on H1. The abstract gives the headline as 'at most 43.5%' while the introduction gives the range 30.0-43.5%; both are used.

Do fuzzer-found miscompilation bugs matter for real-world applications? (Proc. ACM Program. Lang., OOPSLA, 2019)

Programming languages · Proc. ACM Program. Lang. 3, OOPSLA, 2019

A mixed result: how often fuzzer-found compiler bugs affect real programs, against user-reported and verification-found bugs.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 2 support it, 1 oppose it, 8 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Section 1 Introduction (Problem)): “Some (including some reviewers of this paper) argue that any miscompilation bug, whether fuzzer-found or not, is a ticking bomb that should be regarded as severe, or avoided completely via formal verification”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS Impact on non-functional properties (security, performance) was excluded by assumption; metrics for it, and for the attack surface of bugs used as backdoors, remain to be built. (next-step · Section 7 Future Work (second direction); Section 1 footnote 2)
• ISS Test suites are a weak proxy: identical results cannot separate semantic equivalence from weak tests. Symbolic and multi-version execution proposed to probe binaries further. (next-step · Section 3.3, Section 6 (Threats to Internal Validity), Section 7 (third direction))
• ISS May not generalise beyond mature Clang/LLVM: less mature compilers (e.g. Rust) may have higher-impact bugs; sample is 12% of fixed fuzzer-found miscompilations; replication wanted. (open-question · Section 6 (Threats to External Validity), Section 7 (first direction))
• ISS Whether Csmith's higher binary-impact rate than EMI reflects early low-hanging fruit needs more bugs; how to prioritise compiler bug reports better than reporter info or intuition. (open-question · Section 5.2, Section 5.6)

Extraction notes: VERSION: the text is the version of record (Proc. ACM Program. Lang. 3, OOPSLA, Article 155, October 2019, CC BY 4.0), with PDF extraction artefacts: section signs rendered as 'ğ', quotation marks as 'ł'/'ž', dashes as 'Ð', tables flattened to one row per line (still readable), and one appendix formula garbled ('c1 /nequalc2'). HYPOTHESIS IS IMPLICIT: the paper does not state a hypothesis of its own; the authors say they 'approached this study with an open mind'. The hypothesis label here is my reading of the community claim the paper examines (the 'ticking bomb' quote) combined with the authors' stated prior intuition in Section 5.6 that bugs found while compiling real code 'might trigger more often over applications in the wild than bugs discovered while compiling artificial code', which the results contradicted ('counter-intuitive', 'unexpected result'). The paper's own two conclusions are that (a) miscompilation bugs in a mature compiler seldom impact deployed applications and (b) fuzzer-found bugs have at least as much impact as bugs from other sources; so the headline outcome splits: part (a) opposes 'fuzzer bugs matter', part (b) supports 'fuzzer bugs matter at least as much as user-reported ones'. I set the question outcomeClass to 'partial' for that reason. VALENCE CHOICES THAT ARE MY READING: (1) 'single-test-failure-zsh' is classed negative-no-effect / opposes although there is no statistical test; the paper reports sensitivity checks instead (median 47% statement coverage on the 39 packages gcov could measure, SQLite at more than 98%, divergent runs repeated in fresh environments) and the authors themselves say coverage is 'high enough for not attributing the absence of test suite failures to very poor test coverage', while also calling test suites 'clearly a weak proxy'. A stricter reading would make it inconclusive / consistent-with. (2) 'severity-poor-predictor-of-impact' is classed inconclusive-underpowered because the authors say confidence 'would be increased by bigger samples at the enhancement and release blocker levels' (8 and 4 bugs). (3) The Alive and user-reported comparisons rest on 8 and 10 bugs with no significance test; the paper itself says the Csmith-vs-EMI trend 'should be confirmed using a larger number of bugs'. COUNTS: bugs studied = 45 total: 27 fuzzer-found (10 Csmith, 10 EMI, 5 Orange, 2 Yarpgen), 8 Alive, 10 user-reported. Table 1 lists 35 tool-reported bugs (27 fuzzer + 8 Alive) from 1,033 listed bugs of which 234 were fixed miscompilations; 799 were discarded as unfixed or not miscompilations. The '12%' figure for the fuzzer sample is the paper's own statement; I did not derive it. Packages = 309 (8,283,744 lines C, 782,703 lines C++, 1,250,948 header lines; the paper rounds to 'more than 10 million'). Machine time about 5 months. TABLE ODDITIES: Table 4 shows '0% [130]' for Alive #21242 and Table 2 shows '0% [14]' for Orange #15959, i.e. 0% with a nonzero function count, presumably rounding; the paper does not comment. Function percentages are relative to an approximate average of 202K functions (SD 5%) counted for three compiler versions, as stated in Section 5.2. The abstract counts 'two failures' for fuzzer-found bugs (zsh plus SQLite's test bed) while Table 2 counts 1 because SQLite was an additional experiment outside the 309 packages. Section 5.2 mentions one bug x package pair where the build failed with the buggy compiler because the build ran the test suite; the text says this is detailed 'in the next section' but does not name the package, so I have not attached it to zsh. NOT STATED: no confidence intervals or variance for any rate; the paper gives no per-bug list of which 'manual' inspections were performed beyond the Table 2/4 'manual' column (low / very low / '-').

ChatGPT code-generation non-determinism (ACM TOSEM, 2025)

Software engineering · ACM Trans. Softw. Eng. Methodol. 34(2), 2025

A negative result for a working assumption: one request is not representative, and temperature zero does not make it so.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 0 support it, 5 oppose it, 5 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Section 4.2 (RQ2); also Section 1 and Abstract): “many people's belief that setting the temperature to 0 can make ChatGPT deterministic [7,13,39], because when setting the temperature to 0, the model applies greedy sampling which should indicate full determinism”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS Why temperature 0 is still non-deterministic is unresolved; the hypotheses named are floating point, unreliable GPU calculation and a sparse mixture-of-experts architecture. (open-question · Section 4.2)
• ISS Only Python was studied; whether other programming languages change the degree of non-determinism is untested. (untried · Section 5)
• ISS Only ChatGPT (GPT-3.5, GPT-4) was studied; the authors say the findings may not apply to other LLMs. (untried · Section 5)
• ISS Semantic similarity via test outputs may oversimplify code meaning; embedding-based measures (CodeBERT, GraphCodeBERT) untried; voting and LLM repair loops proposed as mitigations. (next-step · Sections 5 and 7.3)

Extraction notes: VERSION: the text is the ACM version of record, ACM Trans. Softw. Eng. Methodol. 34(2), Article 42, publication date January 2025 (received 3 Nov 2023, revised 7 Aug 2024, accepted 23 Aug 2024); the task brief dated it 2024, the paper's own citation line says 2025, so the title uses 2025. The ACM reference line gives 28 pages; CC BY 4.0 licence stated. MODELS: gpt-3.5-turbo-0125 and gpt-4-0613 (Section 3.3). HYPOTHESIS: the paper is a measurement study with no claim of its own; the hypothesis is the belief the paper sets out against (stability and temperature-0 determinism), so findings showing variation are classed 'refuted' / 'opposes'. My reading of the fairness condition: the comparison is against identical-output determinism itself, so no external baseline needed reproducing; the Kruskal-Wallis test with p<0.05 is reported only for RQ1.1 (92.1% / 39.4% / 40% of tasks), not for the temperature-0 or GPT-4 comparisons, which rest on counts of tasks with zero equal outputs out of 165 / 500 / 164 problems with 5 requests each. SAMPLE SIZES: 829 problems in RQ1 (CodeContests 165 = test set, APPS 500 randomly sampled from 5,000, HumanEval 164 = whole set); RQ2, RQ3, RQ5, RQ6 tables are CodeContests only (165), other datasets 'on our homepage'; the APPS 27.40% and HumanEval 18.29% temperature-0 figures appear only in the RQ2 prose, with no table in the paper. Only 5 requests per task; no CI on any ratio. SELF-INCONSISTENCIES: (a) Table 6, R1 (t=0): max diff 0.00 and mean max diff 0.00 but 'ratio of worst cases' 1.20% for test pass rate, which cannot both hold; (b) 1.20% (Table 6 R1) and 1.21% (Table 7 GPT-4) appear for what should be the same 165-problem denominator, unexplained; (c) RQ5 prose says GPT-4 is more deterministic 'with ... lower OER and OER (no ex.), lower LCS, higher LED and lower structural similarity' but Table 7 shows the opposite direction (higher OER 0.35 vs 0.09, higher LCS 0.61 vs 0.22, lower LED); the table direction is the one consistent with 'more deterministic', so I used the table; (d) RQ6 prose gives Concise-prompt mean LCS at t=0 as 0.07 while Table 8 gives 0.70 (I did not use RQ6 as an experiment); (e) RQ2 prose says temperature 0 is more deterministic 'for all three types of similarities', but Table 5 structural mean values are non-monotonic (United_Diff 0.41 at t=0, 0.61 at t=0.5, 0.33 at t=1; Tree_Diff 0.50 / 0.69 / 0.41), only the pair-mean column is highest at t=0; (f) Section 7.2 improvement factors (16.13x etc.) are given without variance or definition of the averaging. OMITTED: RQ4 (correlation of description length and difficulty with similarity, CodeContests heatmap Figure 6, qualitative examples) and RQ6 (Concise vs Chain-of-Thought prompts: at t=0 Concise OER worst-case 41.82%, CoT 46.06%, Base 43.64%; at t=1 no obvious difference) were left out to keep within five experiments; also the EvalPlus robustness check (Section 5, no numbers given) and the shared-error-type analysis (Section 7.2: at least 65.85% / 73.83% / 90.00% share an error type when all fail, at temperature 0). The 'positive' outcomeClass on the literature survey is my reading (a measurement that answered its question), not the authors' wording. The Section 1 phrasing of the survey result is 'only 21.1% of these papers consider the non-determinism threat in their experiments'. Equations in Section 3.4.2 are garbled in the extraction; no numbers were taken from them.

ChatGPT for unit test generation: correctness gaps and the ChatTester refiner (Proc. ACM Softw. Eng. FSE, 2024)

Software engineering · Proc. ACM Softw. Eng. 1, FSE, 2024

A mixed result: generated tests often fail to compile or pass, but the ones that pass resemble human tests; then a method that improves them.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 6 support it, 0 oppose it, 5 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Abstract): “Our evaluation demonstrates the effectiveness of ChatTester by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS The refiner fixes only compilation errors, not execution failures, because the authors cannot tell whether a failing test or a buggy focal method is at fault. (untried · Section 5.2)
• ISS ChatTester takes about 99 s per test; the authors name locally deployed small LLMs and incremental compilation of the generated test as untried efficiency improvements. (next-step · Section 6.2.1)
• ISS The empirical study (RQ1-RQ4) was run once, not repeated, because of user-study effort; only gpt-3.5-turbo was studied, GPT-4 was excluded for API rate limits. (open-question · Sections 3.4 and 7)
• ISS Manual tests may be in ChatGPT's training data; Levenshtein distance to manual tests averages 1,027 (median 609) characters, 5% of ChatTester tests under 200 characters. (open-question · Section 7)

Extraction notes: Version: the text is the version of record, Proc. ACM Softw. Eng. Vol. 1, No. FSE, Article 76, July 2024, received 2023-09-28, accepted 2024-04-16, CC BY 4.0. Hypothesis choice (my reading): the paper is a two-part study plus method; I used the ChatTester claim from the abstract as the hypothesis and treated RQ1-RQ4 as experiments that inform it rather than test it, so the study findings carry 'informs'. Numbers: 17.3% (compile but fail) comes from the Introduction, not Table 2; 85.5% assertion-error share is the paper's own figure (Table 4 rows sum to 173, which is my addition, not stated). The coverage in Table 5 is computed on the subset of focal methods where both ChatGPT and Evosuite produced executable tests; the subset size is not stated. Table 5: the paper says ChatGPT has the highest coverage among existing techniques, but Evosuite union branch coverage (77.3%) exceeds ChatGPT (65.6%); this holds only against Evosuite first/worst/best/average (my reading). User studies (RQ3, RQ4, RQ7) are reported only as stacked bar charts (Figures 5, 6, 10) with no numeric summaries and no statistical test in the text, so I set their valence to consistent-with rather than supports; the RQ7 user study (219 tests per technique, 0.05 margin at 95% confidence) is mentioned in the RQ7 experiment label but has no finding of its own because no numbers are given. Fairness: the ChatTester vs ChatGPT comparison uses the authors' own baseline run, averaged over 3 runs, but no variance, CI or significance test is reported; AthenaTest is a reproduction by fine-tuning CodeT5 because the original BART model was not released, so 'ChatGPT beats AthenaTest' is against a proxy. Evosuite's higher correctness (97.6%/91.4%) is classed inconclusive because the authors themselves say the comparison does not decide (Evosuite filters invalid tests post-generation while ChatGPT has no validation). Table 8 (iteration distribution) columns sum to 100 tests, 69 compiled and 42 passing, which differs from the Table 7 averages of 73.3% and 41.0%; Table 8 appears to be a single run (my inference, not stated). Table 9 header is garbled in the extraction (model-name subscripts rendered as Unicode math codepoints); I read the columns as CodeLlama-34B default/ChatTester and CodeFuse-34B default/ChatTester, which matches the text's 21.0%/23.0% and 11.0%/18.0% improvements. Table 10 prints EvoSuite jInstagram branch coverage as '13,3', presumably 13.3; I copied it as printed. Self-inconsistency: the RQ7 summary says ChatTester 'consistently outperforms baselines in coverage', but Table 10 shows EvoSuite higher on jInstagram statement coverage (45.3 vs 41.8) and tabula-java branch coverage (9.9 vs 9.2); the body text's 'comparable and even better' is the accurate statement. Whether RQ6 and RQ7 were repeated three times is not explicitly stated (Section 7 says repetition applied 'when automatically evaluating the effectiveness of ChatTester'). The Introduction's phrase 'since the idea of ChatGPT is general and not limited to the specific model' presumably means ChatTester. The HumanEval Python result appears only in Threats to Validity with no table. The abstract's '18.7% more tests with correct assertions' corresponds to the passing-execution rate in Table 7; the paper equates passing with correct assertions. Assertion-count distribution (Figure 4) is described only qualitatively and is omitted.

Community Notes roll-out and engagement with misinformation on X/Twitter (Proc. ACM HCI, CSCW, 2024)

Human-computer interaction · Proc. ACM Hum.-Comput. Interact. 8, CSCW2, 2024

A null result: no evidence that the intervention reduced engagement with misleading posts, with an explanation of why.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 0 support it, 6 oppose it, 6 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Section 5.1 Research Implications): “From this perspective, misleading tweets with CRH notes are expected to have lower engagement, compared to misleading tweets without CRH notes.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS Analysis is limited to English tweets; a multilingual evaluation of Community Notes is needed as the program establishes itself elsewhere. (untried · Section 5.3 Limitations and Future Research)
• ISS Only aggregated retweet and like counts were available; per-tweet engagement timelines before and after a note appears remain unexamined. (next-step · Sections 5.1 and 5.3)
• ISS 14% of fact-checked tweets were unavailable via the API (82.6% retrieval for CRH tweets); removed misleading tweets could mean the effect of Community Notes is underestimated. (open-question · Sections 3.1 and 5.3)
• ISS A pre-selection mechanism that pushes likely-misleading tweets to contributors early in diffusion is proposed to cut fact-checking delay but not tested. (untried · Section 5.2 Practical Implications)

Extraction notes: Version: the text is the ACM version of record, Proc. ACM Hum.-Comput. Interact. Vol. 8, No. CSCW2, Article 428, November 2024, 52 pages, CC BY 4.0; received July 2023, revised January 2024, accepted March 2024. Author list in the text is Yuwei Chuai, Haoye Tian, Nicolas Pröllochs and Gabriele Lenzini, not the 'Chuai, Tilly, Pröllochs, Feuerriegel' given in my brief; I used the text. The paper poses a research question rather than a numbered hypothesis; the hyp quote is the authors' statement of the expected direction in Section 5.1 (framed via X/Twitter's pilot A/B test), which is my choice of the sentence that best states the tested expectation. Classifying the first-intervention F-CRH vs F-NCRH result (Table 2) as 'partial'/'informs' rather than 'positive'/'supports' is my reading of the authors' own reinterpretation: Section 4.2 says Community Notes 'seems to reduce the engagement', but Sections 4.3 and 4.5.2 show the same drop for non-misleading tweets and its disappearance against same-author controls. The 'inconclusive-controls-failed' class for the T-NMR placebo finding is also my reading (the placebo group showed a 'treatment effect', so the F-NCRH control comparison does not isolate the note display). The percentage effects and 95% CIs (35.5%, 33.2%, 39.5%, 35.5%) are the authors' own translations of the coefficients, copied verbatim; I did not derive them. Self-inconsistencies in the text: new-contributor count is 48,927 in Sections 4.1 and 5.1 but 48,928 in Section 4.5.1; Section 4.2 twice writes 'October 6, 2023' where the intervention date is October 6, 2022; Table A5 (short-delay subgroup) reports 1,716 and 6,463 observations in the like-count columns, identical to the unrestricted Table 3, while the retweet columns report 1,293 and 4,823, which looks like a copy error; Tables A11-A14 label the control group 'F-CNRH' where the text uses 'F-NCRH'. The single-treatment framework (Table A3: FCRH x Vis -0.487** retweets, -0.455** likes, n = 13,808; Table A6 placebo: FCRH vs T-NMR -0.079 and -0.037 n.s., n = 6,944) is consistent with the two-treatment results and is not listed as a separate finding. Also reported but not listed: misleading tweets receive fewer retweets (KS = 0.121, p < 0.001) and likes (KS = 0.133, p < 0.001) than not-misleading tweets regardless of the roll-out. All figures in the text are garbled glyph codes; I relied on captions and the tables. Dataset: 68,235 notes, 45,664 unique tweet IDs, 39,268 retrieved (86%), 36,077 English, 30,267 after the July 1, 2021 filter. Fairness condition for 'opposes': observational quasi-experiment with matched control groups, robust standard errors, reported sample sizes, parallel-trend tests and placebo tests; no seeds or runs apply.

LLM coding assistants and programmer productivity (Proc. ACM HCI, EICS, 2024)

Human-computer interaction · Proc. ACM Hum.-Comput. Interact. 8, EICS, 2024

A positive result from a controlled user study of code assistants.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 3 support it, 0 oppose it, 7 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (5.2 Usage Strategies): “While these results already look promising and support the hypothesis that AI will yield a considerable productivity benefit, we are likely only at the beginning of a shift in work practices in software development.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS The sample is young (mean age 26.8) and expertise affected only speed; studies of long-term AI users, senior developers and human-AI teams in the field are called for. (next-step · 5.4 Limitations)
• ISS Tasks were pre-split into sub-tasks, so decomposing high-level requirements into programming steps, an essential part of development, was excluded and deferred to future studies. (open-question · 5.4 Limitations)
• ISS The two AI conditions used related but not identical GPT-3-based models, and LLM output is probabilistic; whether the specific model shaped the results is assumed, not tested. (open-question · 5.4 Limitations)
• ISS Hybrid interfaces, inline chatbot-style interaction within auto-complete or notebook-style conversations as executable cells, are proposed but untried. (untried · 5.2 Usage Strategies)

Extraction notes: Version: the text is the version of record (ACM Reference Format with article 256, Proc. ACM HCI vol. 8 EICS, June 2024; 'First Submitted February 2024; Accepted April 2024'; CC BY 4.0). The paper states no numbered hypotheses or research questions; the hypothesis quoted is the authors' own phrase in Discussion 5.2 ('support the hypothesis that AI will yield a considerable productivity benefit'), and the question label is my paraphrase of the abstract and introduction. Design: within-subjects, 29 recruited, 5 excluded (some for using ChatGPT in the browser during the baseline), N = 24; three Python tasks (text analysis, CSV transformation, PDF editing) with a 15-minute limit each, tool-task combination and order permuted by Latin Square; tests chosen by Shapiro-Wilk, Friedman with Wilcoxon post hoc or repeated-measures ANOVA with t-test post hoc, Bonferroni adjusted; effect sizes are Kendall's W or eta-squared from Table 2 (Table 2 gives no effect size for the Maintainability Index). Inferences of mine: (1) the valence 'opposes' on the two code-quality nulls is my application of the brief's rule, since Performance is one of the five SPACE dimensions of productivity and the study has a control condition, a reported test and N; the authors themselves frame the null as reassuring ('the AI assistants produce code that is about equally good'), and no power analysis is reported, so a reviewer may prefer inconclusive-underpowered; (2) the direction of the task-completion-time difference is not stated in the text (only 'significantly different'); the outlier descriptions (P19, P24 finishing in under four minutes) and Fig. 7 imply the AI conditions were faster, but I did not assert it; (3) the headline 'about 65% more requirements' appears only as a sentence in Discussion 5.1 with no table value behind it. Ambiguities and self-inconsistencies in the text: Fig. 5, 7 and 8 captions print the legend as '**: p < 0.001' where '***' is meant; the text writes 'p =< 0.001' and 'p =< 0.0001' in 4.4 (PDF garble of '<'); quoted participant IDs run up to P29 because excluded participants kept their numbers; 2.3 reports Peng et al.'s gain as 'more than half' and Table 1 as 55.8% while 5.1 cites 56%; the auto-complete average AI snippet size is 42.9 (AI source only) versus 42.4 (all sources), which are different metrics, not a contradiction; 'control' and 'Baseline' are used interchangeably. Behavioural observations recorded in the paper and not given their own finding: P19 copied the feature description, let Copilot generate most of the code and did not thoroughly verify it; experienced users (P2, P14, P24) used Copilot line by line and the chatbot for whole functions while P19 did the reverse; five participants struggled to read Copilot output quickly, two saw clearly incorrect code, six criticised the chatbot's context switch and three its verbosity; 21 participants named positives for each AI condition against four for the baseline. Data and scripts are stated to be on OSF (view-only link in the Open Science section). Revision at the coordinator's request: the two code-quality nulls (percent correctly implemented requirements, Maintainability Index) were downgraded from negative-no-effect / opposes to inconclusive-underpowered / uninformative / consistent-with, because the paper reports no power analysis, equivalence test or interval tight enough to exclude the effect of interest, so a p-value from 24 participants cannot tell no effect from an underpowered test; numbers, locators and quotes are unchanged.

AI Ghostwriter Effect: ownership versus declared authorship for AI-generated text (ACM TOCHI, 2024)

Human-computer interaction · ACM Trans. Comput.-Hum. Interact. 31(2), 2024

A positive result with a null inside it: the effect holds in two studies, and personalisation does not change it.

Read left to right. The question (QUE) is addressed by 4 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 3 support it, 0 oppose it, 9 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Section 3, Research Questions and Hypotheses (H1.1 and H1.2)): “Participants will consider the AI (and not themselves) to be the owner of personalized AI-generated texts but They will not declare AI authorship or AI support when publishing personalized AI-generated texts.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS The postcard task is artificial and short; the effect should be replicated in writing tasks that require longer, more nuanced texts and that are actually sent to someone. (next-step · Section 6.5, Limitations (First))
• ISS Fine-tuning used fewer prompt-completion pairs than OpenAI recommends and Study 2 had no true personalization; whether better personalization changes the effect is untested. (open-question · Section 6.5, Limitations (Second))
• ISS AI was declared by about 20% with a free-text author field but about 70% with a selection menu; how to design declaration interfaces so AI contributions are declared is open. (open-question · Section 6.5, Limitations (Third); Section 6.3)
• ISS Whether the ownership-declaration discrepancy also holds for non-personalized AI writing contexts has not been investigated. (untried · Section 6.5, Limitations (Sixth))

Extraction notes: VERSION: the text is the ACM version of record, ACM Transactions on Computer-Human Interaction vol. 31 no. 2, Article 25, publication date February 2024, CC BY 4.0; the task brief labelled it 2023 but the text itself says 2024 throughout, so the title uses 2024. Study 1 ran July 2022, Study 2 December 2022, pre-registered at aspredicted.org/t4xd8.pdf. INFERENCES OF MINE: (a) I took H1 (H1.1 + H1.2, the AI Ghostwriter Effect) as the graph hypothesis, so findings on H2 (control/leadership), H3 (personalization), H4/H5 (human vs AI ghostwriter) and user experience carry valence 'informs' even where they support the paper's own secondary hypotheses; the personalization null (s1-personalization-null) is classed negative-no-effect because it had a working manipulation check and a between-group test, but note the paper's own H3 predicted exactly this null, so relative to H3 it is a confirmation, and the Bayes factor of 2.38 is weak evidence with only 'half' of 30 (I infer 15) per group. (b) outcomeClass 'partial' for s1-declared-authorship follows the paper's own words 'partial support for H1.2'; 'partial' for s2-ghostwriter-effect-replication is my reading of the authors' statement that the effect was 'reduced when switching from open-ended author declaration fields to predefined response suggestions' (Section 7) together with rs = .24. (c) The 'roughly 80% omitting the AI in Study 1' in s2-ghostwriter-effect-replication is my paraphrase of the paper's 'less than one-fifth of the participants declared the AI as an author' (Section 6) and 'only about 20%' (Section 6.5); the paper also gives 'a max. of 23.3%' (Section 5.2) and 'six and seven participants per Interaction Method' (7/30 = 23.3%), so 'less than one-fifth' is slightly inconsistent with its own counts. SELF-INCONSISTENCIES IN THE PAPER: (1) Section 5.2 says 'about 71.3% in Study 2' mentioned AI, but Table 5 reports 71 (73.9%); 71/96 is 73.96%, so 71.3% looks like a slip. (2) In Table 3 the 'Sense of leadership' rows are numerically identical to the 'Sense of control' rows (F = 28.18, df (2.25, 62.95), eta2p .50; Personalization F 1.05, p .315; interaction F 0.60, p .570), which looks like a copy error; the non-parametric leadership check in footnote 19 gives a different F(3, 84) = 30.26, versus 45.24 for ownership. (3) The leadership LMM reports F(3, 95.65) = 5.93 for a single continuous predictor (3 numerator df is odd); copied as given. (4) The H5 paragraph says 'We found differences in the declaration of authorship (H3)', apparently meaning H5. (5) Table 4 control row shows 't (95) 0.28 .784, 0 .03' with no Wilcoxon entry; I read d = 0.03. (6) Table 2 (Likert medians) is garbled by PDF extraction (e.g. '42 . 1 5'), so I did not use it. NOT FOUND IN TEXT: numeric means or SDs of the ownership, control and leadership visual-analog scales (-50 me to +50 AI) for any condition; they appear only in Figures 2-6 and 8-10, so the direction findings rest on F/t statistics and the authors' prose. Group sizes for true vs placebo personalization are given only as 'half the participants'. Percentages in Table 5 for the no-mention AI cell (26.0%) versus 25/96 = 26.04% are consistent. The Study 1 ownership rating was collected after all four methods, which the authors themselves flag as a possible contamination of the Writing baseline (Section 5 intro). Supplementary results (text remembrance, open-ended answers, attitudes to AI) are on OSF and not in the text.

HIVE-COTE vs Flat-COTE and CNNs for time series classification (ACM TKDD, 2018)

Data mining · ACM Trans. Knowl. Discov. Data 12(5), 2018

A positive result that starts with a negative one: two convolutional networks lose to the existing ensemble before the new ensemble beats it.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 3 support it, 0 oppose it, 9 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Abstract): “We demonstrate that HIVE-COTE is significantly more accurate than Flat-COTE (and all other TSC algorithms that we are aware of) over 100 resamples of 85 TSC problems and is the new state of the art for TSC.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS O(n^2 m^4) complexity may be prohibitive for real-time use on low-powered hardware; cutting runtime without losing accuracy, and fair wall-clock comparison, are left open. (next-step · Section 7.4; Section 9)
• ISS HESCA constituents and all internal parameters (except 1-NN) are unoptimised; the number of RISE intervals and alternative feature combinations beyond ACF+PS are untried. (untried · Section 5.2; Section 9)
• ISS HIVE-COTE could not compensate for confounding modules on spectral simulations; alternative fusion strategies (exponential voting, per-class weighting) are untested. (open-question · Section 8.3; Section 9)
• ISS MCNN significantly beats DTW 1-NN on 44 datasets, so deep learning for TSC warrants further investigation; only one benchmark CNN and one published CNN were compared. (open-question · Section 8.1)

Extraction notes: Text appears to be the version of record: ACM TKDD vol. 12 no. 5 article 52, July 2018, received April 2017, revised December 2017, accepted January 2018, CC BY 4.0; some words lack inter-word spaces from PDF extraction. INFERRED: the significance test behind Figures 13 and 15 is not restated in Section 8; Section 5.1 says the pairwise Wilcoxon signed-rank with Holm correction CD method is used 'throughout the remainder of this paper', so I attributed that test to the 85-dataset comparison. The MCNN comparisons (findings flat-cote-beats-mcnn-44, hive-cote-beats-mcnn-44) use numbers copied from Cui et al. (2016) on a single default train/test split for 44 datasets; the authors did not reproduce MCNN, so these are not resample-level comparisons like the rest; the test for the HIVE-COTE vs MCNN comparison is not named (only 'significant'). The benchmark CNN was the authors' own implementation with a small hyper-parameter grid; no accuracy values are given for it, only ranks in Figure 13 (not readable in text). MY COUNT: from Tables 5-6, Flat-COTE exceeds HIVE-COTE on 15 datasets and they tie on Plane, consistent with the stated 69/85; the examples in finding flat-cote-ahead-on-some-datasets are my selection of the larger gaps. Case studies: Table 8 labels stdev as '(%)' but values such as 0.084 look like proportions; Tables 9-10 give stdev with no unit; no significance tests are reported for any case study, so those findings are informs/partial, not supports. Section 8.3 defers simulator details and fuller results to 'Bagnall et al. (2017)', but the reference list has a 2016 UEA technical report on simulated data experiments; likely a citation mismatch, and Table 7 is titled 'Exeriments' (sic). The interval simulator is self-described as poorly designed ('essentially just simulating a standard classification problem with a large number of redundant features'); not listed as an issue only because of the four-issue cap. Runtime: no wall-clock measurements anywhere; the 'at least an order of magnitude' speedup is asserted from internal investigations without data. Quotes are verbatim except that words hyphenated across PDF line breaks were rejoined ('algo-rithms', 'signif-icantly'); 'worst' in the simulation quote is as printed. The hypothesis in Section 8.3 that HIVE-COTE would be not significantly worse than the best module on every data type was partially refuted (interval, spectral); recorded as a finding rather than as the main hypothesis, which I took from the abstract.

TIMESAFE: PTP spoofing and replay attacks on the Open RAN fronthaul and an ML monitor (ACM TOPS, 2025)

Security · ACM Trans. Priv. Secur. 29(1), 2025

A negative result for a security assumption, then a defence: timing attacks on the Open RAN fronthaul take a base station down, and a monitor detects them.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 3 support it, 3 oppose it, 6 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (1 Introduction, Motivation and Scope): “While WG11 acknowledges that attacks on the S-plane can have a high impact, they consider the likelihood low due to the perceived sophistication required by an attacker”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS Only LLS-C2 and LLS-C3 were studied; GNSS jamming of LLS-C4 and supply-chain attacks on primary reference clocks are untried; Prong C redundancy is a candidate mitigation. (untried · 10 Discussion, Attacks in Other Synchronization Topologies)
• ISS TIMESAFE covers two attacks; packet removal and selective delay manipulation are not tested, and multi-class classification of attack types is proposed as a next step. (next-step · 10 Discussion, Additional Attacks Against PTP)
• ISS Loss weights were left at one; adaptive threshold tuning or online re-weighting of the false-positive/false-negative trade-off is left to future work. (next-step · 8.2 ML Model Description)
• ISS Whether the RU crash and manual-reboot requirement after the dynamic-port spoofing attack is specific to the Foxconn RU is left open by the authors. (open-question · 6.1 Production-ready Network Impact)

Extraction notes: Version: the text is the published journal article, ACM Trans. Priv. Sec. vol. 29 no. 1 article 7, December 2025, CC BY 4.0; footer reads 'Received 9 May 2025; revised 9 May 2025; accepted 5 November 2025' (received and revised dates identical, probably a typesetting slip). Hypothesis choice: per the brief's security-paper rule, the hypothesis is the security assumption the attacks test, taken from the authors' own framing of O-RAN WG11's position (likelihood low because of attacker sophistication). The three attack findings with valence 'opposes' and outcomeClass 'refuted' are demonstrated attacks with measured effects on the production network or the emulated twin; 'refuted' is my reading of what a successful attack does to that assumption (the paper's own words are 'high impact'). The twin spoofing-drift finding is 'informs' rather than 'opposes' because it quantifies severity of an attack the production run already decided, and the twin runs without SyncE (Appendix A.2), which the authors flag as a difference from production. The switch-configuration finding (replay blocked) is classed negative-no-effect/informs as an attack condition; that classification is mine. Detector experiments (exp-offline-detector-comparison, exp-production-trace-detection, exp-reshaped-attack-resilience): as the caller asked, their valences are drawn against the detector's claim, quoted from the Introduction: 'our Machine Learning (ML) model can effectively identify malicious PTP attacks with over 97.5% accuracy across diverse environments', not against the WG11 hypothesis; this is a second reference claim living in one graph, so 'supports' on those findings means 'supports the detector claim'. The rule-based baseline was implemented by the authors, so the baseline is reproduced, but NO result anywhere in the paper comes with seeds, repeated runs, confidence intervals or a significance test: Table 3 is one 80/10/10 split, Fig. 12 is one .pcap trace of one attack, Fig. 13 gives no count of tests. Under the strictest reading of the brief's fairness condition the detector 'supports' edges would downgrade to consistent-with/inconclusive; I kept 'supports' because the gap to the reproduced baseline (97.99% vs 50.95% accuracy) is large, and flag the absence of variance in every effect field for the reviewer to decide. CNN failures: the production-trace CNN result is classed negative-not-replicated (its offline 97.05% recall did not transfer), and the reshaped-attack CNN result negative-not-achievable (0.00% detection), both 'informs' because they bear on the CNN, not on the Transformer that carries the claim. Self-inconsistencies in the paper: the abstract and conclusion say 'over 97.5% accuracy', Section 7 says '99% accuracy in a production environment' and Section 9.1 'exceeding 99%', while Fig. 12(a) shows 95.70% of malicious sequences detected (overall accuracy above 99% is plausible only if benign sequences dominate the trace, which the paper does not state); Table 3 names 'Linear Regression' while Appendix A.3 describes 'Logistic Regression'; Section 7.1 says the heuristic was tested 'under the same conditions we use in Section 9.2' (the reshaped-attack twin run) while Table 3 labels its 50.95%/17.80% as the offline dataset, so it is unclear which setting Fig. 8 reports; Section 8.1 says the dataset and pipeline will be open-sourced 'upon acceptance' while the Introduction says they are already released at reference [5]. Figure 4 caption times (50% drop at ~440 s, 75% at ~510 s, crash ~580 s with attack at ~60 s) and the text's 'about 380 seconds after the beginning of the attack' are consistent but I copied both rather than reconciling them. The '66%' DoS finding does not say which ML model was used. Fig. 6 and 7 panel titles name four trace files (run1-12sep-aerial-udpDL, run1-8sep-aerial-increasingDL-withUL, run2-8sep-aerial-increasingDL-noUL, run3-8sep-aerial-maxDLUL), which I take to be the four background traffic traces. Training cost (never exceeded 30 minutes on one A100) and real-time inference on both models are stated without numbers for latency, so not recorded as findings.

ARIoTEDef: self-evolving IoT early-infection detector against white-box evasion (ACM Trans. Internet Things, 2024)

Security · ACM Trans. Internet Things 5(3), 2024

A positive result that starts with a failure: the infection detector is fully evaded by a white-box attack before self-evolution recovers most of its recall.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 5 support it, 0 oppose it, 6 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Abstract): “Our results show that ARIoTEDef is robust and benefits from multiple rounds of evolution.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS Add host information (system calls, CPU/memory and resource usage) and more kill-chain steps (lateral movement, obfuscation) beyond the three network stages modelled. (next-step · Section 9 Conclusion)
• ISS Robustness gain depends on how many adversarial samples the attacker sends per round; with few samples the learnable perturbation patterns are very limited (Setting II). (open-question · Section 8.5.2)
• ISS Not every evolution round beats the attacker: samples used for retraining depend on the sequence analyzer's predictions, which average only 43.54% recall at the chosen threshold. (open-question · Sections 8.3 and 8.5.2, Table 5)
• ISS Class imbalance (3,915 benign vs 318 infection windows) makes the sequence analyzer's score threshold a key sensitivity; 0.01 was found by multiple tries, not a principled rule. (open-question · Section 8.3)

Extraction notes: Version: text is the published ACM Transactions on Internet of Things Vol. 5, No. 3, Article 15 (June 2024) version, received 22 August 2023, revised 2 February 2024, accepted 24 March 2024; the copyright line reads '2023' while the publication date is June 2024. The hypothesis quote is from the abstract; the conclusion states the same claim more fully but spans a hyphenated line break in the text. Reading choices that are mine, not the authors': (1) the question outcomeClass is 'partial' because Setting I strongly supports the claim while Setting II (fresh samples, small batches) gives only 6.67-14.47% recall; the authors present Setting II as showing 'evolution helps to reduce FNs' and do not call it a limitation; a reader could reasonably treat it as opposing the robustness claim for slow or low-volume attackers; (2) the pre-evolution PGD result is classed 'refuted' (the un-evolved detector's robustness) with valence 'informs' because the paper's claim is about the evolved detector; (3) the overhead finding is classed 'positive' only because the authors present it as lightweight; it is a cost measurement with no comparison. Fairness: all baselines (highest probability, Viterbi, episode tree, HAM, SAAE-DNN, PGD-AT) were implemented or trained by the authors; the regular-setting numbers are averages over 30 trials with no variance given; the adversarial-setting tables give per-seed values for 3 seeds (4 seeds for Table 5); the PGD-AT comparison reports only relative differences 'after three experiments' and absolute values appear only in Figures 16-17, which are not in the text. Self-inconsistencies and gaps: the abstract's baseline F1 of 0.31 is not stated anywhere in the body (Figure 7 only); Section 6.7 gives the un-evolved LSTM F1 as 0.14 on a different test set (set2); the round-0 F1 in Table 7 is 53.39% on the Section 8 dataset, so the '0.31 to 0.87' and the '53.39% to 67.64%' improvements are from different datasets and settings; in Section 8.3 the sentence 'the FNR increases but to a limited extent' almost certainly means FPR (Table 5 shows FNR falling and FPR rising as the threshold drops); Table 2 mixes '39.62%' with a percent sign and other recalls without one. Clean-set F1 in Setting I is non-monotonic (66.98% at 20 rounds, 61.04% at 40, 67.64% at 80), which the authors do not discuss. The '8.38 % higher' and similar PGD-AT figures are copied as printed; whether they are percentage points or relative percentages is not stated. Figures 4-17 contain the regular-setting baselines and the PGD-AT absolute values; none of those figure values are in the text. Source code is stated to be at https://github.com/ariotedef; not verified.

MARS: certified robustness for intrusion-detection DNNs with heterogeneous features (ACM TOPS, 2025)

Security · ACM Trans. Priv. Secur. 28(3), 2025

A positive result: a certification method for intrusion-detection models against randomized-smoothing baselines, with certified and empirical robustness.

Read left to right. The question (QUE) is addressed by 5 experiments (EXP), each producing results (RES), and each result bears on the hypothesis (HYP). 4 support it, 0 oppose it, 8 inform it. Issues (ISS) hang below: what the paper leaves open.

HYP, in the authors’ words (Section 1.2 Contributions): “It achieves a tighter l2 robustness guarantee (12.23% average increase compared to BARS) and extends certification from l2 to l1 and l∞ guarantees compared to other advanced methods.”

• supports positive finding
• opposes a null, worse or failed result against a control or tuned baseline, with a reported test, interval or variance to show it is real
• informs bears on the question without deciding the hypothesis
• thin grey a relation between two experiments

• ISS Certification time grows with first-order gradient search; authors propose hardware acceleration or distributed computing to speed up multi-order certification. (next-step · Section 7.8.2)
• ISS Non-lp robustness certification against structural perturbations is left to future work. (next-step · Section 9 Conclusion)
• ISS Uniform-distribution smoothing for l∞ is limited by its truncation bounds, so certification performance depends heavily on the chosen hyperparameters. (open-question · Section 7.7.2)
• ISS Both detectors have inherent robustness limits: CADE certified accuracy drops rapidly beyond radius 0.5 and ACID becomes sensitive beyond 1.5, regardless of certification method. (open-question · Section 7.1.2)

Extraction notes: Text is the ACM Transactions on Privacy and Security version (Vol. 28, No. 3, Article 40, August 2025, CC BY 4.0), an extended version of a TrustCom 2024 paper [16]. The PDF extraction collapses many spaces; quotes have had spacing restored but no words changed. All three baselines (VRS, FRS, BARS) were run by the authors on the same datasets with seeds {42,43,44} and averages reported (Section 6.1), so none of the baseline numbers is copied from another paper; however only Table 3 reports a ± variance; Tables 4-6 and all figures give point values only. Self-inconsistencies in the paper: the abstract says natural-corruption gains of 16.65% (latency) and 18.23% (packet loss) 'compared to the SOTA method' while Section 1.2 says 16.87% and 19.85% 'compared to the base model', and neither pair appears in Section 7.4, which gives only Figure 12; the l∞-PGD gain over Vanilla is 33.93% in Section 1.2 and 33.94% in Section 7.3.2; the headline '12.23% average increase' in l2 certified radius is stated in the abstract and contributions but is not located in any results table, and the per-class l2 MCR margins over BARS in Tables 5 and 6 are mostly in the third decimal place (my reading). Section 6.5.1 and 7.3.1 configure an l1-PGD attack but Table 3 has no l1-PGD column; the text says the base ACID detector was already robust to it, without numbers. Evasion-attack evaluation is on ACID and the Bot class only, following the BARS setup; CADE is not attacked empirically. The multi-norm comparison in Section 7.2 is against FRS alone and reported only as figures. Table 1 (property grid of methods) lost its check marks in extraction. The outcomeClass 'inconclusive' with valence 'informs' on the smoothing-distribution ablation is my call: it is a secondary design-component result with no numbers, not a test of the central claim. Labelling BARS 'negative-no-effect' under corruption is also my reading of the sentence quoted, backed only by Figure 12.

Licence and what this page is

All eleven papers are CC BY 4.0 on the version of record, so the graphs are published here with attribution; each links to the paper it was read from and quotes it only in short. The graphs are a view of what each paper claims and found, drawn by one extraction pass against a written brief and checked by hand on the headline numbers. They are not the authors’ own summary, and the extraction notes under each graph record what was inferred and what could not be resolved. Corrections from the authors are welcome. The proposal is the same as for the other sets: a publisher could authorise graphs of its own papers, drawn into the fields a Negative-Results Record already holds, and these are what the first ones would look like.

Method: OpenAlex, ACM publisher lineage, article type, licence cc-by on the primary location, two searches (most cited; titles and abstracts mentioning replication, reproducibility, null or negative wording) for the first eight, author record A5061694501 filtered the same way for the three security papers, licence verified per paper via Crossref content-version vor; versions of record fetched from dl.acm.org inside a browser session and converted from PDF; one extraction pass per paper following a written brief, each finding with a locator and a quote; the JSON for each graph is kept with the brief.