Is kindergarten the new first grade?

The authors say in Implications and Next Steps that further research is needed and is currently under way on whether the 1998 to 2010 changes helped or harmed children, so this is their own plan; what they leave unspecified is the comparator, how classroom style would be measured in a way that is comparable across two surveys with different item wording, and what result would count as no effect.

Question

Did the 1998 to 2010 shift toward advanced academic content, teacher-directed time, worksheets and testing change children's kindergarten-year gains in reading, mathematics and teacher-rated social and behavioural skills?

Hypothesis

Children in classrooms with the 2010-style profile gain more in reading and mathematics and lose nothing in teacher-rated social and behavioural skills, once entry skills are held constant.

Attempt type

computational
Vocabulary note: This is a secondary statistical analysis of national survey and child assessment microdata. In the NRR vocabulary 'computational' means simulation or in-silico work, and there is no value for observational analysis of an existing cohort, which is what this is.

Design

Pool the child-level files of the two Early Childhood Longitudinal Study, Kindergarten Class cohorts (ECLS-K 1998 and ECLS-K 2011) and keep the public school kindergartners who link to the teacher questionnaires the paper used, about 2,500 teachers in 1998 and about 2,700 in 2010. Build a 2010-style classroom index from only the items the paper shows changed: advanced literacy and mathematics skill coverage, three or more hours per day of teacher-directed whole-class time, daily worksheets and basal readers, under one hour per day of child-selected activity, daily art and music, and presence of art, science and dramatic play areas. Standardise the index within cohort so wording differences do not travel across waves. Outcomes: fall-to-spring item response theory gains in reading and mathematics, plus the teacher-reported approaches-to-learning, self-control, interpersonal and externalizing scales. Estimate within cohort with fall score, child, teacher and school covariates and school fixed effects where a school has more than one sampled classroom, then decompose how much of the cohort gap in gains the index shift explains. Pre-register the index and covariates.

Controls

• within-subject Fall-to-spring gain with the child's own fall score as baseline, estimated within cohort rather than across cohorts.
  Rules out: That the 2010 cohort simply entered kindergarten with stronger skills, which Bassok and Latham 2017 show it did, rather than learning more during the year.
• known-positive-task Reproduce the Engel, Claessens and Watts 2016 result that time on advanced mathematics content is positively and time on basic content negatively associated with kindergarten mathematics learning, in the pooled file, before reporting anything about social and behavioural outcomes.
  Rules out: That a null on the social and behavioural outcomes comes from a broken linkage, a mis-specified weight or a mis-built index rather than from an absence of effect.
  Vocabulary note: The vocabulary means a benchmark task a model is known to pass. Here the positive control is a published association that the analysis pipeline must recover from the same pooled data before any null elsewhere is believed.
• background Regress the fall score, measured before the child experienced the classroom, on the classroom index with the same covariates. The coefficient should be zero.
  Rules out: That children were sorted into classroom styles on prior skill, which would make the spring association an artefact of assignment rather than of instruction.
  Vocabulary note: This is a placebo-outcome falsification test. The NRR negative-control kinds are written for treated and untreated biological samples and have no value for a pre-exposure outcome regressed on later exposure.

Sensitivity

Cannot be derived from what is published. The paper reports only percentages of teachers above thresholds and no child outcomes at all, so no variance is available. A minimum detectable effect needs two inputs: the intraclass correlation of spring reading and mathematics scores within kindergarten classrooms, and the standard deviation of fall-to-spring gains in each cohort. Both are computable from the ECLS-K public and restricted files before the analysis is run, and the design should publish the resulting minimum detectable effect for a one standard deviation move in the index as part of the pre-registration, because the social and behavioural null is the result most likely to be reported.

Would count as

Supported → positive · opposed → refuted
Inconclusive if: The index has almost no within-cohort variance left after full-day status and school covariates are absorbed. Several of the paper's items are near ceiling in one wave, for example literacy instruction reported daily by 96 percent of teachers in 1998, so a within-cohort contrast may have nothing to work with, which would be inconclusive-underpowered rather than a null.

Already done

• Across both ECLS-K cohorts, teachers emphasise basic mathematics content; time on advanced content is positively associated with kindergarten mathematics learning and time on basic content negatively associated with it. (Engel, Claessens and Watts 2016, Educational Researcher, doi:10.3102/0013189X16656841)
  So: The mathematics half of this issue already has an associational answer in the same two datasets, so a new design earns its place only by adding the non-academic outcomes, the non-content dimensions and an identification strategy.
• In a nationally representative kindergarten sample, exposure to advanced academic content was associated with better mathematics and English achievement and also with better interpersonal skills, better approaches to learning, better attentional focus and lower externalizing behaviour, so no trade-off was found. (Le, Schaack and Neishi 2019, American Educational Research Journal, doi:10.3102/0002831218813913)
  So: The harm-to-social-skills branch already has an answer pointing against harm for content coverage, so this design must target the dimensions Le and colleagues did not index, namely teacher-directed time, worksheet and textbook use, testing frequency and the loss of art, music and activity centres.
• In the 1998 cohort, exposure to advanced content was associated with greater learning for all children, including those who had attended preschool, which is the basis of the basic versus advanced content distinction. (Claessens, Engel and Curran 2013, American Educational Research Journal, doi:10.3102/0002831213513634)
  So: Supplies the validated content measure and the expected direction in the earlier wave, so the index should be built to be comparable with theirs rather than invented fresh.
• The effects of instructional approach in kindergarten differ by child characteristics, so a single average effect of didactic versus child-centred instruction is the wrong estimand. (Chiatovich and Stipek 2016, The Elementary School Journal, doi:10.1086/687751)
  So: Interactions with entry skill and family income must be pre-specified rather than discovered, otherwise a null average effect will be reported alongside a search over subgroups.
• The 2010 cohort entered kindergarten with stronger teacher-reported mathematics and literacy skills than the 1998 cohort, with mixed results for behaviour and the largest academic gains among Black children. (Bassok and Latham 2017, Educational Researcher, doi:10.3102/0013189X17694161)
  So: Any cross-cohort outcome comparison that does not condition on fall scores will recover an entry effect and call it a classroom effect.
• Misalignment between preschool and kindergarten teachers' beliefs about early school competencies predicts children's kindergarten adjustment. (Abry, Latham and Bassok 2015, Early Childhood Research Quarterly, doi:10.1016/j.ecresq.2015.01.001)
  So: Gives a belief-to-child-outcome pathway by two of the same authors, which means teacher beliefs should enter the index as a separate dimension rather than be folded into practice.

Searched: Searched OpenAlex for works citing this paper, which has 581 citations, filtered on accountability, content coverage, Common Core and achievement; searched Crossref by bibliographic string for the Engel, Claessens, Le, Chiatovich, Abry and Bassok follow-ups and read their abstracts. I did not find any published study that estimates the effect of the full bundle of documented changes on child outcomes, and I did not find a published output of the research the authors say is under way.

Feasibility

An education policy research group holding a National Center for Education Statistics restricted-use licence, since the teacher-to-child linkage and school identifiers are not in the public files. The authors' own group at the University of Virginia is the obvious candidate. · 9 to 18 months · low, because it is secondary analysis of two existing federal datasets and the cost is analyst time plus a data licence
Blockers: Restricted-use licence and disclosure review before any school-level or state-level output can be released; Item wording differs across the two waves, which is the same problem flagged in iss-3 and limits how much of the index is comparable; Teacher-reported behaviour scales may not be equated across the two cohorts

Strongest objection

The classroom index and the social and behavioural outcomes come from the same respondent. A teacher who reports heavy advanced content is also the person rating the children's self-control, so shared reporter variance can generate the association with no classroom-level cause at all; and because children are not randomly assigned to classrooms, conditioning on a fall score does not remove sorting on parent demand for academic classrooms.

Not derivable from what is published

• Intraclass correlation of spring reading and mathematics scores within kindergarten classrooms
• Standard deviation of fall-to-spring gains in either cohort
• Whether the teacher-reported behaviour scales are equated across the 1998 and 2010 cohorts
• Whether any schools appear in both ECLS-K samples
• What the research the authors say is under way actually measured, and whether it has since been published

The paper documents a change without testing what produced it, and the authors say the policy meaning depends on the answer, because pressure from above and adaptation to better prepared children call for different responses.

Question

How much of the 1998 to 2010 rise in kindergarten academic focus tracks exposure to elementary accountability, and how much tracks children arriving by way of school-based public preschool?

Hypothesis

Accountability exposure dominates: the increase is larger in public schools and in earlier-adopting accountability states than in private schools and later adopters, while the preschool channel adds little because overall preschool attendance barely moved.

Attempt type

computational
Vocabulary note: This is a quasi-experimental policy evaluation on survey microdata with hand-coded policy histories. 'computational' is the closest of the five values but means simulation or in-silico work; the vocabulary has no value for observational policy evaluation.

Design

Three contrasts on the same two ECLS-K waves, all run on the fifteen measures the paper's Table 6 uses. First, a sector contrast: private school kindergarten teachers, who are sampled in both waves but sit outside state accountability, against the public school teachers the paper analysed. Second, a within-sample preschool contrast using the paper's own covariate: schools offering a preschool programme rose from 0.36 to 0.52 of the sample (Appendix A1), and children attending preschool and kindergarten in the same building rose from 0.12 to 0.17, while the share of children who attended any formal preschool was flat at 0.68 and 0.67, so the design asks whether practice changed more in schools that added a preschool programme. Third, a state contrast under a restricted-use licence, attaching dated state policy measures, pre-2002 consequential accountability adoption and kindergarten entry assessment mandates, and estimating cohort-by-exposure interactions with state fixed effects and school-clustered standard errors. Report all three on the probability metric and horse-race the exposures in one model.

Controls

• no-treatment Private school kindergarten teachers in both waves, who faced the same cultural shift, the same parenting trends and much of the same preschool expansion but not state or federal school accountability.
  Rules out: That a general change in beliefs about early childhood, or in what parents did at home, produced the whole shift independently of accountability.
  Vocabulary note: The vocabulary means an untreated biological sample. Here it is a population not exposed to the policy, which is the same logic but a different object.
• designated Daily recess and daily physical education, which the paper shows did not fall and in the case of recess rose nine points, used as outcomes the accountability channel should not move; plus teacher demographics, which no channel should move.
  Rules out: That the accountability exposure measure is simply picking up general state time trends, which would also move outcomes the mechanism cannot touch.
  Vocabulary note: Used here as a falsification outcome rather than a known-good sample, which the vocabulary does not provide for.
• background An independent pre-period trend from the National Center for Education Statistics Schools and Staffing Survey tables for grade one through four self-contained classrooms, which report hours per week on four subjects for selected years from 1987 to 1988 onward.
  Rules out: That the instructional shift was already under way at the same rate before accountability, which would invalidate the before-and-after comparison.
  Vocabulary note: Used for a parallel-trends check, which has no NRR control kind.

Sensitivity

The state contrast cannot be powered from what is published and may not be feasible at all: ECLS-K was not designed to support state estimates, and the paper gives no count of sampled schools per state. The number of sampled schools per state per wave is the input needed, and it is available only under a restricted-use licence. The two other contrasts are better placed: the preschool-offering split is close to balanced in both waves, 0.36 against 0.64 in 1998 and 0.52 against 0.48 in 2010, which is the most favourable split available in these data, and the paper's own significant difference-in-differences estimates on this sample of between 3,700 and 4,000 teachers give a working floor for what the design can resolve. An exact minimum detectable effect still needs the intraclass correlation of each outcome within schools, which the paper does not report.

Would count as

Supported → positive · opposed → refuted
Inconclusive if: Too few schools per state survive disclosure rules to estimate the state interactions, and the private school subsample is too small to detect a sector difference of the size the public-sector change implies. That is inconclusive-underpowered, not evidence that accountability had no role.

Already done

• No Child Left Behind increased instructional time in reading and mathematics in elementary schools and changed school resources and practices, estimated from the Schools and Staffing Survey with a comparison-group design. (Dee, Jacob and Schwartz 2013, Educational Evaluation and Policy Analysis, doi:10.3102/0162373712467080)
  So: The accountability-to-instructional-time link is already established for the elementary grades, so the untested extension is kindergarten, which is not a tested grade, and the design should be framed as a trickle-down test rather than a fresh question.
• In North Carolina's Quality Rating and Improvement System, a quasi-randomly assigned lower quality rating caused measurable improvement in classroom quality and a loss of enrolment, with effects concentrated among providers facing more competition. (Bassok, Dee and Latham 2019, Journal of Policy Analysis and Management, also NBER working paper 23859, doi:10.1002/pam.22149)
  So: Two of the same authors have shown early childhood providers respond to accountability incentives, but in child care ratings rather than public kindergarten, so the mechanism is credible and still untested in the setting this paper documents.
• Children entering kindergarten in 2010 had stronger teacher-reported mathematics and literacy skills than the 1998 cohort. (Bassok and Latham 2017, Educational Researcher, doi:10.3102/0013189X17694161)
  So: The responsive-to-readier-children branch already has direct supporting evidence, so this design must run the two channels against each other rather than test accountability on its own.
• Income and racial school readiness gaps at kindergarten entry narrowed between the 1998 and 2010 cohorts. (Reardon and Portilla 2016, AERA Open, doi:10.1177/2332858416657343)
  So: Gives a differential prediction the design can test: if kindergarten was adapting to readier children, practice should have changed most where readiness rose most, which is the opposite pattern from the one the paper reports for high-poverty schools.
• About 37 percent of the language, literacy and mathematics content covered in kindergarten is redundant with content covered in one state's public preschool programme, with the highest redundancy for basic content and for children in families at or below the poverty line. (Cohen-Vogel, Little and Jang 2021, AERA Open, doi:10.1177/23328584211006163)
  So: If kindergarten were adapting to preschool graduates, redundancy should be falling, so redundancy measured with paired preschool and kindergarten teacher surveys is a direct observable for the preschool channel and an instrument this design can reuse.

Searched: Searched OpenAlex citing works on accountability and preschool expansion, Crossref for the Dee, Bassok, Reardon, Cohen-Vogel and Little papers, and the web and NBER for the Bassok, Dee and Latham accountability paper, whose setting I corrected after reading the abstract: it is North Carolina child care quality ratings, not school accountability. I did not find any study that separates accountability pressure from preschool expansion as drivers of change in kindergarten classroom practice.

Feasibility

A policy evaluation group with a restricted-use licence plus someone able to code state accountability and kindergarten entry assessment histories back to the middle 1990s. · 12 to 24 months · moderate, because the restricted data licence and the hand-coded policy timelines are both slow, though no new data collection is needed
Blockers: ECLS-K is not designed to be state-representative, so the state contrast may fail on cell sizes alone; Disclosure review of any state-identified output; State-funded preschool enrolment series are not available on a comparable basis back to 1998, so the preschool exposure measure leans on the in-sample proxies; The private school subsample in both waves may be too small, and private school teachers were excluded from the paper's analysis so their item coverage needs checking

Strongest objection

No Child Left Behind was federal and arrived in every state in 2002, so the variation this design exploits is pre-existing state accountability, which is correlated with nearly everything else about a state, including preschool investment. A specialist would say the difference-in-differences estimate absorbs general state trends and labels them accountability, and that twelve years with one federal shock is not enough variation to separate two channels that moved together.

Not derivable from what is published

• Number of sampled schools per state in each wave
• State-funded preschool enrolment share of four-year-olds for 1998 on a basis comparable with 2010
• Size and item coverage of the private school kindergarten teacher samples in both waves
• Whether district accountability exposure can be coded at all from the restricted files
• Intraclass correlation of each Table 6 outcome within schools

Footnote 4 says the authors would ideally have compared minutes per day or week per subject and that item wording differences prevented it, and the Analytic Plan says the two datasets do not follow the same teachers or schools, so this is their own stated wish; what is missing is a way to bridge the two wordings and a panel in which the same teacher is observed twice.

Question

Can the 1998 to 2010 change in kindergarten time use be expressed in minutes per week per subject, and does a change of that kind show up within the same teachers rather than only between cohorts?

Hypothesis

A wording calibration will show the binary frequency items understate the change, so that minutes per week on literacy and mathematics rose and minutes on art, music and child-selected activity fell, with the same direction appearing within teacher.

Attempt type

evaluation
Vocabulary note: This is primary survey measurement work: a questionnaire-wording calibration with daily teacher logs and independent classroom observation. In the NRR vocabulary 'evaluation' means benchmarking a model or system, and there is no value for instrument calibration or for primary field measurement in a school.

Design

Two parts. Part one is a bridge study. Recruit currently serving kindergarten teachers and have each complete both the 1998 wording and the 2010 wording of the subject frequency and time-use items, in randomised order, two weeks apart, plus a five-day minute-level daily log of the kind Rowan, Camburn and Correnti used for literacy teaching, and in a subsample one independent observer day using the time-sampling method of Pianta and colleagues. Estimate the mapping from each wording to logged minutes, including the differential item functioning between the two wordings, then apply that mapping to the published 1998 and 2010 marginals to produce minutes estimates with an uncertainty band that carries the wording error. Part two is a panel: resurvey the same teachers and the same schools three years later with fixed wording and the same logs, so a within-teacher change can be set beside the between-cohort change. Carry the calibration into the ECLS-K 2024 teacher questionnaire so the third wave does not repeat the problem.

Controls

• within-subject Each teacher answers both item wordings and logs her own minutes, so the wording effect is identified inside the teacher.
  Rules out: That differences between the two wordings reflect differences between the kinds of teacher who answered each survey rather than the wording itself.
• designated Items whose wording did not change across the two waves, for example daily physical education and daily recess, carried through the bridge as anchors.
  Rules out: That the estimated wording effect is really a change in how teachers interpret all time questions, which would show up as a spurious anchor-item effect and invalidate the mapping.
  Vocabulary note: These are anchor items in the equating sense. The NRR control kinds have no value for an anchor item whose measurement is assumed stable.
• background Independent observer time-sampling in a subsample, compared with the same teachers' self-reported logs for the same days.
  Rules out: That teachers inflate reported minutes on subjects they believe they should be teaching, which would bias the mapping in exactly the direction the hypothesis predicts.
  Vocabulary note: An external measurement comparator rather than a background signal level.

Sensitivity

Cannot be set from what is published. The paper reports only the share of teachers above a frequency threshold, never minutes or their spread, so no minimum detectable minutes difference can be computed from the record. Two inputs would settle it: the between-teacher standard deviation of logged minutes per subject per week in kindergarten, which the teacher-log literature can supply, and the test-retest reliability of each of the two item wordings, which only the bridge study can supply. One external anchor exists and should be used to sanity-check magnitude: the National Center for Education Statistics Schools and Staffing Survey publishes average hours per week on four subjects for grade one through four self-contained classrooms for selected years from 1987 to 1988 through 2011 to 2012, so a minutes trend already exists one grade above kindergarten.

Would count as

Supported → positive · opposed → refuted
Inconclusive if: The differential item functioning between the two wordings turns out to be as large as the twelve-year change itself, or the mapping from wording to minutes is not monotone. The minutes comparison then stays unidentified no matter how large the bridge sample is, which is a measurement failure rather than a finding about kindergarten.

Already done

• Daily teacher logs can measure the enacted curriculum at scale, demonstrated for literacy teaching in third-grade classrooms. (Rowan, Camburn and Correnti 2004, The Elementary School Journal, doi:10.1086/428803)
  So: Supplies the instrument and the feasibility evidence, so the bridge study is applying an established method rather than inventing one, and the same group's time-diary work gives the analysis conventions.
• Direct observation of elementary classrooms yields minute-level time allocation across subjects and activity formats, and observed time use differs from what reports imply. (Pianta, Belsky, Houts and Morrison 2007, Science, doi:10.1126/science.1139719)
  So: Gives the observational comparator and the reason to include it: a self-report bridge without an observational arm cannot detect reporting drift.
• Average hours per week on four subjects for public school teachers of grade one through four self-contained classrooms are published for selected years from 1987 to 1988 through 2011 to 2012. (National Center for Education Statistics, Schools and Staffing Survey table series, table sass1112_20161012001_t1n, https://nces.ed.gov/surveys/sass/tables_list.asp)
  So: A minutes trend over the same period already exists for grades one to four but not for kindergarten, so part of this issue is answered one grade up and the kindergarten estimate is the real gap. The table title does not name the four subjects, so that needs checking before use.
• Science instructional time declined in elementary schools over the accountability period, measured in time rather than in frequency categories. (Blank 2013, Science Education, doi:10.1002/sce.21078)
  So: The sign of the science decline is already corroborated outside ECLS-K in minutes terms, so a design that only reproduces the direction adds little; the contribution has to be the kindergarten-specific magnitude.
• Elementary instructional time shifted toward tested subjects and away from specific other subjects after No Child Left Behind, reported as changes in minutes. (Center on Education Policy summary, Arts Education Policy Review 2008, doi:10.3200/AEPR.109.6.23-28)
  So: Gives a benchmark magnitude in minutes for the elementary grades against which any kindergarten estimate can be judged plausible. The Crossref record for this item carries no author field, so the attribution should be confirmed against the journal before citing.
• A fourth cohort, the Early Childhood Longitudinal Study Kindergarten Class of 2023 to 2024, collected nationally representative kindergarten teacher data in 2023 and 2024, with the first data file release planned for early 2026. (National Center for Education Statistics, ECLS-K 2024 programme pages and blog, 2024 to 2026, https://nces.ed.gov/learn/blog/celebrating-ecls-k-2024-learning-about-our-nations-teachers-principals-and-schools)
  So: A third comparison point is arriving, so the calibration is worth building now and applying to the new questionnaire; otherwise the 2024 wave inherits the same non-comparability the authors flagged in 2016.

Searched: Searched Crossref for the teacher-log and classroom-observation measurement literature, the Center on Education Policy instructional time work and the elementary science time literature; checked the National Center for Education Statistics Schools and Staffing Survey table list directly to confirm the grade range and years of the hours-per-week series; and searched the web for the existence and status of a third ECLS-K cohort. I did not find any study that reconciles the 1998 and 2010 ECLS-K time-use wordings, and I did not find a kindergarten minutes-per-subject series for the United States.

Feasibility

A survey methodology group with access to a standing teacher panel, for example a federal survey contractor, a state education agency or a research organisation that maintains a national teacher panel. · about 12 months for the bridge study, three years or more for the panel part · moderate, because it needs primary collection from teachers and paid observer days, though no equipment and no laboratory
Blockers: Teacher burden and district approval for five-day logs plus observer visits; Current teachers cannot be asked to reproduce 1998 or 2010 conditions, so the bridge assumes the wording effect is stable over time; Whether the ECLS-K 2024 teacher questionnaire repeats either earlier wording is not yet known; Attrition in the panel part, which in a teacher workforce over three years is substantial

Strongest objection

A mapping calibrated on teachers working in 2026 cannot be assumed to hold for respondents in 1998, because the meaning of the terms has moved with practice. A teacher in 1998 and a teacher today may both report one hour of child-selected activity and mean different things by it. The bridge would then transport a present-day mapping onto historical marginals and produce minutes estimates with false precision, which is worse than the authors' decision to leave the comparison undone.

Not derivable from what is published

• Between-teacher standard deviation of logged minutes per subject per week in kindergarten
• Test-retest reliability of either item wording
• Whether the ECLS-K 2024 teacher questionnaire uses the 1998 wording, the 2010 wording or a third
• Whether any teachers or schools appear in both ECLS-K samples, which would permit a limited twelve-year within-unit comparison
• The four subjects covered by the Schools and Staffing Survey hours series, which the table title does not name

Two robustness worries the authors raise themselves: that the Table 6 cohort-by-school-type interactions are reported on the odds metric and can look different on a probability metric, where they show one case that loses significance, and that Footnote 3 excludes the first-grade teachers of children held back in kindergarten from the comparison that answers the title question.

Question

Do the paper's claims about widening and narrowing school-type gaps, and about kindergarten in 2010 against first grade in 1999, survive a change of metric and the inclusion or bounding of the excluded first-grade teachers?

Hypothesis

Metric choice changes the sign or significance for a minority of the fifteen outcomes, leaving the belief and textbook claims intact but not the activity-centre claims, and the retention exclusion moves the first-grade testing figure too little to overturn the kindergarten-versus-first-grade comparison.

Attempt type

computational
Vocabulary note: This is a pre-registered reanalysis of an existing dataset under several model specifications. The vocabulary has no value for reanalysis, robustness replication or a multiverse analysis, and 'computational' here means only that the work happens in a statistics package.

Design

A pre-registered multiverse reanalysis. For each of the fifteen Table 6 outcomes and both school-type definitions, free or reduced-price lunch quartile and non-White quartile, estimate the cohort-by-school-type interaction four ways: as published on the log-odds scale; as average marginal effects on the probability scale with delta-method standard errors, which is the Ai and Norton correction; as a linear probability model with school-clustered standard errors; and as a log-binomial or Poisson model giving risk ratios. Report all four side by side under a rule fixed in advance, that a gap-widened claim counts as supported only if the interaction is significant with the same sign on both the odds and the probability scale, with a stated correction for the thirty tests. For the retention branch, re-run the kindergarten 2010 against first grade 1999 standardized-testing comparison as published, then with the first-grade teachers of retained children included if the restricted files hold them, and in any case as a bounding exercise recomputing the 1999 marginals under the extreme assumptions that every excluded teacher never tested or tested at least weekly, bounding the 11 percent figure and the authors' 2.6 times statement.

Controls

• internal-positive The outcomes with the largest unambiguous cohort main effects, above all the rise from 31 percent to 80 percent of teachers agreeing most children should learn to read in kindergarten, must show a significant cohort effect under every one of the four specifications.
  Rules out: That a specification which finds nothing is doing so because it is mis-implemented rather than because the interaction is metric-dependent.
• designated Exact reproduction of the published Table 6 cells on the odds metric before any metric is changed. This doubles as a transcription check: in the published PDF as I extracted it, the 1998 high free or reduced-price lunch row carries negative values in the classroom space and art and music columns, for example -4.08 and -3.29, which cannot be odds.
  Rules out: That differences between metrics are really differences between the reanalysis and the original estimation sample, weights or specification, or an artefact of a mis-set table.
  Vocabulary note: A replication target standing in for a known-good reference sample, which the vocabulary does not provide for.
• background The two extreme-assumption bounds on the first-grade 1999 standardized-testing marginals for the excluded teachers of retained children.
  Rules out: That the kindergarten-versus-first-grade conclusion depends on an untestable assumption about who was left out, since a conclusion that holds at both bounds does not depend on it at all.
  Vocabulary note: A worst-case bound rather than a background measurement.

Sensitivity

Partly derivable, unusually for this set. The paper supplies one probability-metric difference-in-differences it could not detect: the share of teachers who thought knowing the alphabet before kindergarten was important rose by .37 in high free or reduced-price lunch schools against .33 in lower-poverty schools, a four percentage point difference-in-differences that the authors report as not statistically significant on a sample of between 3,700 and 4,000 teachers, rounded to the nearest 50 under National Center for Education Statistics disclosure rules. A reanalysis that aims to adjudicate metric dependence therefore needs to detect a difference-in-differences of about four percentage points. Whether it can is not derivable from the paper, because the intraclass correlation within schools and the number of sampled schools are not reported, and both are needed to turn 3,700 teachers into an effective sample size. The design should also note that thirty interaction tests are reported with no multiplicity correction, so some of the metric dependence it finds will be sampling noise.

Would count as

Supported → partial · opposed → refuted
Inconclusive if: The published odds-metric cells cannot be reproduced from the data, whether because the table is mis-set or because the original specification cannot be recovered. Metric differences then cannot be attributed to the metric, which is inconclusive-controls-failed rather than a robustness verdict.

Already done

• In a nonlinear model the interaction effect is not the coefficient on the interaction term, and computing it correctly requires the cross-partial derivative. (Ai and Norton 2003, Economics Letters, doi:10.1016/S0165-1765(03)00032-6)
  So: The correction this reanalysis implements was already thirteen years old when the paper appeared, so the work is a reanalysis of a known problem and should be presented that way rather than as a new method.
• Odds ratios depend on the amount of unexplained variance in the model, so comparing odds ratios across groups or across time conflates the effect of interest with differences in residual variation. (Norton and Dowd 2017, Health Services Research, doi:10.1111/1475-6773.12712)
  So: Gives the precise reason a 1998 against 2010 comparison of odds can diverge from the same comparison in probabilities, and the reason to make average marginal effects the primary estimate.
• Practical guidance on computing and reporting interaction effects in logit models, with worked cases. (Norton 2012, NBER working paper 18252, Log Odds and Ends, doi:10.3386/w18252)
  So: Supplies the reporting template, which matters here because the contribution is a table of four specifications rather than a new estimate.
• Income and racial school readiness gaps at kindergarten entry over the same two cohorts, estimated on a continuous achievement metric where no odds-versus-probability choice arises. (Reardon and Portilla 2016, AERA Open, doi:10.1177/2332858416657343)
  So: An external consistency check: a metric-robust reanalysis of whether high-poverty and high non-White schools converged over this period should not contradict a continuous-outcome estimate from the same two cohorts.

Searched: Searched Crossref for the logit interaction methodology literature, which is settled and well cited, and for the prevalence of kindergarten retention in the national longitudinal studies, where I found only dataset records and no usable rate. I did not find any published reanalysis of this paper's Table 6 on a probability metric, and I did not find any treatment of the retained-children exclusion in the kindergarten-versus-first-grade comparison.

Feasibility

Any quantitative methods group with a National Center for Education Statistics restricted-use licence, or the original authors, who already hold the estimation code. · 3 to 6 months · low, since it is a reanalysis of data already collected and needs no new instrument
Blockers: Restricted-use licence and disclosure review; Some published Table 6 cells appear inconsistent with an odds metric, so the replication target may have to come from the authors rather than the paper; Footnote 3 says the ECLS-K design does not include the first-grade teachers of children retained in kindergarten, so the inclusion arm may be impossible and only the bounds available

Strongest objection

A specialist would say this re-litigates a caveat the authors already stated and partly answered. With a binary outcome and very large cohort main effects, the odds-metric interaction answers a legitimate question about relative change, the paper reports a probability-metric check in the text, and the substantive conclusions are carried by main effects of twenty to forty nine percentage points rather than by the interactions. On the retention branch the objection is stronger still: if the design never collected those teachers, no reanalysis can add them, and a bounding exercise on an exclusion the authors already flagged may be the whole available answer.

Not derivable from what is published

• The share of children in either cohort retained in kindergarten, and therefore the size of the excluded group of first-grade teachers
• Intraclass correlation within schools for any of the fifteen Table 6 outcomes
• The number of sampled schools behind the 3,700 to 4,000 teachers
• The true Table 6 cell values, since the published PDF as extracted contains negative values in rows that should hold odds
• Whether the restricted files contain any first-grade teacher data for children retained in kindergarten