id: an-051 hypothesis: methodology-flexibility-a type: descriptive status: done status_date: 2026-06-02 confidence: green created: 2026-06-02 script: source/llm/pilot_questionario.py + source/analysis/an-051-questionnaire-rotation-by-sponsor.py target: build/table/an-051-questionnaire-rotation-by-sponsor.csv headline: Sponsored polls dramatically under-document candidate-name rotation in the questionnaire (5.4 % vs 26.1 % when rotation+random+rotated combined; McNemar p < 10⁻⁷). Sponsored polls file FEWER vote-intention scenarios on average (3.73 vs 4.49, Wilcoxon p = 0.002) — refutes the within-poll scenario-selection / cherry-pick hypothesis. [2026-06-02 update — see AN-052/AN-053:] The rotation contrast is a between-firm composition effect, not a within-firm sponsor-choice effect (AN-052 firm-FE LPM p = 0.37). The direct candidate-position priming test refutes the sponsor-exploitation reading (AN-053: sponsored polls list sponsor's candidate LATER, McNemar p = 0.001 in WRONG direction). The AN-051 finding is recontextualized as a firm-tier composition signal — sponsors choose low-discipline firms that don't document rotation — rather than a within-firm Channel-A lever. design: sample: 241 sponsored × independent pairs (of 244 in curated_pairs) with both questionnaires successfully extracted specification: within-pair McNemar tests on binary fields (name_rotation_documented, approval_question_present, rejection_question_present, nonresponse_instruction_present); paired Wilcoxon on n_vote_intention_scenarios; bias-carrier Mann-Whitney on differing-rotation subset comparator: independent poll within the matched pair notes: First quantitative output of the new questionario_pesquisa LLM extractor (PollQuestionario schema). Pilot was on 338 protocols (the curated-pair universe), $1-2 cost at gpt-4o-mini, 19 min wall.
AN-051: Does questionnaire-side name rotation and scenario filing differ between sponsored and matched independent polls?
Question
First quantitative output of the new poll_questionario LLM extractor.
Sources two probes from the source-of-bias agenda that the registration
narrative cannot reach: (1) within-poll scenario selection (cherry-pick
hypothesis — does a sponsored poll file multiple estimulada scenarios
and report the most favorable?), (2) name-order priming (do sponsored
polls document randomization of candidate-name order across respondents,
which neutralizes the documented 2-3 pp name-order effect?).
Design
Source: 244 curated sponsored × independent pairs in
build/llm/curated_pairs/pairs_with_extractions.parquet. New
PollQuestionario extraction ran on the 338 unique protocols across
both sides (build/llm/questionario_pilot/extractions.parquet); 335
valid extractions, 3 image-only PDFs skipped. The 241 pairs with both
sides successfully extracted form the analysis sample.
Schema fields tested:
name_rotation_documented(bool) — does the questionnaire explicitly instruct the interviewer to rotate candidate name order across respondents?headline_candidate_order—by_listed(fixed printed order, the default) vsrandom/rotated(explicit randomization) vsalphabetical/by_number/by_party(rules that constrain order without randomizing).n_vote_intention_scenarios— count of distinctestimulada+espontâneablocks (the cherry-pick diagnostic).approval_question_present/rejection_question_present— framing-level diagnostics.nonresponse_instruction_present— does the questionnaire print a rule for handling undecided / refusal / blank / null?
Tests:
- Marginals per field, sponsored side vs independent side.
- Paired McNemar on the binary fields.
- Paired Wilcoxon signed-rank on n_vote_intention_scenarios.
- Bias-carrier MW on the differing-rotation subset.
- Between-pair MW asking whether sponsored polls that do document rotation have lower within-pair bias than those that don't.
Results
Headline marginals (n = 241 pairs):
| Field | Sponsored | Independent | Δ | McNemar p |
|---|---|---|---|---|
name_rotation_documented |
5.4 % | 20.3 % | −14.9 pp | < 10⁻⁵ |
| Headline order ∈ {random, rotated} | 2.5 % | 18.3 % | −15.8 pp | — |
| Combined randomization (any of the above) | 5.4 % | 26.1 % | −20.7 pp | ≈ 4 × 10⁻⁸ |
approval_question_present |
26.1 % | 32.8 % | −6.6 pp | 0.089 |
rejection_question_present |
50.2 % | 51.9 % | −1.7 pp | 0.78 |
nonresponse_instruction_present |
1.2 % | 1.7 % | −0.4 pp | 1.0 |
asks_party_first |
0 % | 0 % | 0 | — |
The combined randomization contrast is the sharpest within-pair methodology asymmetry surfaced in any source-of-bias probe to date. Discordant cells: b = 12 (sp-only) vs c = 62 (ind-only) — sponsored polls are 5× less likely to document name rotation than the matched independent.
Scenario filing — refutes cherry-pick:
| Statistic | Sponsored | Independent |
|---|---|---|
| Mean n_vote_intention_scenarios | 3.73 | 4.49 |
| Median | 3 | 4 |
| 75th pct | 5 | 6 |
| Max | 12 | 14 |
- Paired Wilcoxon:
sp − indmean = −0.76, p = 0.0019. - 44 % of pairs have sp < ind; 37 % sp > ind; 18 % equal.
- Among 6 pairs where sponsored has ≥ 8 scenarios: mean bias contrast +4.48 pp (vs overall +5.46 pp) — no high-tail cherry-pick story.
The cherry-pick / scenario-selection hypothesis is refuted on this sample: sponsored polls file fewer vote-intention scenarios on average, not more.
Bias-carrier test (the lever-mechanism question):
| Subset | n | Mean contrast (sp_error − ind_error) | MW p (vs other) |
|---|---|---|---|
| Pairs where sponsored documents rotation | 13 | +6.97 pp | 0.54 |
| Pairs where sponsored does NOT document rotation | 228 | +5.38 pp | — |
| Differing-rotation subset (sp-only) | 12 | +7.34 pp | 0.33 |
| Differing-rotation subset (ind-only) | 48 | +3.98 pp | — |
The bias contrast does not decrease when the sponsored side documents rotation — direction is the opposite (more bias when rotation is documented). The within-pair carrier test is also null. Rotation status appears to be a within-pair signal without being a within-pair carrier of the slant.
Interpretation
2026-06-02 update. The two natural follow-ups (AN-052 and AN-053) have walked back the "first sharp Channel-A signal" framing below. AN-052 shows the contrast vanishes within firm (firm-FE LPM p = 0.37; 19 of 20 both-side firms have identical rotation rates). AN-053 shows sponsored polls list the sponsor's candidate later, not earlier (McNemar p = 0.001 in the wrong direction). The rotation gap is a firm-tier composition signal — sponsors choose low-discipline firms that don't document rotation — and not a within-firm sponsor lever. The cherry-pick refutation (sp fewer scenarios than ind, Wilcoxon p = 0.002) stands. Original interpretation preserved below for the audit trail.
This is the first source-of-bias probe to land a sharp Channel-A within-pair signal. Sponsored polls dramatically under-document candidate-name rotation in the questionnaire — by a factor of 4-5×. The lever is real (the name-order-priming literature documents a 2-3 pp effect of listing a candidate first), the direction is right, and the McNemar significance is extreme (p ≈ 4 × 10⁻⁸ on the combined randomization indicator).
But the carrier test is null, repeating the AN-042 selective-disclosure pattern: the documentation contrast is loud and the within-pair contrast doesn't track the slant. Two readings:
Selective disclosure of methodology that doesn't carry bias. Sponsored polls may genuinely use fixed candidate order while independents randomize, but the resulting name-order priming doesn't dominate the slant — the +5-7 pp comes mostly from sample-shape levers (population frame, coverage class) that the AN-019 / AN-022 evidence pointed at.
Underpowered carrier test with directional but noisy contrast. With only 60 differing-rotation pairs and 12 sp-only sponsored documenting, the bias-carrier test is statistically thin. The +3.98 pp (when independent rotates but sponsored doesn't) vs +7.34 pp (when sponsored rotates but independent doesn't) is directionally consistent with "no-rotation → more bias" but MW p = 0.33.
The cherry-pick / scenario-selection hypothesis is refuted on this sample. Sponsored polls file fewer scenarios than matched independents, not more (Wilcoxon p = 0.002 in the wrong direction for cherry-pick). This is consistent with two readings: sponsored polls have already locked in their preferred scenario at sample design, so they don't need to file alternatives; OR sponsored polls under-document multi-scenario filings the same way they under- document rotation. Either way, the cherry-pick lever is not the carrier.
The combined picture is now: of the six concrete-design candidates in the source-of-bias agenda, AN-019 (coverage class, +2 pp), AN-022 (methodology completeness, t-rejection), the +4.6 pp population-frame interaction, and now AN-051 (name rotation, sp-vs-ind 5.4 % vs 26.1 %) are all directionally consistent with Channel A but each individually under-carry the +7 pp headline. The size-mismatch problem persists; the agenda has identified a multi-lever Channel-A pattern rather than a single quantitative carrier.
Follow-ups
Pollster fixed effects on the rotation contrast (puzzle, high priority). Independent polls that document rotation have higher mean bias than independents that don't (+1.95 vs −0.85 pp). This suggests rotation status correlates with firm characteristics — potentially firm-size: large pollsters (Datafolha, IBOPE, QUAEST) may use boilerplate that prints rotation instructions regardless of sponsor, and sponsored polls in this sample concentrate on smaller firms. A within-firm refit (restrict to the 31 AN-016 firms with ≥ 5 sponsored polls) would tell us whether the rotation gap is a sponsor- choice effect or a firm-size effect. Suggested script:
source/analysis/an-NNN-rotation-within-firm.py.Universe-scale rotation extraction (extension). Confirm the 5.4 % vs 26.1 % gap at full universe. The questionnaire extractor is already wired; running on the ~14k mayoral poll universe is ~$50-100 at gpt-4o-mini. Would give a regression- grade test with statistical power for the carrier hypothesis.
Candidate position of the sponsor's candidate (extension, the direct name-order test). The current AN tests whether rotation is documented; the sharper question is whether the sponsor's own candidate appears earlier in the printed roster than chance would predict. Requires joining the sponsor identity from the curated_pairs frame to the
candidates_listedroster of the headline scenario. ~30 min of script work. Suggested script:source/analysis/an-NNN-candidate-position-by-sponsor.py.AN-051 finding into paper §5 (writeup task). The new "Size mismatch and ruled-out levers" paragraph in paper.tex currently lists deferral / mode / interviewer / nonresponse as ruled-out concrete-design levers. Add candidate-name rotation as the FIRST lever-confirmed-but-not-quantitatively-carrying finding from the questionnaire layer. Bumps the running multi- lever count.
Promote
PollQuestionarioschema to the politica pipeline (infra). The schema and extractor live inpipelines/politica/source/llm/. If we want to run on the ~14k poll universe, add it to the production extraction orchestrator alongside the methodology / coverage / operations / sampling tasks.