A research question before a metric
In our research manuscript, Exploring Power Creep in Magic: The Gathering, we asked whether newer cards increasingly outperform older ones. The term power creep is easy to recognize in player discussions, but difficult to operationalize: a card can be powerful because of its raw statistics, rules, interactions with other cards, or place in the competitive environment.
I led the project and am a co-first author. I defined the research question, designed the experimental structure, assigned the work across the team, and synthesized what the evidence could support. The original report was developed in November–December 2025; the manuscript has not been submitted.
The most useful lesson for me is about measurement. Before combining more data sources, we need to ask whether they measure the same construct, and whether their errors are genuinely different.
The experimental structure I led
I organized the project around a question that no single convenient metric could settle. Raw statistics make comparisons simple but miss rules and synergy. Simulation adds game context but depends on the policy and scenarios. LLM analysis makes rules text tractable at scale but creates an annotation-validity problem. Tournament decks reflect real selection but introduce availability and metagame effects.
I divided the work into these complementary strands and brought the results back to the same research question. The goal was to understand what each view added, and where its interpretation stopped. A large simulation count or an attractive trend line was not, by itself, an answer about card strength.
This structure also made a practical tradeoff explicit. Full-deck simulations would have been costly and sensitive to the engine’s skill at particular strategies. We used shorter scenarios to obtain broad coverage, while treating deck-level effects as a limitation of that design. Choosing a tractable experiment and stating what it measures was central to my role.
Four ways to observe card power
We combined four approaches. Each answers a related but distinct question.
| Approach | What we measured | Main limitation |
|---|---|---|
| Numerical card analysis | Creature power, toughness, mana cost, and keyword counts | Misses many effects, interactions, and context |
| Scenario simulation | Changes in an engine’s state score after a card is played | Depends on the scenarios, horizon, policy, and scoring function |
| LLM-based rules analysis | Features inferred from card rules, including flexibility and resource generation | Depends on the model, prompt, and annotation validity |
| Tournament analysis | Age composition of cards in winning Modern-format decks | Reflects selection, availability, and the competitive environment as well as strength |
The numerical analysis used Scryfall card records, with filters appropriate to each metric. For example, a creature efficiency measure based on power plus toughness divided by mana cost excluded zero-cost and nonnumeric cases. The numerical analysis used original printings to avoid counting reprints as new designs.
For simulation, the report describes approximately 30,000 cards assessed in ten predefined scenarios using XMage. We used a short, two-turn horizon and a state-scoring heuristic instead of attempting a large tournament of complete decks. That made the study tractable, but changed what the score represented.
For rules analysis, we used Gemini 2.5 Flash to extract features from a subset of approximately 7,000 cards spanning release years and card types. The tournament analysis used winning Modern deck lists from 2011–2025, drawing on MTGTop8.
These figures describe the project report. I have not rerun those data collection or evaluation pipelines for this blog article.
Complementary evidence is not independent evidence
The report found broad upward trends in several numerical and simulated measures. The rules-text analysis was more mixed: card advantage and flexibility had clearer patterns than resource generation and risk. That variation matters. It prevents the result from becoming the simple claim that every dimension of card power increased in the same way.
For a concrete example, the report’s appendix gives rounded mean simulation scores of about 1,200 for cards released in 1993 and 2,200 for cards released in 2025. These are units of the chosen state-scoring heuristic. They are not win rates, and their ratio is not a statement that newer cards are nearly twice as strong. The observation is a trend under the specified scenarios and evaluation function.
The approaches also share information. A simulator’s scoring heuristic can directly reward power, toughness, or mana-related properties, so its agreement with a numerical metric is not an entirely independent replication. Tournament popularity and LLM judgments can both reflect recognizable card reputation. Four views of the same dataset do not automatically provide four independent confirmations.
Today I would make this dependence explicit in the experimental design. I would identify what each method can contribute beyond the others, then test where the methods disagree. A card that scores highly by one measure and poorly by another is useful evidence about the construct being measured, rather than merely an inconvenient example.
A card can have more than one age
One consequential detail is the definition of card age. The tournament section defines it using the most recent printing available at the time of the tournament:
That is not the same as the age of the card’s original design. An old card reprinted recently can appear young under this definition. A trend toward recently printed cards may therefore capture reprint policy and availability as well as a preference for newly designed, more powerful cards.
The numerical analysis and tournament analysis consequently use different temporal concepts. This does not make either variable useless, but it changes the interpretation. For a stronger follow-up, I would report both original-design age and latest-printing age, match reprints to a stable card identity, and examine whether the conclusion survives both definitions.
I would also stratify by card type and format. Changes in the mixture of card types, the number of releases, and format rules can shift an aggregate trend without every category changing in the same direction. A descriptive association is a starting point for these checks, not a causal explanation of why a publisher or player made a decision.
Treat LLM features as measurements
An LLM can make rules text easier to analyze, but its output needs validation just like any other measurement instrument. In the report, adding examples to the prompt appeared helpful. That anecdotal observation is not a calibrated estimate of annotation accuracy.
I would now set aside a stratified set of cards for human annotation, define a rubric for each feature, and compare agreement across release eras and card types. I would include difficult cases and repeated model judgments. A feature that is easy to annotate on recent, verbose cards but harder on older wording can create a temporal trend partly through measurement error.
Another useful control would remove card names and release information where possible. If the goal is to extract properties from rules text, recognition of a famous card should not silently become the label. This experiment would distinguish text-grounded annotation from a mixture of annotation and remembered reputation.
What a short simulation can support
Two turns in ten scenarios do not represent the space of possible decks and games. A short horizon can favor immediate effects over delayed or synergistic ones. The engine’s policy may also handle some mechanics better than others. A simulator can enforce game rules correctly while still providing a limited measure of strategic strength.
The next useful experiment would vary the scenario distribution, horizon, and scoring heuristic, then inspect which conclusions remain stable. I would add a modest number of deck-level evaluations as a check, rather than assuming that a scalar card score composes into full-deck performance.
The result I want is a scoped answer: which aspects of card design changed, in which populations, under which measurement choices? That is a more defensible research outcome than treating “power” as a single ground-truth label. The same question appears in language-model evaluation, where an accessible proxy can be mistaken for the capability it was intended to measure.
Manuscript and background
Exploring Power Creep in Magic: The Gathering — unsubmitted research manuscript. Authors listed in the original report: Beck Thompson, Suqin Zhang, Sayuj Raj Shahi, Denis Gavriloff, Aiden Katsuragawa, and Jay Dharmadhikari. My role: project lead and co-first author; research framing, experimental design, task allocation, and evidence interpretation.
The team’s methods and observations above are summarized from the manuscript. The proposed validation improvements are my present retrospective analysis; they are not additional experiments claimed by the manuscript.
- Scryfall bulk card data.
- XMage source and documentation.
- MTGTop8 and EDHREC for the deck-data context.
- Daniel Sumner Magruder, A Conservative Metric of Power Creep, a related effort to formalize a conservative measure.