Abhisek Behera PRO

Thanks, I like both of your ideas.

The two-layer label idea is good. One field for the guessed cause (from error patterns), and a separate field for the confirmed cause (checked by a human). This way, nobody confuses a guess with a verified fact. I will use this structure.

The visibility rule is also a good idea. The record itself should say what the agent is allowed to see: the goal, the Python version, the resolver output, the package list, and the wheel info. And what it must not see: anything from the smoke test. With this rule written down, the test is fair and nobody can argue about cheating.

About the competition: it is an interesting idea. One thing worth mentioning: I designed a final test set with 9 new packages, but I have not run it yet. So the results do not exist anywhere. No model or bot could have seen them. That kind of unseen data is what a fair challenge would need.

The task you described makes sense: predict pass or fail, guess the cause, suggest a fix, and score against the real result.

Let me think about all this. Thanks again for the detailed suggestions.

This is a very useful list, thank you. Let me sort it into what the dataset already has and what it doesn't.

Already in v1.0.0:

• Tool and environment details: exact uv version and exact CPython build are recorded per experiment (in runtime). One caveat: the environment is fixed by design, everything ran on Linux x86_64, so the OS fields never vary. They confirm the setup rather than describe differences. Expanding to more OS types (Windows, macOS) is something I may explore in the future, and the schema already has the fields ready for it.
• The exact input spec: both packages, versions, Python version (in spec).
• The full resolved package graph, not just the two headline packages (in installed_environment).
• Every stage's exact command, exit code, stdout/stderr, plus a timeout flag and peak memory (in stages).
• Retry count, normalized error text next to the raw output, and full wheel provenance: ABI/platform tags, sha256, source URL.

Not covered today, but I will try to include these in a future v2 release:

• The smoke-test code snippet itself (only its output is captured)
• Explicit timeout and resource limit values
• Confirming a failure reproduces across retries
• A "likely cause" label: ABI mismatch, removed symbol, missing optional dependency, build backend, platform wheel

Your likely-cause idea is the most interesting one to me. Since all raw tracebacks are preserved, labels like that could in principle be derived from the existing records without rerunning anything.

The benchmark framing also makes sense: an agent proposes a requirements change and gets scored against known real outcomes before anything is installed. That's much closer to real developer work than a synthetic task. Thanks for laying it out so clearly.

Well said. That's exactly why every record here keeps the full environment state: the resolved package set, installed wheels with ABI/platform tags, lockfile hash, and per-stage stdout/stderr.

One use I'd love to see: as an eval or reward signal for coding agents. An agent proposes a requirements change, and you score it against known real outcomes without installing anything. The 3,049 resolved-but-broken cases are the interesting ones, since they look fine to any metadata-only check.

Our own DepLab agent already works this way: models trained on these outcomes, with resolver facts kept separate from ML risk. Happy to hear what fields would make the dataset more useful for agent workflows.

Thank you for doing this properly. You are right on every point, and I appreciate that you verified the tallies reproduce before critiquing the framing.

I agree the full-set accuracy mixes two tasks. Predicting resolution failures that uv already reports is the easy part, and the 90.0% headline benefits from the validation split being 79.1% resolution failures. The number that matters for the dataset's core claim is the resolved-only subset, and I should have reported it separately from the start. I will update the dataset card to show resolved-only metrics beside full-set metrics.

You are also right that the sealed split is 4 pairs, not 6 packages crossed, and that all 79 post-resolve failures come from the celery/kombu/billiard family. So the honest statement is: the post-resolve signal was validated on one unseen family, not on unseen packages broadly. I will correct that wording too.

Your question, which development pairs produce the most import failures: pandas+seaborn is the largest (354 import failures plus 92 smoke-test failures out of 588 resolved rows, 75.9% post-resolve failure rate), then matplotlib+seaborn (264 import failures, 51.0%), flask+jinja2 (238, 46.1%), and numpy+opencv-python (196, where every resolved row fails import). Overall the development resolved-only subset is 15,080 rows: 12,031 pass, 2,448 import failures, 601 smoke-test failures, so always-pass scores 79.8% there.

Two pairs are degenerate in the other direction: fastapi+starlette (all 338 resolved rows fail the smoke test) and numpy+opencv-python (all 196 resolved rows fail import). A constant predictor is unbeatable on those, so they would make poor holdouts too. The most informative holdout candidates are the mixed pairs where neither constant baseline wins: matplotlib+seaborn (300 pass vs 312 post-resolve failures, almost exactly balanced), flask+jinja2 (291 vs 249), and pandas+seaborn (142 vs 446).

For the next release: (1) resolved-only accuracy reported beside full-set accuracy, (2) a re-cut sealed split holding out at least one mixed family, likely matplotlib+seaborn, and (3) the reserved 9-package final test set run with that design. If you see a better holdout choice in these numbers, I am genuinely interested.

Thanks John, this is a great workflow, especially the part about having the model summarize what did and did not work, then feeding those summaries back in as context. That "what broke and what fixed it" record is basically the signal my dataset tries to capture, just at the dependency-upgrade level. Drift really is the quiet killer. Appreciate you sharing the prompt link too, I'll take a look.

Nice!

Drift is always, always really troublesome. If there were no drift, Python would be heaven. Well, it does have weaknesses like speed, GIL constraints, and indentation, but...

It’s not that I’m seriously working on it as research; it’s just a bit of everyday wisdom, but when I debug Spaces, if they are meant to be open/public, I first feed the code into ChatGPT. Then I deliberately debug it within the chat, whether by vibe coding or not, and at the end I have it summarize what didn’t work and what made it work.

Then, if I feed the generated file into ChatGPT and use it as-is, or feed several of them in together, add context, and have it summarize them into another Markdown document before attaching it, it tends to refer to them fairly well.😅

Nice!

Drift is always, always really troublesome. If there were no drift, Python would be heaven. Well, it does have weaknesses like speed, GIL constraints, and indentation, but...

It’s not that I’m seriously working on it as research; it’s just a bit of everyday wisdom, but when I debug Spaces, if they are meant to be open/public, I first feed the code into ChatGPT. Then I deliberately debug it within the chat, whether by vibe coding or not, and at the end I have it summarize what didn’t work and what made it work.

Then, if I feed the generated file into ChatGPT and use it as-is, or feed several of them in together, add context, and have it summarize them into another Markdown document before attaching it, it tends to refer to them fairly well.😅

Good question. It is for training and evaluating models that fix Python code after a dependency upgrade breaks it. If you have ever bumped numpy or pandas and watched old code stop working, this is the data to teach a model to make those fixes automatically. Each example pairs the broken code with the fix and a short note on the API change that caused it. There is a live demo in the collection if you want to see it fix real code.