Update src/tasks_content.py
Browse files- src/tasks_content.py +4 -4
src/tasks_content.py
CHANGED
|
@@ -24,11 +24,11 @@ TASKS_DESCRIPTIONS = {
|
|
| 24 |
|
| 25 |
"ci_builds_repair": """# CI builds repair\n
|
| 26 |
|
| 27 |
-
Our CI
|
| 28 |
-
includes manually curated and assessed
|
| 29 |
|
| 30 |
-
We use Pass@1 metric for CI repair.
|
| 31 |
-
Models can be evaluated in three
|
| 32 |
* `full` β **no** ground truth diffs are used for model evaluation;
|
| 33 |
* `oracle: files` β ground truth diffs are used to select files that should be corrected to fix the issue;
|
| 34 |
* `oracle: files, lines` β ground truth diffs are used to select files and code blocks that should be corrected to fix the issue;
|
|
|
|
| 24 |
|
| 25 |
"ci_builds_repair": """# CI builds repair\n
|
| 26 |
|
| 27 |
+
Our CI builds repair benchmark π€ [JetBrains-Research/lca-ci-builds-repair](https://huggingface.co/datasets/JetBrains-Research/lca-ci-builds-repair)
|
| 28 |
+
includes 77 manually curated and assessed data points coming from 32 Python repositories, which are used to make a model fix a failed build.
|
| 29 |
|
| 30 |
+
We use the `Pass@1` metric for CI repair.
|
| 31 |
+
Models can be evaluated in three types of tasks:
|
| 32 |
* `full` β **no** ground truth diffs are used for model evaluation;
|
| 33 |
* `oracle: files` β ground truth diffs are used to select files that should be corrected to fix the issue;
|
| 34 |
* `oracle: files, lines` β ground truth diffs are used to select files and code blocks that should be corrected to fix the issue;
|