Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate APPS benchmarking #1051

Merged
merged 24 commits into from
Mar 23, 2024
Merged

Integrate APPS benchmarking #1051

merged 24 commits into from
Mar 23, 2024

Commits on Feb 20, 2024

  1. Initial APPS benchmark integration 

    Benchmark against three simple problems
    @azrv
    azrv committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    29f4680 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. Download and cache dataset 

    Iterate over predefined set of problems
TODOs left
    @azrv
    azrv committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    4ddfb5e View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. WIP APPS benchmarking 

    - try using command line arguments
- integrate self-healing
    @azrv
    azrv committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    cf3db0e View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. Merge remote-tracking branch 'origin/main' into apps_self_healing

    @azrv
    azrv committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    ccb00ee View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. Multiple inputs, results

    @azrv
    azrv committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    3c6c4aa View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Add subprocess execution timeout 

    Add parse_diffs timeout
Temporary except diffs related issues (Maybe some `try` statements are not necessary at this point)
Handle problems with starter_code
Improve results
    @azrv
    azrv committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    833e3c9 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2024

  1. Remove unused code

    @azrv
    azrv committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    5e54e5b View commit details
    Browse the repository at this point in the history

  2. Drop retry logic and outer loop in run.py 

    Drop self_healing_agent and benchmark_agent
    @azrv
    azrv committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    e804c0b View commit details
    Browse the repository at this point in the history

  3. Get rid of inputs 

    Store commands within assertions
    @azrv
    azrv committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    95c88f3 View commit details
    Browse the repository at this point in the history

  4. Get rid of inputs 

    Store commands within assertions
    @azrv
    azrv committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    5f6f7d6 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'apps2' into apps_self_healing

    @azrv
    azrv committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    c25c1d1 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Remove AppsAssertion class

    @azrv
    azrv committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    b6ecace View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into apps_self_healing

    @azrv
    azrv committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    a526a91 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2024

  1. Merge branch 'main' into apps_self_healing

    @azrv
    azrv committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    37df506 View commit details
    Browse the repository at this point in the history

  2. Add AppsAssertion class, simplify run.py

    @azrv
    azrv committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    cb0d9ca View commit details
    Browse the repository at this point in the history

  3. Return run.py to its initial state

    @azrv
    azrv committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    50cd771 View commit details
    Browse the repository at this point in the history

  4. Run pre-commit; Update lock file

    @azrv
    azrv committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    20890d3 View commit details
    Browse the repository at this point in the history

  5. Run pre-commit on all files

    @azrv
    azrv committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    befa702 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. Merge remote-tracking branch 'origin/main' into apps_self_healing

    @azrv
    azrv committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    adb60a2 View commit details
    Browse the repository at this point in the history

  2. Revert assertions back to using dicts

    @azrv
    azrv committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    450c7e2 View commit details
    Browse the repository at this point in the history

  3. Create new execution environment for every run to avoid side effects

    @azrv
    azrv committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    b175938 View commit details
    Browse the repository at this point in the history

  4. Clean up

    @azrv
    azrv committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    911e092 View commit details
    Browse the repository at this point in the history

  5. Revert DiffError; Temporary except RecursionError in benchmarks

    @azrv
    azrv committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    d380f05 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. Remove RecursionError try-catch

    @azrv
    azrv committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ce35057 View commit details
    Browse the repository at this point in the history