-
Notifications
You must be signed in to change notification settings - Fork 76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tests for remote examples #800
Merged
Merged
Changes from 25 commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
2536c94
🐛 Fixed bug with remoteFunction() not existing
Etesam913 025db41
✅ Added tests for remote_procedure template
Etesam913 c41c801
Merge branch 'add-cypress-to-static-task' of https:/faceb…
Etesam913 2607e70
✅ Added action for remote_procedure template
Etesam913 eae5377
✅ Added tests for toxicity_detection example
Etesam913 67e53c0
✏️ Added github action for toxicity_detection
Etesam913 6ece9f5
✏️ Fixed typo in name of workflow
Etesam913 d996e6a
🔥 Removed wait() statements
Etesam913 adb023d
🔥 Removed spinner assertion test
Etesam913 9895ca5
⏳ Added timeout
Etesam913 82c7f31
✅ Added tests for remote_procedure/mnist
Etesam913 10ae5f8
✏️ Added mnist job to github action
Etesam913 2ac5d62
➕ Added dependencies to github action
Etesam913 1d1c855
🔥 Removed linking from github action
Etesam913 21a1c4f
🔀 Merged with add-cypress-to-static-task
Etesam913 5fa8caf
Merge branch 'add-cypress-to-static-task' of https:/faceb…
Etesam913 2799908
✨ Added link script to all remote_examples
Etesam913 7fbe70a
⏱ Increased wait time for toxicity alert
Etesam913 cd7a9fa
🥅 Added exception handling for post_build script
Etesam913 169ef90
✅ Reformat to pass code-style
Etesam913 3c5ed24
🔀 Merged with main
Etesam913 494b384
🔀 Merged with main
Etesam913 b550ec3
✏️ Updated post_build_script to post_install_script
Etesam913 719f812
✅ Added an alert check for template
Etesam913 277003c
✨ Added drawing to mnist test
Etesam913 2f82350
✅ Changed 1 to 3 in mnist test for more consistency
Etesam913 93c6311
💡 Uncommented out submit button click
Etesam913 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,6 +10,7 @@ jobs: | |
steps: | ||
- name: 🔀 Checking out repo | ||
uses: actions/checkout@v2 | ||
|
||
- name: 🐍 Installing python | ||
uses: actions/setup-python@v2 | ||
|
||
|
@@ -29,8 +30,7 @@ jobs: | |
mkdir data | ||
|
||
- name: 📂 Set the data directory | ||
run: | | ||
mephisto config core.main_data_directory ~/mephisto/data | ||
run: mephisto config core.main_data_directory ~/mephisto/data | ||
|
||
- name: 📦 Setting up mephisto-task package | ||
run: | | ||
|
@@ -43,9 +43,154 @@ jobs: | |
uses: cypress-io/[email protected] | ||
with: | ||
install: false | ||
browser: chrome | ||
project: ./examples/static_react_task/webapp | ||
config-file: ./cypress.config.js | ||
start: python examples/static_react_task/run_task.py mephisto.task.post_install_script=link_mephisto_task.sh | ||
wait-on: "http://localhost:3000/?worker_id=x&assignment_id=1" | ||
command-prefix: yarn dlx | ||
headless: true | ||
|
||
remote_procedure_template: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: 🔀 Checking out repo | ||
uses: actions/checkout@v2 | ||
|
||
- name: 🐍 Installing python | ||
uses: actions/setup-python@v2 | ||
|
||
- name: 🪨 Setup Node | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 16 | ||
|
||
- name: 🤖 Install Mephisto | ||
run: pip install -e . | ||
|
||
- name: 🖋 Create data directory | ||
run: | | ||
cd ~ | ||
mkdir mephisto | ||
cd mephisto | ||
mkdir data | ||
- name: 📂 Set the data directory | ||
run: mephisto config core.main_data_directory ~/mephisto/data | ||
|
||
- name: 📦 Setting up mephisto-task package | ||
run: | | ||
cd packages/mephisto-task | ||
yarn install | ||
yarn build | ||
npm link | ||
|
||
- name: ⌛️ Running cypress tests | ||
uses: cypress-io/[email protected] | ||
with: | ||
install: false | ||
browser: chrome | ||
project: ./examples/remote_procedure/template/webapp | ||
config-file: ./cypress.config.js | ||
start: python examples/remote_procedure/template/run_task.py mephisto.task.post_install_script=link_mephisto_task.sh | ||
wait-on: "http://localhost:3000/?worker_id=x&assignment_id=1" | ||
command-prefix: yarn dlx | ||
headless: true | ||
|
||
remote_procedure_mnist: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: 🔀 Checking out repo | ||
uses: actions/checkout@v2 | ||
|
||
- name: 🐍 Installing python | ||
uses: actions/setup-python@v2 | ||
|
||
- name: 🪨 Setup Node | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 16 | ||
|
||
- name: 🤖 Install Mephisto | ||
run: | | ||
pip install -e . | ||
pip install torch pillow numpy | ||
|
||
- name: 🖋 Create data directory | ||
run: | | ||
cd ~ | ||
mkdir mephisto | ||
cd mephisto | ||
mkdir data | ||
|
||
- name: 📂 Set the data directory | ||
run: mephisto config core.main_data_directory ~/mephisto/data | ||
|
||
- name: 📦 Setting up mephisto-task package | ||
run: | | ||
cd packages/mephisto-task | ||
yarn install | ||
yarn build | ||
npm link | ||
|
||
- name: ⌛️ Running cypress tests | ||
uses: cypress-io/[email protected] | ||
with: | ||
install: false | ||
browser: chrome | ||
project: ./examples/remote_procedure/mnist/webapp | ||
config-file: ./cypress.config.js | ||
start: python examples/remote_procedure/mnist/run_task.py mephisto.task.post_install_script=link_mephisto_task.sh | ||
wait-on: "http://localhost:3000/?worker_id=x&assignment_id=1" | ||
command-prefix: yarn dlx | ||
headless: true | ||
|
||
remote_procedure_toxicity_detection: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: 🔀 Checking out repo | ||
uses: actions/checkout@v2 | ||
|
||
- name: 🐍 Installing python | ||
uses: actions/setup-python@v2 | ||
|
||
- name: 🪨 Setup Node | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: 16 | ||
|
||
- name: 🤖 Install Mephisto | ||
run: | | ||
pip install -e . | ||
pip install detoxify | ||
|
||
- name: 🖋 Create data directory | ||
run: | | ||
cd ~ | ||
mkdir mephisto | ||
cd mephisto | ||
mkdir data | ||
|
||
- name: 📂 Set the data directory | ||
run: mephisto config core.main_data_directory ~/mephisto/data | ||
|
||
- name: 📦 Setting up mephisto-task package | ||
run: | | ||
cd packages/mephisto-task | ||
yarn install | ||
yarn build | ||
npm link | ||
|
||
- name: ⌛️ Running cypress tests | ||
uses: cypress-io/[email protected] | ||
with: | ||
install: false | ||
browser: chrome | ||
project: ./examples/remote_procedure/toxicity_detection/webapp | ||
config-file: ./cypress.config.js | ||
start: python examples/remote_procedure/toxicity_detection/run_task.py mephisto.task.post_install_script=link_mephisto_task.sh | ||
wait-on: "http://localhost:3000/?worker_id=x&assignment_id=1" | ||
command-prefix: yarn dlx | ||
headless: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
module.exports = { | ||
video: false, | ||
|
||
e2e: { | ||
baseUrl: "http://localhost:3000/?worker_id=x&assignment_id=1", | ||
supportFile: false, | ||
}, | ||
}; |
87 changes: 87 additions & 0 deletions
87
examples/remote_procedure/mnist/webapp/cypress/e2e/remote_procedure_mnist.cy.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,87 @@ | ||
describe("Loads remote_procedure_mnist", () => { | ||
it("Makes request for agent", () => { | ||
cy.intercept({ pathname: "/request_agent" }).as("agentRequest"); | ||
cy.visit("/"); | ||
cy.wait("@agentRequest").then((interception) => { | ||
expect(interception.response.statusCode).to.eq(200); | ||
}); | ||
}); | ||
it("Loads correct react elements", () => { | ||
cy.visit("/"); | ||
cy.get('[data-cy="canvas-container-0"]'); | ||
cy.get('[data-cy="clear-button-0"]'); | ||
cy.get('[data-cy="correct-checkbox-0"]'); | ||
cy.get('[data-cy="correct-text-input-0"]'); | ||
|
||
cy.get('[data-cy="canvas-container-1"]'); | ||
cy.get('[data-cy="clear-button-1"]'); | ||
cy.get('[data-cy="correct-checkbox-1"]'); | ||
cy.get('[data-cy="correct-text-input-1"]'); | ||
|
||
cy.get('[data-cy="canvas-container-2"]'); | ||
cy.get('[data-cy="clear-button-2"]'); | ||
cy.get('[data-cy="correct-checkbox-2"]'); | ||
cy.get('[data-cy="correct-text-input-2"]'); | ||
|
||
cy.get('[data-cy="submit-button"]').as("submitButton"); | ||
cy.get("@submitButton").should("be.disabled"); | ||
}); | ||
|
||
it("Submitting with three corrected annotations", () => { | ||
cy.visit("/"); | ||
cy.on("window:alert", (txt) => { | ||
expect(txt).to.contain("The task has been submitted!"); | ||
}); | ||
|
||
cy.get('[data-cy="clear-button-0"]').as("clearButton0"); | ||
cy.get('[data-cy="clear-button-1"]').as("clearButton1"); | ||
cy.get('[data-cy="clear-button-2"]').as("clearButton2"); | ||
|
||
// draw 4 | ||
cy.get('[data-cy="canvas-mouse-down-container-0"]') | ||
.trigger("mouseover") | ||
.trigger("mousedown", 20, 20) | ||
.trigger("mousemove", 40, 150) | ||
.trigger("mousemove", 150, 120) | ||
.trigger("mousemove", 150, 10) | ||
.trigger("mouseup", 150, 220); | ||
|
||
// There is a wait statement here because it takes some time for the model to calculate that it is a 4 | ||
cy.wait(2000); | ||
cy.get('[data-cy="current-annotation-0"]').should("contain.text", "4"); | ||
cy.get('[data-cy="correct-checkbox-0"]').check(); | ||
cy.get('[data-cy="correct-text-input-0"]').should("not.exist"); | ||
|
||
// draw 1 | ||
cy.get('[data-cy="canvas-mouse-down-container-1"]') | ||
.trigger("mouseover") | ||
.trigger("mousedown", 100, 20) | ||
.trigger("mousedown", 120, 200) | ||
.trigger("mouseup", 120, 200); | ||
|
||
cy.wait(2000); | ||
cy.get('[data-cy="current-annotation-1"]').should("contain.text", "1"); | ||
cy.get('[data-cy="correct-checkbox-1"]').check(); | ||
cy.get('[data-cy="correct-text-input-1]').should("not.exist"); | ||
|
||
// draw gibberish | ||
cy.get('[data-cy="canvas-mouse-down-container-2"]') | ||
.trigger("mouseover") | ||
.trigger("mousedown", 10, 20) | ||
.trigger("mousedown", 120, 200) | ||
.trigger("mouseup", 120, 200); | ||
|
||
cy.get('[data-cy="clear-button-2"]').click(); | ||
|
||
// draw 7 | ||
cy.get('[data-cy="canvas-mouse-down-container-2"]') | ||
.trigger("mouseover") | ||
.trigger("mousedown", 30, 30) | ||
.trigger("mousedown", 220, 40) | ||
.trigger("mouseup", 40, 200); | ||
|
||
cy.get('[data-cy="correct-text-input-2"]').type("7"); | ||
|
||
cy.get('[data-cy="submit-button"]').click({ force: true }); | ||
}); | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
#!/bin/sh | ||
npm link mephisto-task |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This draws a 1, but in the GitHub action it seems to sometimes think it is a 8 leading to the test failing about 1 out of every 4 times. Might make more sense to draw a number that is clearer to the model.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels kinda weird though because given the exact same mouse down values I feel like the model should give me the exact same result, but I guess that is not the case. AI stuff I guess 🤷
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed the 1 to a 3, hopefully this is more consistent
Edit:
Seems to be consistent, haven't had any github actions fail