Skip to content

Releases: facebookresearch/Mephisto

v1.5.2

17 Oct 02:29
@meta-paul meta-paul
v1.5.2
672a8f2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.2 Latest
Latest

Overview

Mephisto now offers a UI for managing qualifications and qualification assigns. We've nested it under TaskReview app since it already can run a local web server. Some new capabilities were also added:

  • Now workers can be granted qualifications directly (without context of a specific Task Unit)
  • Qualifications now support not only name, but also description
  • Multiple qualifications can now be assigned to a single Task Unit
  • Screenshots and all details are described in the docs

Note that this code change includes a database migration. It should port all your existing qualifications data automatically. Though just in case we'd recommend backing up the qualifications part of your database (using mephisto db export --qualification-only command as described here)

Some other changes include bug fixes and reorganization of local Mephisto packages for more transparent naming and versioning.

What's Changed

Full Changelog: v1.5.1...v1.5.2

Contributors

meta-paul
Assets 2
Loading

v1.5.1

02 Oct 18:17
@meta-paul meta-paul
v1.5.1
f6037c8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.1

Overview

Patch with small fixes to VideoAnnotator and docs update.

VideoAnnotator

  • Display validation error for overlapping video segments on the same track
  • Add support for multiple video formats (not just mp4)
  • Fix a few outdated unittests for generators

For docs

  • Add "Known Issues" page in the docs
  • Add FormComposer screenshots

What's Changed

Full Changelog: v1.5.0...v1.5.1

Contributors

meta-paul
Assets 2
Loading

v1.5.0

26 Sep 16:13
@meta-paul meta-paul
v1.5.0
641994a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.0

Overview

This release introduces a couple of key features, and a number of smaller improvements.

Video Annotator

VideoAnnotator is a new Task Generator for convenient and functional video annotation.

Key features:

  • Worker can indicate multiple video segments along multiple tracks, and annotate each one
  • For each segment, you can specify (via JSON config) desired annotation input (as a list of standard form fields)
  • VideoAnnotator JSON config works similar to FormComposer (including pre-signed S3 URLs, and support for custom JS triggers)
  • Workers inputs are downloadable in JSON
  • When viewing a completed unit, worker inputs are also available in WebVTT format

InHouse provider

The new InHouse provider enables using Mephisto with your own "in-house" crowd - and basically crowdsourcing your task to people you know. (This used to be "mock" provider last year, but now we're using "mock" only for testing).

The initial implementation is quite simple: a worker navigates to your Task page, types in their worker nickname, and proceeds to the next available Task unit (if any). Afterwards you can review their results and take notes or grant qualifications, just like you do that with the 3rd party workers.

Future roadmap for InHouse provider would include: proper worker authorization; allowing workers to view researcher notes; restricting worker acces based on their qualification.

Feature improvements

TaskReview app

  • Visualized selection-based response stats in Task Review
  • Added ability to view completed task units, without assessing them

DataPorter feature

  • Now you can port only granted worker qualifications (without all other data), to easily build pool of known workers collaboratively

FormComposer

  • Enabled CSS insertions in FormComposer

Grafana

  • Updated Grafana dockerized deployment and docs
  • Added access to Grafana dashboard from TaskReview app

General housekeeping

  • Updated and added missing architecture diagrams in Mephisto docs
  • Refactored increasingly nested CLI commands, and updated their Help strings
  • Add example of GoldUnit and Worker Onboarding
  • upgraded NodeJS to v20

Included PRs

Full Changelog: v1.4.0...v1.5.0

Contributors

JackUrb, dependabot, and meta-paul
Assets 2
Loading

v1.4.0

13 May 02:38
@meta-paul meta-paul
v1.4.0
5ac0e4b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.4.0

Overview

In this release we added Data Porter feature that allows to move full/partial data dumps between different isntances of Mephisto. This enables things like:

  • backups/restores of Mephisto data
  • consolidating task data across different Mephisto instances
  • sharing task data with your colleagues

We also introduced a simple (forward-only) workflow for database migrations, enabled custom code snippets in Form Composer, and updated numerous library versions.

Last but not least, mephisto.ai documentation got a major update as well.

Included PRs

Full Changelog: v1.3.0...v1.4.0

Contributors

JackUrb, dependabot, and meta-paul
Assets 2
Loading

v1.3.0

27 Feb 22:01
@meta-paul meta-paul
v1.3.0
2dd08af
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.3.0

Overview

This release introduces FormComposer task generator. It aims to simplify creating React front-end for form-based Tasks. Main highlights:

  • Task front-end can be defined via JSON configuration (no need for custom code)
  • Multiple form versions are supported within a single Task
  • Multi-version forms can be generated within the same Task, based on variables' values
  • Expiring (presigned) URLs are supported for S3 files (for data safety)
  • Task Review app support includes preview of uploaded media files

Included PRs

Full Changelog: v1.2.1...v1.3.0

Contributors

meta-paul
Assets 2
Loading

v1.2.1

01 Feb 22:16
@meta-paul meta-paul
v1.2.1
5686c28
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.2.1

Overview

This release rounded out TaskReview app functionality. Now you can conveniently give feedback and bonuses to individual workers while performing review of their submissions. This feature works with all currently supported providers.

Included PRs

Full Changelog: v1.2.0...v1.2.1

Contributors

meta-paul
Assets 2
Loading

Mephisto 1.2.0

19 Jan 20:54
@meta-paul meta-paul
v1.2.0
aac76d7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Mephisto 1.2.0

Overview

This release is focused on adding a new provider (Prolific) and better data review experience.
It has a larger diff than usual, and we will return to a more frequent release cadence afterwards.

See the list of changes when upgrading from v1.1.0 below. Note that only essential dependency updates are listed.

Big Changes

  • Added integration with Prolific provider, as an alternative to Mturk for data collection. Expect some minor additions in the subsequent releases, but the core functionality is already there. #1008
  • Added UI-driven review of collected task results, a major upgrade over the previously used command-line script. Support for bonuses and feedback for workers to be added in the subsequent releases. When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #1058
  • Added dockerization support for running Mephisto. #1008

Included PRs

Feature + UX Improvements

Bugfixes

Docs + Examples

Dependencies

New Contributors

Full Changelog: v1.1.0...v1.2.0

Contributors

benmatselby, pringshia, and 8 other contributors
Assets 2
Loading

Mephisto 1.1.0

28 Oct 17:47
@JackUrb JackUrb
v1.1.0
66be164
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Mephisto 1.1.0

Overview

This release contains a number of Stability and Quality of Life improvements from v1.0.0 and v1.0.2. The big focus has been on collection success rate, documentation, and various bugfixes.

For those updating from 1.0, see the 1.0.2 changes as well for a complete changelog.

Breaking changes

  • Mephisto no longer supports Python 3.7. This is as of #898, wherein to get the ParlAI peer imports working we needed to bump to ParlAI 1.7.0, which no longer supports Python 3.7.

Big Changes

  • When launching on MTurk, Mephisto will request that you run the mephisto scripts mturk cleanup script inline before launching. This exercise prevents having long-running tasks that haven't been cleaned up properly. #918
  • When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #919

Big Fixes

  • Lots of documentation, thanks @Etesam913!
  • Significant fixes to proper execution and state tracking in Mephisto, which should improve stability across all job types (#875, #878, #902, #922)
  • Major fix for Heroku tasks using the useMephistoLiveTask hook, which was causing all messages to be double sent in both directions. (#921)

Included PRs

Feature + UX Improvements

Bugfixes

Testing & DevX

Docs + Examples

Dependencies

New Contributors

Full Changelog: v1.0.3...v1.1.0

Contributors

pringshia, JackUrb, and 7 other contributors
Assets 2
Loading

v1.0.3: Last version, but ParlAI-stable

22 Jul 16:05
@JackUrb JackUrb
v1.0.3
210e407
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.0.3: Last version, but ParlAI-stable

Overview

v1.0.2 shipped with a bug that broke ParlAI-chat related tasks. This release only includes the fix for that (#859), but is otherwise equivalent to v1.0.2.

Assets 2
Loading

v1.0.2: Key Bugfixes and QoL Upgrades

21 Jul 21:13
@JackUrb JackUrb
v1.0.2
c085203
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.0.2: Key Bugfixes and QoL Upgrades

Overview

Some issues still remaining from v1.0.0 (and even sooner) have been ironed out, so this release aims to fix those as well as include some new QoL upgrades.

Highlights

  • A status mismatch issue related to collection slowdowns on long-running tasks has been squashed (#830)
  • Adding testing for our example files (#795, #800)
  • Upgrading completely to Poetry as our package manager (#779)
  • Core work for new worker addons package, to be included in coming release notes (#791, #785)

Additional Changes

  • MTurk improvements and fixes:
    • #810: Functionality to specify the lifetime of a task
    • #780: Fix to HIT tags that were being improperly submitted as individual characters
    • #767: Fix to storing and restoring AWS credentials
  • Bugfix for deploying to the HerokuArchitect via git (#824)
  • Locking down the version of Node that our router uses to solve issues deploying on the EC2Architect (#762)
  • Cleanup to the Operator.shutdown method to allow disconnected onboardings to be cleared (#769)
  • Developer QoL:
    • #814: Consolidating the AgentState classes and introducing a metadata attribute
    • #770: Making Agent and OnboardingAgent pull from the same base class for code deduplication
  • Bugfixes to the RemoteProcedureBlueprint:
    • #763: Fixing an issue that prevented reloading from restoring the state
    • #798, #782: Cleaning up on the RemoteProcedure examples
    • #784: Adding a new RemoteProcedure example using a text model-in-the-loop
Assets 2
Loading