Replace built-in measures with measures in StatisticalMeasures.jl #909
Conversation
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## for-a-0-point-22-release #909 +/- ##
============================================================
- Coverage 87.78% 86.23% -1.55%
============================================================
Files 40 32 -8
Lines 3659 2833 -826
============================================================
- Hits 3212 2443 -769
+ Misses 447 390 -57
☔ View full report in Codecov by Sentry. |
update `evaluate` docstring
further tweak
|
I'm not expecting detailed review of this PR, but think someone should at least look over the planned breakages, and so forth. |
There was a problem hiding this comment.
I don't understand all the nuances of this large change, but here are some extra thoughts:
- Due to the large amount of changes, is this a nice moment to go to v1.0.0? Versions of 1 and up have the benefit that there is more control due to MAJOR.MINOR.PATCH instead of MAJOR.MINOR/PATCH. Also there is a risk of being mocked at https://0ver.org/ when staying at <1.0.0 😛
- Should there also be a migration guide somewhere like in the docs? The PR description would be reasonable, but also contains non-essential information making it harder for clients to migrate.
| # for julia < 1.9 | ||
| if !isdefined(Base, :get_extension) | ||
| include(joinpath("..","ext", "DefaultMeasuresExt.jl")) | ||
| @reexport using .DefaultMeasuresExt.StatisticalMeasures |
There was a problem hiding this comment.
So StatisticalMeasures is only available in the public (re-exported) scope in Julia 1.9 or does the new extensions system from Julia automatically bring things to the public scope? (I'm not familiar with the new system yet)
There was a problem hiding this comment.
If using MLJBase without MLJ, then, in Julia 1.9 or higher, StatisticalMeasures must be explicitly imported to use measures that were previously part of MLJBase (the names are automatically re-exported by a pkg extension). For earlier of Julia versions, StatisticalMeasures.jl is a hard dependency (through a hack that is standard practice). If using MLJ, then all previous measures are still available, because StatisticalMeasures.jl will be a hard dep of MLJ.
| results = NamedTuple{modelnames}( | ||
| [( | ||
| measure = stack.measures, | ||
| measurement = Vector{Any}(undef, n_measures), | ||
| operation = _actual_operations(nothing, stack.measures, model, verbosity), | ||
| per_fold = [Vector{Any}(undef, nfolds) for _ in 1:n_measures], | ||
| per_observation = Vector{Union{Missing, Vector{Any}}}(missing, n_measures), | ||
| per_observation = [Vector{Vector{Any}}(undef, nfolds) for _ in 1:n_measures], |
There was a problem hiding this comment.
Just to double check, per_observation is now a Vector{Vector{Vector{Any}}}?
There was a problem hiding this comment.
Yeah, per_observation[i][j][k] is the the measurement for the kth observation, in the jth fold, for the ith measure. There no longer can be missings because measures that don't compute per-observation measurements, simply report copies of the aggregated instead.
|
Overall this PR looks like an improvement. The main idea was to make In general I'm in favor of merging this. It looks like a clear improvement. |
Version 1.0.0 it is.
|
In this PR we:
(breaking) Remove the definition of all measures (
mae,log_loss, etc), which can instead be loaded from StatisticalMeasures.jl, which provides like-named measures, with the exception of those previously wrapped from LossFunctions.jl, which now must be explicitly imported and wrapped. (Julia >= 1.9). Breaking behaviour and workarounds are detailed below. For Julia < 1.9, make StatisticalMeasures a hard dependency and re-export it's names. For Julia 1.9 provide default measures for models using a Pkg extension, including StatisticalMeasures as a weak dependency.Remove the LossFunctions.jl dependency to close Investigate loading time of LossFunctions.jl #570.
Add StatisticalMeasures.jl as a weak dependency, so that default measures can be provided through an Pkg extension, ext/DefaultMeasuresExt.jl.
Adapt the resampling (
evaluate!) code (and a semi-duplication for modelStacks) to the new measure API set out in StatisticalMeasuresBase.jl.This PR closes #502, as StatisticalMeasures.jl provides some out-of-the-box multi-target measures and StatisticalMeasuresBase.jl provides a wrapper
multimeasureto get more. Similarly, closes #529.Breaking behaviour relevant to many users
In Julia 1.9 or higher, StatisticalMeasures.jl must be explicitly imported to use measures that were previously part of MLJBase (but new releases of MLJ will import StatisticalMeasures.jl, so they will be available that umbrella pkg).
Measures that in MLJBase skipped
NaNvalues will now (at least by default) propagate those values, when imported from StatsticalMeasures.jl. Missing value behaviour is unchanged, except some measures that previously did not supportmissingnow do.All measures return a single aggregated measurement. In other words, measures
previously reporting a measurement per-observation (previously subtyping
Unaggregated) no longer do so. To get per-observation measurements, use the new methodmeasurements(measure, ŷ, y[, weights, class_weights])from StatisticalMeasures.jl.Measures with a "feature argument"
X, as insome_measure(ŷ, y, X), are no longersupported. See What is a
measure?
for allowed signatures in measures.
Aliases for measure types have been removed. For example
RMSE(alias forRootMeanSquaredError) is gone. Aliases for instances, such asrmsandcross_entropyhave been preserved. The exception isprecision, for whichppvcanbe used in its place. (This is to avoid conflict with
Base.precision, which waspreviously pirated; you can also do
StatisticalMeasures.precision.)info(measure)has been decommissioned; query docstrings or access the new measuretraits
individually instead. These traits are now provided by StatisticalMeasures.jl and not
re-exported.
Behaviour of the
measures()method, to list all measures and associated traits, haschanged. It now returns a dictionary instead of a vector of named tuples;
measures(predicate)is decommissioned, butmeasures(needle)is preserved. (This method,owned by StatisticalMeasures.jl has some new search options.)
Measures that were wraps of losses from LossFunctions.jl are no longer exposed by
MLJBase.jl. To use such a loss, you must explicitly
import LossFunctionsand wrap theloss appropriately. See Using losses from
LossFunctions.jl
for examples.
Some user-defined measures working in previous versions of MLJBase.jl may not work
without modification, as they must conform to the new StatisticalMeasuresBase.jl
API. See
this tutorial on
how define new measures.
The default measure for regression models (used in
evaluate/evaluate!whenmeasuresis unspecified) is changed fromrmstol2=LPLoss(2)(mean sum of squares).Breaking behaviour likely relevant only to developers of some client packages
The abstract measure types
Aggregated,Unaggregated,Measurehave beendecommissioned. (A measure is now defined purely by its calling
behaviour.)
What were previously exported as measure types are now only constructors.
target_scitype(measure)is decommissioned. Related isStatisticalMeasures.observation_scitypewhich declares an upper bound on the allowedscitype of a single observation
prediction_type(measure)is decommissioned. Instead useStatisticalMeasures.kind_of_proxy.The trait
reports_each_observationis decommissioned. Related isStatisticalMeasures.can_report_unaggregatedwhich isfalsewhen the newmeasurementsmethod returnsncopies of the aggregated measure, wherenis thenumber of observations provided, instead of individual observation-dependent
measurements.
aggregation(measure)has been decommissioned. Instead useStatisticalMeasures.external_mode_of_aggregation(measure).instances(measure)has been decommissioned; query docstrings for measure aliases, orfollow this example:
aliases = measures()[RootMeanSquaredError].aliases.is_feature_dependent(measure)has been decommissioned. Measures consuming feature data are not longersupported; see above.
distribution_type(measure)has been decommissioned.docstring(measure)has been decommissioned.Behaviour of
aggregatehas changed.The following traits, previously exported by MLJBase, cannot be applied to measures:
supports_weights,supports_class_weights,orientation,human_name. Instead usethe traits with these names provided by StatisticalMeausures.jl (they will need to be
qualified, as in
using StatisticalMeasures; StatisticalMeasures.orientation(measure)).