Profile: improve profiling code #1447
Conversation
Implement two targets: higher and lower. Choose which one to go to depending on first guess. Different calculation of next_obj_target TODO: change 1.5 to magic factor
Implemented adaptive max and min steps in profiling. If the optimization during profiling fails (results in inf value), the algorithm will first try to iteratively decrease `max_step_size` to be closer to the last point that had a successful optimization. If that doesn't work (if we reduce max_step_size below min_step_size), then max_step_size is set back to the default and we try to increase min_step_size to "jump over" the problematic area. Resampling random points and start from those is only the last resort and will be done if these two do not work. The idea is that we want to stay as close as we can to the last profiling point. TODO: Put the adaptive reduction/increase of max_step_size/min_step_size into options
- BUGFIX: absolute value in objective targets at last_delta_fval - BUGFIX: extrapolation explosions if we start at boundary - Feature: Trust region on extrapolation - Added y ticks back into the plot, sometimes the range is completely different. - Added points to the plotting of profiles (in case of one result and one profile list id) - Added color change to plotting of profiles (in case of one result and one profile list id) - LOGGING: added logging.INFO with informations of steps made and successful optimizations.
Doresic
requested review from
stephanmg,
m-philipps and
PaulJonasJost
as code owners
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## develop #1447 +/- ##
===========================================
- Coverage 82.88% 82.71% -0.18%
===========================================
Files 163 163
Lines 13786 13894 +108
===========================================
+ Hits 11427 11492 +65
- Misses 2359 2402 +43 ☔ View full report in Codecov by Sentry. |
Test were testing whether the profiling method was making a lot of steps. Now we're making a lot more steps due to higher robustness.
pypesto/profile/options.py
Outdated
| if self.magic_factor_obj_value < 0 or self.magic_factor_obj_value >= 1: | ||
| raise ValueError("magic_factor_obj_value must be >= 0 and < 1.") | ||
| if self.magic_factor_obj_value < 1: | ||
| raise ValueError("magic_factor_obj_value must be > 1.") |
There was a problem hiding this comment.
The magic factor now should be larger than 1 rather than between 0 and 1. The next_obj_target now depends on the objective function value difference of the last two profile points, rather than the difference of the last profile point to the global minimum. The former is in general much smaller than the latter, so we want to increase it rather than decrease it using a factor.
There was a problem hiding this comment.
Is there a better name for this? magic_factor sounds fun but is rather uninformative. Also: if it stays like that, documentation in line 35-37 needs to be adjusted.
There was a problem hiding this comment.
True...
It is kinda the scaling factor of the adaptive target value.
Changed to adaptive_target_scaling_factor.
There was a problem hiding this comment.
Take a look at the documentation of it in ProfileOptions. Not sure if informative enough.
| step_length = problem.lb_full[par_index] - x[par_index] | ||
| return x + step_length * delta_x_dir | ||
| step_length = abs(problem.lb_full[par_index] - x[par_index]) | ||
| return clip_to_bounds(x + step_length * delta_x_dir) | ||
|
|
||
| if par_direction == 1 and (min_delta_x > problem.ub_full[par_index]): | ||
| step_length = problem.ub_full[par_index] - x[par_index] | ||
| return x + step_length * delta_x_dir | ||
| step_length = abs(problem.ub_full[par_index] - x[par_index]) | ||
| return clip_to_bounds(x + step_length * delta_x_dir) |
There was a problem hiding this comment.
This is where the bug with the exit condition was. Absolute value fixes it.
| # Determine the direction of the step | ||
| if next_obj >= current_obj: | ||
| next_obj_target = high_next_obj_target | ||
| if next_obj >= high_next_obj_target: | ||
| direction = "decrease" | ||
| decreasing_to_high_target = True | ||
| else: | ||
| direction = "increase" | ||
|
|
||
| elif next_obj < current_obj: | ||
| next_obj_target = low_next_obj_target | ||
| if next_obj <= low_next_obj_target: | ||
| direction = "decrease" | ||
| decreasing_to_low_target = True | ||
| else: | ||
| direction = "increase" |
There was a problem hiding this comment.
If the initial guess is between the lower and higher target, we increase step size till we hit any of them. (exit conditions below)
If the initial guess is below lower target or above higher target, we decrease the step size till we cross the lower or higher target, respectively.
|
|
|
As a reference, this is the objective function value profile using the old code: |
pypesto/profile/options.py
Outdated
| if self.magic_factor_obj_value < 0 or self.magic_factor_obj_value >= 1: | ||
| raise ValueError("magic_factor_obj_value must be >= 0 and < 1.") | ||
| if self.magic_factor_obj_value < 1: | ||
| raise ValueError("magic_factor_obj_value must be > 1.") |
There was a problem hiding this comment.
Is there a better name for this? magic_factor sounds fun but is rather uninformative. Also: if it stays like that, documentation in line 35-37 needs to be adjusted.
Co-authored-by: Maren Philipps <[email protected]>
I implemented improvements and changes to the profiling algorithm and visualization. The following was tested on 5 models (Boehm, Rosenbrock, Randall, Yuhong, Wuyan) and has seen improvements in robustness and stability of profiles.
Major changes:
next_guess_methodfromadaptive_step_regressiontoadaptive_step_order_1. The former had a problem with poor extrapolation -- polynomials of high degree are known to diverge outside of the fitting region. In the testing I've done, the latter was more robust and faster.next_obj_targetand changes the step length until the proposed objective function value is stepped over. Until now, the adaptive method considered only ahigh_next_obj_targetvalue that is higher than the current objective function value. This caused it to make large steps if the profiler was in a region of decreasing objective function values. Thus, I added thelow_next_obj_targetas well. If any of the two is reached, the corresponding step is returned.walk_along_profilefailed. In most cases, this was due to a poor initial proposal ofnext_xfrom which the optimization was started. In cases where this happened, the only fail-safe was to try to resample thenext_xuniformly. This made the profile very discontinuous as it jumped randomly through the parameter space. Thus, in case of optimization failure before resorting to resampling ofnext_xI made the algorithm re-try thenext_guessgeneration, but this time with differentmax_step_sizeandmin_step_sizehyperparameters. First, thenext_guessfunction is called with an increasingly smallermax_step_sizeto try to stay closer to the last point which had a successful optimization. This is done until optimization is successful ormax_step_sizeis reduced belowmin_step_size. Then, the idea is to try to jump over the "region of poor simulation/optimization" by iteratively callingnext_guesswith an increasingly biggermin_step_size. If these two do not succeed thenext_xis randomly re-sampled as a last resort.profiles()visualization:plot_objective_valuesoption. This makes the plotting routine plot the objective function value across the parameter values instead of the default likelihood ratio. It was interesting to inspect here and there, so I left it there as I think it might be nice for others.quality_colorsoption. As the profile now has adaptivemax_step_sizeandmin_step_sizevalues, and sometimes even re-samples the points, it's nice to know when what happens. Thus, red points indicate a step with a reducedmax_step_size, blue points indicate a step with an increasedmin_step_sizeand green points indicate that the next guessx_nextwas resampled. If none of these happened, the color is black. These colors are saved throughout profiling inProfilerResult.x_color_path.Minor changes:
next_obj_targetto depend on the objective function distance to the last profile point (last_delta_fval) rather than the distance to the global optimum objective function value (delta_obj_value). The latter made it make larger steps in low-likelihood regions. However, I'm not sure whether it is desirable to that degree.Fixed some bugs: