How to get all parameters inside a step.

[valayDave]

I am trying to get all the parameters I have declared in my step but I am not able to with the _get_parameters() method for the FlowSpec object.

# Use the specified version of python for this flow.
@conda_base(python=get_python_version(),libraries={'pytorch':'1.4.0'})
class ParameterDebugFlow(FlowSpec):
    
    some_data = Parameter('some_data',help='Mock Data',default=128)

    @step
    def start(self):
        for var, param in self._get_parameters():
            print(param)

        self.next(self.end)

    @step
    def end(self):
        print("Flow Endeded")


if __name__ == '__main__':
    ParameterDebugFlow()

Is this intended functionality? If so how do I retrieve the Parameters I have declared in my class? I am porting a DL project which is fairly well parameterized using argparse for GPU based instance training. I wanted to move the project to metaflow for performing broader hyperparameter search.

[romain-intel]

The _get_parameters function is meant to be used internally and won't return what you expect because, by the time you call it in your step, we have replaced all parameters with a read-only property (look for _init_parameters in task.py).

What is your specific need for needing to know the names of the parameters (you can access the values simply using self.)?

[valayDave]

I was moving a DL Project into Metaflow for a broader hyperparameter search. I have more than 15+ Parameters in the experiment. Dynamic extraction of those parameters can be useful when taking previous flow runs and comparing the best model parameters from the previous runs.

When you are experimenting with DL you have varying number of parameters many of the times. And so after testing with lots of flows with varying params, it would be a very useful utility to be able to extract them after each run or even during a run.

[romain-intel]

Got it, so you effectively want a way to iterate over the names of the parameters, something like:

for param_name in self.parameters():
    print("Parameter %s has value %s" % (param_name, getattr(self, param_name)))

Is this correct?

[valayDave]

Yes. Exactly Like that.