When doing a grid-search over some hyper-parameters of an algorithm I often need to specify the possible value space for each parameter and then evaluate on every combination of these values.

In this case, the product function of the itertools module is handy:

from itertools import product

def train_and_evaluate(lr, num_layers, dataset):
    print(f'training on {dataset} with {num_layers=} and {lr=}')

params = (
    (0.01, 0.1, 0.5), #learning rate
    range(10, 40, 10), #number of hidden neurons
    ('MNLI', 'SNLI', 'QNLI') # training dataset
)

for config in product(*params):
    train_and_evaluate(*config)

However, this code relies on the order of parameters in the function train_and_evaluate, which could be considered not very pythonic, as explicit is better than implicit.

In the same way, the 'intention' for the values is only clear with the context of the function signature (or the comments, if someone bothered to write them).

Therefore I propose a new variant of the product() function that supports Mappings instead of Iterables to enable to explicitly specify the function of an iterables values in the product. A trivial implementation could be:

def named_product(repeat=1, **kwargs):
    for combination in itertools.product(*kwargs.values(), repeat=repeat):
    yield dict(zip(kwargs.keys(), combination))

which could then be used like this:

params = {
    'lr': (0.01, 0.1, 0.5), #learning rate
    'num_layers': range(10, 40, 10), #number of hidden neurons
    'dataset': ('MNLI', 'SNLI', 'QNLI') # training dataset
}

for config in named_product(**params):
    train_and_evaluate(**config)

This has the advantage that the order of the parameters in the kwargs does not depend on the method signature of train_and_evaluate.

Open Questions

• Support scalar values?

  I the example use-case it may be nice to not have each kwarg to be an iterable, because you may want to specify all parameters to the function, even if they do not vary in the product items (factors?)

    params = named_product(a=(1, 2), b=3)
  

  instead of

    params = named_product(a=(1, 2), b=(3, ))
  

  However, this may be unexpected to users when they supply a string which would then result in an iteration over its characters.

• Would this may be suited for the more-itertools package?

  However, I would love to have this build-in and not depend on a non-stdlib module

• More functionality:

  In my toolbox-package I implemented a variant with much more functionality: https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html. This version offers additional functionality compared to the proposed function:

  • accept both, a dict as *arg or **kwargs and merges them
  • accepts scalar values, not only iterables but treats strings as scalars
  • if the value of any item is a dict itself (not scalar or Sequence), iterate over the nested dict as a seperate call to named_product would and update the 'outer' dict with those values
  • copies values before yielding which is useful if the returned values are mutable (can be excluded)

  Maybe some of these features would also be useful for a general version? However, this version is mostly developed for my special use-case and some behaviour is solely based on my peculiarities

Example usage:

Here is an example from my actual code which may clarify some decisions on functionality of the features of https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html

configs = named_product({
    'nb_epoch': 3,
    'acquisition_iterations': 0, 
    'training_function': {
        partial(train_bert, decoder_function=create_decoder):{
            'decoder_name': 'FFNN'
        },
        partial(train_bert, decoder_function=create_cnn_decoder):{
            'decoder_name': 'CNN'
        }
    },
    'training_data': partial(load_glue, initial_training_data_size=10),
    'number_queries': 100,
    'batch_size': 64,
    'dataset_class': [QNLIDataset, MNLIDataset],
    'num_frozen_layers': 0,
    'acquire_function': {
        acquire_uncertain: {
            'certainty_function': calculate_random, 
            'predict_function': 'predict',
            'dropout_iterations': 1
        }
    }
})