Timed Pool#

misc.timed_pool.run(max_child_proc_cnt=4, func=None, args=None, max_time=None, delete_data=True)#

Creates a subprocess loop to work the issue task defined by func and it’s arguments. This subprocess loop is different in two key elements from the default python processpool.

Only a single subprocess can return data at a time. This drastically decreases the odds of a memory utilization spikes which otherwise would cause a crash. This is linked to how pickle handles data transfer via pipes.
A maximum time can be set after which a subprocess is terminated and restarted.

Parameters:

max_child_proc_cnt – Maximum number of child processes.
func – The function to be processed.
args – List of arguments. Every element in the list is handled by a separate process.
max_time – Maximum time to wait for a processe prior to cancellation.
delete_data – Flag on whether input data is deleted after successful computation of it’s results.

:return The processed information from func and args as a list. The order is identical to the order in which the argument blocks were given.

The following code example shows how to use the multiprocessing pool.

import finn.misc.timed_pool as tp

NUMBER_OF_PROCESSES = 10

def bar(param1, param2):
     return foo(param1, param2)

def main():

   output = tp.run(NUMBER_OF_PROCESSES, bar, [(arg1[idx], arg2[idx],) for idx in range(100)], max_time = 100, delete_data = True)

main()

FiNNPY

Timed Pool#