Timed Pool#

misc.timed_pool.run(max_child_proc_cnt=4, func=None, args=None, max_time=None, delete_data=True)#

Create a subprocess loop to work the issue task defined by func and it’s arguments.

This subprocess loop is different in two key elements from the default python processpool. Only a single subprocess can return data at a time. This drastically decreases the odds of a memory utilization spikes, which otherwise would cause a crash. This is linked to how pickle handles data transfer via pipes. A maximum time can be set after which a subprocess is terminated and restarted.

Parameters:

  • max_child_proc_cnt (int) – Maximum number of child processes.

  • func (callalble) – The function to be processed.

  • args (Any) – List of arguments. Every element in the list is handled by a separate process.

  • max_time (float) – Maximum time to wait for a processe prior to cancellation.

  • delete_data (boolean) – Flag on whether input data is deleted after successful computation of it’s results.

Returns:

The processed information from func and args as a list. The order is identical to the order in which the argument blocks were given.

Return type:

Any

The following code example shows how to use the multiprocessing pool.

import finn.misc.timed_pool as tp

NUMBER_OF_PROCESSES = 10

def bar(param1, param2):
     return foo(param1, param2)

def main():

   output = tp.run(NUMBER_OF_PROCESSES, bar, [(arg1[idx], arg2[idx],) for idx in range(100)], max_time = 100, delete_data = True)

main()
On this page

This Page