Skip to content

Array inputs, v2

Aaron Graubert requested to merge array_redux into master

Created by: julianhess

A lot has changed since the last array input PR, so I rewrote it. The behavior is considerably different than before: any array input will get saved as a newline delimited file in the job directory, e.g.

inputs = { "foo" : [[0, 1, 2, 3], [4, 5, 6, 7]] }

will save foo_array.txt in jobs/0 containing

0
1
2
3

and another foo_array.txt in jobs/1 containing

4
5
6
7

For each shard, the environment variable foo will contain the path to the respective foo_array.txt. It is up to the user to handle this file as they see fit.

This is mainly to address the issue of the environment growing too big, e.g. in the case of a massive scatter, exporting a variable containing the contents of ",".join(<scatter array>) can exceed the ~2 MB environment limit.

Merge request reports