Array inputs, v2
Created by: julianhess
A lot has changed since the last array input PR, so I rewrote it. The behavior is considerably different than before: any array input will get saved as a newline delimited file in the job directory, e.g.
inputs = { "foo" : [[0, 1, 2, 3], [4, 5, 6, 7]] }
will save foo_array.txt
in jobs/0
containing
0
1
2
3
and another foo_array.txt
in jobs/1
containing
4
5
6
7
For each shard, the environment variable foo
will contain the path to the respective foo_array.txt
. It is up to the user to handle this file as they see fit.
This is mainly to address the issue of the environment growing too big, e.g. in the case of a massive scatter, exporting a variable containing the contents of ",".join(<scatter array>)
can exceed the ~2 MB environment limit.