I read the doc about parallelism, parallel execution and job scheduling but
however I have some doubts about parallelism.
In my first try I unset parallelism in my code and commented
parallelism.default key in link-conf file. In this case I supposed the
parallelism was set by Flink automatically on operator basis. Is this
In a second try I unset parallelism in my code but I set
parallelism.default: 2 in flink-conf file.
In my code I have some source, some sink and two custom function from an
external library supported by Flink. These one don’t have setParallelism()
method so I can’t set a specific parallelism for them.
Anyway when I tried to execute it I obtain the following error:
/java.lang.UnsupportedOperationException: Forward partitioning does not
allow change of parallelism. Upstream operation: Learn-11 parallelism: 1,
downstream operation: Select-13 parallelism: 3 You must use another
partitioning strategy, such as broadcast, rebalance, shuffle or global./
This lead me to the second question. Am I constrained to set
parallelism.default: 1 to respect parallelism of “learn” method? In this way
I need to set parallelism to each operator in Flink (for example 2) and
leave “select” parallelism to the default value (1) since I can’t set a
specific parallelism on it (I can’t set 3 as suggested in the error).
Moreover, I searched a lot on relations between partitioning and parallelism
on doc but everything I read seems a bit unclear for me. Can you explain it
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/