What is L in Figure 6?
Closed this issue · 3 comments
What is L in Fig. 6 and how do you ensure the predicted degree parallelism is greater than the number of executors assigned to this job?
L is the number of executors.
Hongzi: We used a masking mechanism to ensure the executor assignment is always valid. These are the code snippets for computing the mask https://github.com/hongzimao/decima-sim/blob/master/actor_agent.py#L424-L451
I'm a little confused about the “source_job” variable of the above link code. Do you know what it stands for?
source_job is where the executor comes from. If source_job == curr_job, it means allocating the executor won't pay a migration cost (simulating how Spark shuts down/launches a JVM, more details see section 3 and 4 for "moving cost" in the paper).