A question about the npy files used in spark_env

Question:
There are some npy files in your code, including "task_duration" and "stage_id_to_node_idx_map", and could you please tell me how to generate these files? I'm trying to reuse your code, but I don't know what these files mean.

Thank you!

We load the TPCH jobs here: https://github.com/hongzimao/decima-sim/blob/master/spark_env/job_generator.py#L12-L15 We measure the task duration and job structure information by running those TPCH jobs in a real Spark cluster. For example, in task_duration_*.npy, we measure the task duration for different "wave" of task execution (e.g., first wave, when doing more IO, typically has longer task duration). Those files are just the job information loaded into the simulator. Hope this helps!

So first wave means time spent on IO or something else? And what does "fresh duration" mean? Does it mean the whole time a task need?
And I notice that in first wave there are some keys such as "0", "40", what does these mean? In addition, did you use "stage_id_to_node_idx_map" files? Because I didn't find them in your code.

So first wave means time spent on IO or something else? --- Yes

And what does "fresh duration" mean? Does it mean the whole time a task need? --- this is used here

decima-sim/spark_env/node.py

Lines 103 to 116 in c010dd7

    
           # the executor never runs a task in this job 
        
           # fresh executor incurrs a warmup delay 
        
           if len(self.task_duration['fresh_durations'][executor_key]) > 0: 
        
               # (1) try to directly retrieve the warmup delay from data 
        
               fresh_durations = \ 
        
                   self.task_duration['fresh_durations'][executor_key] 
        
               i = np.random.randint(len(fresh_durations)) 
        
               duration = fresh_durations[i] 
        
           else: 
        
               # (2) use first wave but deliberately add in a warmup delay 
        
               first_wave = \ 
        
                   self.task_duration['first_wave'][executor_key] 
        
               i = np.random.randint(len(first_wave)) 
        
               duration = first_wave[i] + args.warmup_delay

if we measure the "fresh duration", that will be more accurate to simulate; otherwise we will fallback to first wave with a executor warmup delay

And I notice that in first wave there are some keys such as "0", "40", what does these mean? --- IIRC, these are the task duration under different number of executors assigned to the node. When we simulate the runtime of a task, we find (or interpolate) the task duration based on corresponding degree of parallelism.

In addition, did you use "stage_id_to_node_idx_map" files? Because I didn't find them in your code. --- I used it for other bookkeepings, like generating the demo. It's just an index mapping for the nodes in the graph I think. The simulator doesn't need this info if you can run the code without it.

	# the executor never runs a task in this job
	# fresh executor incurrs a warmup delay
	if len(self.task_duration['fresh_durations'][executor_key]) > 0:
	# (1) try to directly retrieve the warmup delay from data
	fresh_durations = \
	self.task_duration['fresh_durations'][executor_key]
	i = np.random.randint(len(fresh_durations))
	duration = fresh_durations[i]
	else:
	# (2) use first wave but deliberately add in a warmup delay
	first_wave = \
	self.task_duration['first_wave'][executor_key]
	i = np.random.randint(len(first_wave))
	duration = first_wave[i] + args.warmup_delay