Extremely slow compilation
Closed this issue · 2 comments
In my code there are several commented-out if statements like below.
Note that the line under the if is live.
#if save_all_steps:
cc1[d1][ny][d3][0][2][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
I want them all to be like this:
if save_all_steps:
cc1[d1][ny][d3][0][2][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
Compilation gets slower and slower the more of them I uncomment.
With all of them commented out it takes under a minute.
With all of them uncommented it takes over 100 minutes, and I didn't wait any longer.
This is the code:
import numpy
#pythran export input_order
input_order = [
'a1', 'a2', 'a3', 'a4',
'b', 'c',
'g', 'h',
'a5', 'a6', 'a7', 'a8', 'a9', 'a10',
'a11', 'a12', 'a13', 'a14', 'a15',
'ba', 'ir', 'al',
'oi',
'nai', 'nlp', 'nbp', 'ni', 'ned', 'ndg', 'naf',
'cebi', 'ceg', 'cedy', 'cei', 'cci', 'cbbi', 'cbi', 'cni', 'cpro', 'ccgi', 'cimi', 'cgi',
'clip', 'cpay',
'tli',
'ins', 'asu', 'asi', 'hp', 'target', 'tad', 'ap',
'ai',
's', 'as',
'fav',
'cf',
'cpp1', 'cpp2', 'cpp3',
'o1', 'o2', 'o3',
'n1', 'n2', 'n3',
'cc1', 'cc2', 'cc3',
'cache_x0',
'c1', 'c2', 'c3', 'c4', 'c5', 'c6', 'c7', 'c8', 'c9', 'c10', 'c11', 'c12', 'c13', 'c14', 'c15'
]
#pythran export input_indices
input_indices = dict([(k,i) for i,k in enumerate(input_order)])
def get_index (ai, acc, d3):
for item in ai:
if item[0] == acc and item[1] == d3:
return item[2]
return None
#pythran export ncs
ncs = 9
#pythran export ncsv
ncsv = 13
#pythran export set_cc1 (
# float64 [], int, int, float64 [][][][][][] order(C), bool,
# (
# int, int, int, int,
# int, int,
# float64 list, float64 list,
# float64 list, float64 list, float64 list, float64 list, bool list, int list,
# bool, bool [], (bool, bool) list, bool [], str,
# float64 [], float64 [], bool list,
# float64 [][] order(C),
# float64 [], float64 [], float64 [][] order(C), float64 [][] order(C), float64 [][] order(C), float64 [][] order(C), float64 [][] order(C),
# float64 [], float64 [], float64 [], float64 [], float64 [], float64 [], float64 [], float64 [], float64 [][] order(C), float64 [], float64 [], float64 [],
# (float64, float64) list, float64 list,
# float64 [],
# float64 [][] order(C), float64 [], float, float64 [], float64 list, float64 list, float64 [][] order(C),
# (str, int, int) list,
#
# float64, float64 list,
# bool list,
# int,
#
# float64 [][][] order(C), float64 [][] order(C), float64 [][] order(C),
# float64 [][][] order(C), float64 [][] order(C), float64 [][] order(C),
# float64 [][][] order(C), float64 [][] order(C), float64 [][] order(C),
# float64 [][][][][][] order(C), float64 [][] order(C), float64 [][] order(C),
# float64 [],
#
# (float, float, float) list, float, float, float, float, float, float, float, float, float, float, float, float, float, float
# )
#)
def set_cc1 (x, d3, d1, cc1, save_all_steps, inputs):
a1 = inputs[input_indices['a1']]
a4 = inputs[input_indices['a4']]
ceg = inputs[input_indices['ceg']][d3]
cedy = inputs[input_indices['cedy']][d3]
cbi = inputs[input_indices['cbi']][d3]
cpro = inputs[input_indices['cpro']][d3]
ai = inputs[input_indices['ai']]
s = inputs[input_indices['s']]
c13 = inputs[input_indices['c13']]
c14 = inputs[input_indices['c14']]
c15 = inputs[input_indices['c15']]
coi = get_index (ai, 'corp', d3)
rdwa = 0.
ed = 0.
id = 0.
pcdc = 0.
sh = 0.
# ----------------------------------
hbitf = 0.12
# ----------------------------------
redf = 0.31
nedf = 0.37
ltc = 200
cgtf = 4/5
cdcf = 1/3
# ------------------
def gf (eb, bb):
if bb != 0:
ef = abs(eb / (eb + bb))
bf = 1 - ef
else:
ef = 1
bf = 0
return ef, bf
if coi is not None:
for ny in range(a1 + 1):
if ny != 0:
y = ny - 1
rdwa = x[coi + y * a4]
else:
y = 0
rdwa = 0.
ed = 0.
id = 0.
pcdc = 0.
sh = 0.
if ny != 0:
eb = cc1[d1][ny - 1][d3][0][ncs - 1][0]
bb = cc1[d1][ny - 1][d3][0][ncs - 1][1]
red = cc1[d1][ny - 1][d3][0][ncs - 1][2]
nred = cc1[d1][ny - 1][d3][0][ncs - 1][3]
gri = cc1[d1][ny - 1][d3][0][ncs - 1][4]
cgai = cc1[d1][ny - 1][d3][0][ncs - 1][5]
cdc = cc1[d1][ny - 1][d3][0][ncs - 1][6]
id_max = cc1[d1][ny - 1][d3][0][ncs - 1][7]
else:
eb = inputs[input_indices['cebi']][d3]
bb = inputs[input_indices['cbbi']][d3]
red = inputs[input_indices['cei']][d3]
nred = inputs[input_indices['cni']][d3]
gri = inputs[input_indices['cgi']][d3]
cdc = inputs[input_indices['cci']][d3]
cgai = inputs[input_indices['ccgi']][d3]
id_max = inputs[input_indices['cimi']][d3]
for j in range(2):
if j == 0:
li = False
else:
li = True
ed = 0.
id = 0.
pcdc = 0.
sh = 0.
skip = False
# -------------------
if ny != 0 and j != 1:
# ----------------
#if save_all_steps:
cc1[d1][ny][d3][0][0][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
# -------------------
#if save_all_steps:
cc1[d1][ny][d3][0][1][:] = eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh
# -------------------
t1 = min([cpro[y], ltc]) * c13
t2 = 789 + cgai
gri_ch = max([cpro[y] - ltc, 0]) - t2
gri += gri_ch
rem = cpro[y] - t1 - t2
tb = eb + bb
if tb != 0:
eb_ch = rem * eb / tb
eb += eb_ch
else:
eb += rem / 2
#if save_all_steps:
cc1[d1][ny][d3][0][2][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
else:
#if save_all_steps:
cc1[d1][ny][d3][j][0][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
cc1[d1][ny][d3][j][1][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
cc1[d1][ny][d3][j][2][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
# -----------
# 1.
if li:
ed = red / redf
else:
ed = min([rdwa, red / redf])
if rdwa == 0 and not li:
skip = True
#if save_all_steps:
cc1[d1][ny][d3][j][3][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
# 2.
if not skip:
if li:
id = id_max + 0.
else:
id = min([rdwa, id_max])
if rdwa == 0 and not li:
skip = True
#if save_all_steps:
cc1[d1][ny][d3][j][4][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
# 3.
if not skip:
oef, obf = gf(eb, bb)
if cgai != 0 or cdc != 0:
mnc = cgai * cdcf
mda = cdc + mnc
if cgai != 0:
if li:
fts = 1
else:
if cdc > rdwa:
fts = 0
elif mda > rdwa:
fts = (rdwa - cdc) / mnc
else:
fts = 1
if fts * eb != 0:
ets = fts * eb
if cdc < rdwa or li:
rdwa -= cdc
cdc = 0.
else:
sh -= rdwa
rdwa = 0.
#if save_all_steps:
cc1[d1][ny][d3][j][5][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
# 4.
if not skip:
if rdwa > 0 or li:
if li:
ed_ch = min([gri, sh])
else:
ed_ch = min([gri, sh, rdwa])
if li:
id_ch = sh + 0.
else:
id_ch = min([sh, rdwa])
#if save_all_steps:
cc1[d1][ny][d3][j][6][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
# 5.
# 5.1
if not skip:
if eb != 0:
t = cgai * cgtf * c14
else:
esp = 0.
sh += bb + 0.
bb = 0.
if rdwa > 0 or li:
if li:
skip = True
#if save_all_steps:
cc1[d1][ny][d3][j][7][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
# 5.2
if not skip:
if rdwa > 0:
esp = rdwa * oef
bsp = rdwa * obf
else:
eb += sh * oef
bb += sh * obf
sh = 0.
cc1[d1][ny][d3][j][8][:] = [eb, bb, red, nred, gri, cgai, cdc, id_max, ed, id, pcdc, rdwa, sh]
This is an obfuscated version with a lot of critical math changed or removed, but the issue occurs exactly the same as in my real code.
Using the latest master.
.
I also want to take this opportunity to say that set_cc1 function was an absolute nightmare to compile.
Usually pythran would crash with something like "None-type object has no attribute line_no".
In one version simply changing e.g. "a = b + c" to "a = b + c + 0" would make the crash go away.
I'm not kidding!
The version where that happened was already confirmed to be running perfectly in python.
There were all sorts of other insane crashes that were fixed by just moving some things around in a way that doesn't affect the result.
This function is quite a monstrosity, especially in its full form, but I have other big functions too.
But this one is the only one that gave me trouble every step of the way.
At one point I had some debugging printouts in there, where I would assemble a string to print over a few lines, e.g.:
s = 'blabla: '
s += 'a = %d | ' % a
s += 'b = %d | ' % b
print (s)
Ultimately I had to replace all of that with calls to this:
def concat_str (a,b):
return '%s%s' % (a, b)
Otherwise completely unrelated changes to math would make pythran crash with a message about some string concatenation line being broken, when it most certainly was not.
A whole ton of other crashes too, that were not caused by broken/buggy python code or by wrong pythran directives.
What is it about this function that confuses pythran so much?
Slowdown issue fixed, I'd be grateful if you could isolate the other errors and submit issues for them, it helps me a lot!
Fantastic, thank you!
I'll try to isolate the other issues but at this time I'm not 100% sure that I'll be able to.
After I encountered the issues I probably changed things significantly before committing.
In any case it'll likely be at least a few weeks before I can look into it.