OverflowError: Python int too large to convert to C long
Closed this issue · 8 comments
SteveHere commented
Essentially, it craps out when trying to do 13/13 (2**12).
Error (& tracceback) produced:
Traceback (most recent call last):
File "C:\Users\User\Documents\JWD\Python\side_projects\conflict_show_benchmark.py", line 72, in <module>
equality_check=lambda x,y: x == y
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\perfplot\main.py", line 205, in show
out = bench(*args, **kwargs)
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\perfplot\main.py", line 142, in bench
timings[k, i] = t
OverflowError: Python int too large to convert to C long
Code used:
import functools
import itertools
import operator
import random
from collections import defaultdict
from datetime import datetime
from typing import Tuple, DefaultDict, List, Any, Set
import perfplot
def overlap(interval_1: Tuple[int, datetime, datetime], interval_2: Tuple[int, datetime, datetime]) -> bool:
return interval_1[1] < interval_2[2] and interval_2[1] < interval_1[2]
def list_conflicts_1(events: List[Tuple[int, datetime, datetime]]) -> DefaultDict[Any, List]:
results = defaultdict(list)
for i in range(len(events)):
event_selected = events[i]
for e in itertools.islice(events, i):
if overlap(event_selected, e):
results[e[0]].append(event_selected[0])
results[event_selected[0]].append(e[0])
return results
def list_conflicts_2(events: List[Tuple[int, datetime, datetime]]) -> DefaultDict[Any, List]:
results = defaultdict(list)
time_list = sorted(functools.reduce(operator.iconcat,
list(([('s', id, start), ('e', id, end)] for id, start, end in events))
, []), key=lambda a: a[2])
stack = []
for token, id, time in time_list:
if token == 's':
stack.append(id)
if len(stack) > 1:
for i in stack[:-1]:
results[i].append(id)
results[id].append(i)
if token == 'e':
if id in stack:
stack.remove(id)
return results
create_dt = lambda a, b: datetime.now().replace(hour=random.randrange(a, b))
create_dt_h = lambda x: datetime.now().replace(hour=x)
# This portion is a sanity test: Can be ignored
sanity_test_case = sorted([
(1, create_dt_h(4), create_dt_h(7)),
(2, create_dt_h(5), create_dt_h(8)),
(3, create_dt_h(9), create_dt_h(12)),
(4, create_dt_h(13), create_dt_h(14)),
(5, create_dt_h(9), create_dt_h(15)),
(6, create_dt_h(10), create_dt_h(17)),
], key=lambda x: x[1])
assert (list_conflicts_1(sanity_test_case) == list_conflicts_2(sanity_test_case))
# Worst case scenario
perfplot.show(
setup=lambda n: sorted(((i, create_dt(1, 12), create_dt(13, 23)) for i in range(n)), key=lambda a: a[1]),
kernels=[list_conflicts_1, list_conflicts_2],
n_range=[2 ** k for k in range(13)],
logx=True,
logy=False,
xlabel='num lists',
equality_check=lambda x, y: x == y
)nschloe commented
Runs fine here. Output:
(I set logy=True.)
Not sure how the error can occur on your computer. Is sys.maxsize super small?
SteveHere commented
I thought it was that, so I updated to 3.7.4
C:\Users\User\Documents\Doc\Python\side_projects>python
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxsize
9223372036854775807
>>>
Output is still the same:
C:\Users\User\Documents\Doc\Python\side_projects>python conflict_show_benchmark.py
0%| | 0/13 [00:00<?, ?it/s]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:00<00:00, 1.83it/s]
8%|██████▍ | 1/13 [00:00<00:11, 1.05it/s]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:00<00:00, 1.68it/s]
15%|████████████▊ | 2/13 [00:02<00:10, 1.01it/s]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:00<00:00, 1.39it/s]
23%|███████████████████▏ | 3/13 [00:03<00:10, 1.10s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:00<00:00, 1.12it/s]
31%|█████████████████████████▌ | 4/13 [00:05<00:11, 1.27s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:01<00:01, 1.02s/it]
38%|███████████████████████████████▉ | 5/13 [00:06<00:11, 1.45s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:01<00:01, 1.03s/it]
46%|██████████████████████████████████████▎ | 6/13 [00:08<00:11, 1.61s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:01<00:01, 1.03s/it]
54%|████████████████████████████████████████████▋ | 7/13 [00:10<00:10, 1.73s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:01<00:01, 1.02s/it]
62%|███████████████████████████████████████████████████ | 8/13 [00:12<00:09, 1.82s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:01<00:01, 1.08s/it]
69%|█████████████████████████████████████████████████████████▍ | 9/13 [00:15<00:07, 1.90s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:01<00:01, 1.05s/it]
77%|███████████████████████████████████████████████████████████████ | 10/13 [00:17<00:05, 1.97s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:01<00:01, 1.20s/it]
85%|█████████████████████████████████████████████████████████████████████▍ | 11/13 [00:19<00:04, 2.16s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|██████████████████████████████████████████ | 1/2 [00:02<00:02, 2.52s/it]
92%|███████████████████████████████████████████████████████████████████████████▋ | 12/13 [00:24<00:02, 2.96s/it]
0%| | 0/2 [00:00<?, ?it/s]Traceback (most recent call last):
File "conflict_show_benchmark.py", line 72, in <module>
equality_check=lambda x, y: x == y
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\site-packages\perfplot\main.py", line 205, in show
out = bench(*args, **kwargs)
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\site-packages\perfplot\main.py", line 142, in bench
timings[k, i] = t
OverflowError: Python int too large to convert to C long
nschloe commented
Absolutely no idea why this could be and since I cannot reproduce it, you'll have to do some more digging. Which of k, i is large and why?
SteveHere commented
...I think I know what the problem is. But first, the output. print(f"\n{k}, {i} = {t}") is inserted before perfplot's line 142: timings[k, i] = t.
Output:
C:\Users\User\Documents\Doc\Python\side_projects>python conflict_show_benchmark.py
0%| | 0/13 [00:00<?, ?it/s]
0%| | 0/2 [00:00<?, ?it/s]
0, 0 = 853
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 3.25it/s]
1, 0 = 2133
8%|█████▌ | 1/13 [00:00<00:08, 1.43it/s]
0%| | 0/2 [00:00<?, ?it/s]
0, 1 = 2133
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 7.48it/s]
1, 1 = 3839
15%|███████████▏ | 2/13 [00:01<00:07, 1.46it/s]
0%| | 0/2 [00:00<?, ?it/s]
0, 2 = 5546
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.29it/s]
1, 2 = 7253
23%|████████████████▊ | 3/13 [00:02<00:08, 1.15it/s]
0%| | 0/2 [00:00<?, ?it/s]
0, 3 = 17920
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.08it/s]
1, 3 = 16639
31%|██████████████████████▍ | 4/13 [00:04<00:09, 1.11s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 4 = 66133
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.51it/s]
1, 4 = 44373
38%|████████████████████████████ | 5/13 [00:05<00:09, 1.23s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 5 = 258133
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.05s/it]
1, 5 = 136106
46%|█████████████████████████████████▋ | 6/13 [00:07<00:10, 1.45s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 6 = 1022719
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.03it/s]
1, 6 = 468053
54%|███████████████████████████████████████▎ | 7/13 [00:09<00:09, 1.60s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 7 = 4195412
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.13it/s]
1, 7 = 1767679
62%|████████████████████████████████████████████▉ | 8/13 [00:11<00:08, 1.70s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 8 = 17044473
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.01s/it]
1, 8 = 6825384
69%|██████████████████████████████████████████████████▌ | 9/13 [00:13<00:07, 1.81s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 9 = 71459809
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.03s/it]
1, 9 = 28675401
77%|███████████████████████████████████████████████████████▍ | 10/13 [00:15<00:05, 1.92s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 10 = 294163074
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.20s/it]
1, 10 = 115212325
85%|████████████████████████████████████████████████████████████▉ | 11/13 [00:18<00:04, 2.12s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 11 = 1217587107
50%|█████████████████████████████████████ | 1/2 [00:02<00:02, 2.47s/it]
1, 11 = 479134943
92%|██████████████████████████████████████████████████████████████████▍ | 12/13 [00:23<00:03, 3.04s/it]
0%| | 0/2 [00:00<?, ?it/s]
0, 12 = 4920614861
Traceback (most recent call last):
File "conflict_show_benchmark.py", line 82, in <module>
equality_check=lambda x, y: x == y
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\site-packages\perfplot\main.py", line 206, in show
out = bench(*args, **kwargs)
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\site-packages\perfplot\main.py", line 143, in bench
timings[k, i] = t
OverflowError: Python int too large to convert to C long
Python is unable to fit 4920614861 into C's long (Range: −2,147,483,647 to +2,147,483,647), which is why OverflowError occurs.
nschloe commented
Okay, interesting. That means that int isn't numpy.int64 on your machine. It's nice that your python actually throws an error there instead of silently overflowing.
The timings array is
timings = numpy.empty((len(kernels), len(n_range)), dtype=int)Can you try setting that to dtype=numpy.int64 and see what happens?
SteveHere commented
Used dtype=numpy.uint64 in order to give it a little more headroom. Got it working now.
Output:
0%| | 0/13 [00:00<?, ?it/s]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.41it/s]
8%|█████▌ | 1/13 [00:01<00:13, 1.11s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 3.15it/s]
15%|███████████▏ | 2/13 [00:01<00:11, 1.03s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.32it/s]
23%|████████████████▊ | 3/13 [00:03<00:11, 1.16s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.09it/s]
31%|██████████████████████▍ | 4/13 [00:05<00:11, 1.33s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.00it/s]
38%|████████████████████████████ | 5/13 [00:06<00:11, 1.46s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.03s/it]
46%|█████████████████████████████████▋ | 6/13 [00:08<00:11, 1.62s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.05s/it]
54%|███████████████████████████████████████▎ | 7/13 [00:10<00:10, 1.74s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.03s/it]
62%|████████████████████████████████████████████▉ | 8/13 [00:12<00:09, 1.82s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.02s/it]
69%|██████████████████████████████████████████████████▌ | 9/13 [00:14<00:07, 1.88s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:00<00:00, 1.13it/s]
77%|███████████████████████████████████████████████████████▍ | 10/13 [00:16<00:05, 1.90s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:01<00:01, 1.19s/it]
85%|████████████████████████████████████████████████████████████▉ | 11/13 [00:19<00:04, 2.08s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:02<00:02, 2.36s/it]
92%|██████████████████████████████████████████████████████████████████▍ | 12/13 [00:24<00:02, 2.97s/it]
0%| | 0/2 [00:00<?, ?it/s]
50%|█████████████████████████████████████ | 1/2 [00:09<00:09, 9.75s/it]
100%|████████████████████████████████████████████████████████████████████████| 13/13 [00:43<00:00, 7.73s/it]
Figure produced:
SteveHere commented
I'm going to make a pull request for this now.