angles tests
mmann1123 opened this issue · 2 comments
mmann1123 commented
@jgrss On conda for the py3.8 and py3.9 builds I am getting the following error. Just wondering if I am missing something
For linux:
=================================== FAILURES ===================================
_________________________ TestRadiometrics.test_angles _________________________
self = <tests.test_rad.TestRadiometrics testMethod=test_angles>
def test_angles(self):
subsample = 10
nodata = -32768
resampling = 'bilinear'
df = get_pathrow()
with tempfile.TemporaryDirectory() as gz_tmp:
angle_paths = extract_espa_tools(gz_tmp)
with tempfile.TemporaryDirectory() as tmp_full:
(
out_path,
angle_paths_out,
l57_angles_path,
l8_angles_path,
) = landsat_angle_prep(
ref_file=l7_225078_20110306_B1,
out_dir=tmp_full,
l57_angles_path=str(angle_paths.l57_angles_path),
l8_angles_path=str(angle_paths.l8_angles_path),
)
> angle_paths_in = run_espa_command(
angle_paths_out.vaa,
str(l7_225078_20110306_ang),
'l7',
str(angle_paths.l57_angles_path),
str(angle_paths.l8_angles_path),
subsample,
out_path,
0,
)
$SRC_DIR/tests/test_rad.py:116:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ref_angle_file = '/tmp/tmp_0ogvkrt/LE07_L2SP_225078_20110306_02_T1_sensor_azimuth.tif'
angles_file = '/home/conda/feedstock_root/build_artifacts/geowombat_1682086414215/_test_env_placehold_placehold_placehold_placehold_...ython3.8/site-packages/geowombat/data/LE07_L2SP_225078_20110306_02_T1_LE07_L2SP_225078_20110306_20200910_02_T1_ANG.txt'
sensor = 'l7', l57_angles_path = '/tmp/tmp1dwnbevl/ESPA/landsat_angles'
l8_angles_path = '/tmp/tmp1dwnbevl/ESPA/l8_angles', subsample = 10
out_dir = PosixPath('/tmp/tmp_0ogvkrt'), verbose = 0
def run_espa_command(
ref_angle_file: T.Union[str, Path],
angles_file: T.Union[str, Path],
sensor: str,
l57_angles_path: T.Union[str, Path],
l8_angles_path: T.Union[str, Path],
subsample: T.Union[float, int],
out_dir: T.Union[str, Path],
verbose: int,
) -> namedtuple:
AnglePaths = namedtuple('AnglePaths', 'sensor solar out_order')
angle_paths = AnglePaths(sensor=None, solar=None, out_order=None)
with tempfile.TemporaryDirectory() as tmp_dir:
if not Path(ref_angle_file).is_file():
# Setup the command.
if sensor.lower() in ['l5', 'l7']:
angle_command = '{PATH} {META} -s {SUBSAMP:d} -b 1'.format(
PATH=str(Path(l57_angles_path).joinpath('landsat_angles')),
META=angles_file,
SUBSAMP=subsample,
)
# 1=zenith, 2=azimuth
out_order = dict(azimuth=2, zenith=1)
# out_order = [2, 1, 2, 1]
else:
angle_command = (
'{PATH} {META} BOTH {SUBSAMP:d} -f -32768 -b 4'.format(
PATH=str(Path(l8_angles_path).joinpath('l8_angles')),
META=angles_file,
SUBSAMP=subsample,
)
)
# 1=azimuth, 2=zenith
out_order = dict(azimuth=1, zenith=2)
----------------------------- Captured stderr call -----------------------------
*** buffer overflow detected ***: /tmp/tmp1dwnbevl/ESPA/landsat_angles/landsat_angles terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fc497cd57a7]
/lib64/libc.so.6(+0x116922)[0x7fc497cd3922]
/tmp/tmp1dwnbevl/ESPA/landsat_angles/landsat_angles(+0x1bdb)[0x5574a3c01bdb]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc497bdf555]
/tmp/tmp1dwnbevl/ESPA/landsat_angles/landsat_angles(+0x25aa)[0x5574a3c025aa]
======= Memory map: ========
5574a3c00000-5574a3c14000 r-xp 00000000 00:2c 3635533 /tmp/tmp1dwnbevl/ESPA/landsat_angles/landsat_angles
5574a3e13000-5574a3e14000 r--p 00013000 00:2c 3635533 /tmp/tmp1dwnbevl/ESPA/landsat_angles/landsat_angles
5574a3e14000-5574a3e15000 rw-p 00014000 00:2c 3635533 /tmp/tmp1dwnbevl/ESPA/landsat_angles/landsat_angles
5574a3e15000-5574a3e16000 rw-p 00000000 00:00 0
5574a49c4000-5574a49e5000 rw-p 00000000 00:00 0 [heap]
7fc4979a7000-7fc4979bc000 r-xp 00000000 00:2c 1106793 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc4979bc000-7fc497bbb000 ---p 00015000 00:2c 1106793 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc497bbb000-7fc497bbc000 r--p 00014000 00:2c 1106793 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc497bbc000-7fc497bbd000 rw-p 00015000 00:2c 1106793 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fc497bbd000-7fc497d81000 r-xp 00000000 00:2c 1376228 /usr/lib64/libc-2.17.so
7fc497d81000-7fc497f80000 ---p 001c4000 00:2c 1376228 /usr/lib64/libc-2.17.so
7fc497f80000-7fc497f84000 r--p 001c3000 00:2c 1376228 /usr/lib64/libc-2.17.so
7fc497f84000-7fc497f86000 rw-p 001c7000 00:2c 1376228 /usr/lib64/libc-2.17.so
7fc497f86000-7fc497f8b000 rw-p 00000000 00:00 0
7fc497f8b000-7fc49808c000 r-xp 00000000 00:2c 1376279 /usr/lib64/libm-2.17.so
7fc49808c000-7fc49828b000 ---p 00101000 00:2c 1376279 /usr/lib64/libm-2.17.so
7fc49828b000-7fc49828c000 r--p 00100000 00:2c 1376279 /usr/lib64/libm-2.17.so
7fc49828c000-7fc49828d000 rw-p 00101000 00:2c 1376279 /usr/lib64/libm-2.17.so
7fc49828d000-7fc4982af000 r-xp 00000000 00:2c 1376219 /usr/lib64/ld-2.17.so
7fc4984a6000-7fc4984a9000 rw-p 00000000 00:00 0
7fc4984ac000-7fc4984ae000 rw-p 00000000 00:00 0
7fc4984ae000-7fc4984af000 r--p 00021000 00:2c 1376219 /usr/lib64/ld-2.17.so
7fc4984af000-7fc4984b0000 rw-p 00022000 00:2c 1376219 /usr/lib64/ld-2.17.so
7fc4984b0000-7fc4984b1000 rw-p 00000000 00:00 0
7ffcad1f1000-7ffcad213000 rw-p 00000000 00:00 0 [stack]
7ffcad314000-7ffcad318000 r--p 00000000 00:00 0 [vvar]
7ffcad318000-7ffcad31a000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
=========================== short test summary info ============================
FAILED tests/test_rad.py::TestRadiometrics::test_angles - ValueError: The angles were not created.
========================= 1 failed, 2 passed in 16.34s =========================
Tests failed for geowombat-2.1.8-py38h01eb140_0.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken
WARNING:conda_build.build:Tests failed for geowombat-2.1.8-py38h01eb140_0.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken
TESTS FAILED: geowombat-2.1.8-py38h01eb140_0.conda
for OSX
self = <tests.test_rad.TestRadiometrics testMethod=test_angles>
def test_angles(self):
subsample = 10
nodata = -32768
resampling = 'bilinear'
df = get_pathrow()
with tempfile.TemporaryDirectory() as gz_tmp:
angle_paths = extract_espa_tools(gz_tmp)
with tempfile.TemporaryDirectory() as tmp_full:
(
out_path,
angle_paths_out,
l57_angles_path,
l8_angles_path,
) = landsat_angle_prep(
ref_file=l7_225078_20110306_B1,
out_dir=tmp_full,
l57_angles_path=str(angle_paths.l57_angles_path),
l8_angles_path=str(angle_paths.l8_angles_path),
)
> angle_paths_in = run_espa_command(
angle_paths_out.vaa,
str(l7_225078_20110306_ang),
'l7',
str(angle_paths.l57_angles_path),
str(angle_paths.l8_angles_path),
subsample,
out_path,
0,
)
$SRC_DIR/tests/test_rad.py:116:
ref_angle_file = '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpgr4dn1y7/LE07_L2SP_225078_20110306_02_T1_sensor_azimuth.tif'
angles_file = '/Users/runner/miniforge3/conda-bld/geowombat_1682086647616/_test_env_placehold_placehold_placehold_placehold_placehol...ython3.8/site-packages/geowombat/data/LE07_L2SP_225078_20110306_02_T1_LE07_L2SP_225078_20110306_20200910_02_T1_ANG.txt'
sensor = 'l7'
l57_angles_path = '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpazcybidb/ESPA/landsat_angles'
l8_angles_path = '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpazcybidb/ESPA/l8_angles'
subsample = 10
out_dir = PosixPath('/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpgr4dn1y7')
verbose = 0
def run_espa_command(
ref_angle_file: T.Union[str, Path],
angles_file: T.Union[str, Path],
sensor: str,
l57_angles_path: T.Union[str, Path],
l8_angles_path: T.Union[str, Path],
subsample: T.Union[float, int],
out_dir: T.Union[str, Path],
verbose: int,
) -> namedtuple:
AnglePaths = namedtuple('AnglePaths', 'sensor solar out_order')
angle_paths = AnglePaths(sensor=None, solar=None, out_order=None)
with tempfile.TemporaryDirectory() as tmp_dir:
if not Path(ref_angle_file).is_file():
# Setup the command.
if sensor.lower() in ['l5', 'l7']:
angle_command = '{PATH} {META} -s {SUBSAMP:d} -b 1'.format(
PATH=str(Path(l57_angles_path).joinpath('landsat_angles')),
META=angles_file,
SUBSAMP=subsample,
)
# 1=zenith, 2=azimuth
out_order = dict(azimuth=2, zenith=1)
# out_order = [2, 1, 2, 1]
else:
angle_command = (
'{PATH} {META} BOTH {SUBSAMP:d} -f -32768 -b 4'.format(
PATH=str(Path(l8_angles_path).joinpath('l8_angles')),
META=angles_file,
SUBSAMP=subsample,
)
)
# 1=azimuth, 2=zenith
out_order = dict(azimuth=1, zenith=2)
# out_order = [1, 2, 1, 2]
if verbose > 0:
logger.info(' Generating pixel angles ...')
# Create the angle files.
os.chdir(str(tmp_dir))
subprocess.call(angle_command, shell=True)
os.chdir(os.path.expanduser('~'))
# Get angle data from 1 band.
sensor_angle_file = list(Path(tmp_dir).glob('*sensor_B04.img'))
solar_angles_file = list(Path(tmp_dir).glob('*solar_B04.img'))
if not sensor_angle_file or not solar_angles_file:
> raise ValueError('The angles were not created.')
E ValueError: The angles were not created.
$PREFIX/lib/python3.8/site-packages/geowombat/radiometry/angles.py:491: ValueError
----------------------------- Captured stderr call -----------------------------
/bin/sh: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpazcybidb/ESPA/landsat_angles/landsat_angles: cannot execute binary file
=========================== short test summary info ============================
FAILED tests/test_rad.py::TestRadiometrics::test_angles - ValueError: The angles were not created.
========================= 1 failed, 2 passed in 32.72s =========================
Tests failed for geowombat-2.1.8-py38hcafd530_0.conda - moving package to /Users/runner/miniforge3/conda-bld/broken
WARNING:conda_build.build:Tests failed for geowombat-2.1.8-py38hcafd530_0.conda - moving package to /Users/runner/miniforge3/conda-bld/broken
TESTS FAILED: geowombat-2.1.8-py38hcafd530_0.conda
##[error]Bash exited with code '1'.
Finishing: Run OSX build
jgrss commented
These are errors from the ESPA angle creation tools. I think they should only work on Linux, so that would explain errors on OSX. Not sure why they're failing on Linux for you. Do you happen to know which flavor of Linux you're using? We might need to run make on different systems because the ESPA tarball contains pre-compiled code.
mmann1123 commented
Maybe the RAM is limited on the azure service?? Hence the ======= Memory map: ======= and buffer overflow detected?
Here's the azure pipeline for linux, its ubuntu