Test suite fails
Opened this issue · 4 comments
jappeace commented
Hi, I'm trying to build grenade with nix, the test suite fails however:
━━━ Test.Grenade.Recurrent.Layers.LSTM ━━━
✗ prop_lstm_reference_forwards failed after 54 tests and 55 shrinks.
┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
44 ┃ prop_lstm_reference_forwards =
45 ┃ property $ do
46 ┃ input :: S.R 3 <- forAll randomVector
┃ │ (vector54319561065,0.7925737992825796,-6.308042121263235e-2] :: R 3)
47 ┃ cell :: S.R 2 <- forAll randomVector
┃ │ (vector102552342276,-0.5733412138993579] :: R 2)
48 ┃ net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
┃ │ LSTM
49 ┃
50 ┃ let actual = runRecurrentForwards net (S1D cell) (S1D input)
51 ┃ case actual of
52 ┃ (_, (S1D cellOut) :: S ('D1 2), (S1D output) :: S ('D1 2)) ->
53 ┃ let cellOut' = Reference.Vector . H.toList . S.extract $ cellOut
54 ┃ output' = Reference.Vector . H.toList . S.extract $ output
55 ┃ refNet = Reference.lstmToReference lstmWeights
56 ┃ refCell = Reference.Vector . H.toList . S.extract $ cell
57 ┃ refInput = Reference.Vector . H.toList . S.extract $ input
58 ┃ (refCO, refO) = Reference.runLSTM refNet refCell refInput
59 ┃ in do toList refCO ~~~ toList cellOut'
┃ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
┃ │ Failed (- lhs ~/~ + rhs)
┃ │ - [ 8.600932134870043e-2 , 0.10405281905297237 ]
┃ │ + [ 8.016375201087303e-2 , 7.653272311821224e-2 ]
60 ┃ toList refO ~~~ toList output'
This failure can be reproduced by running:
> recheck (Size 53) (Seed 10790320388850313198 5063647523751707663) prop_lstm_reference_forwards
✗ prop_lstm_reference_backwards failed after 58 tests and 36 shrinks.
┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
63 ┃ prop_lstm_reference_backwards =
64 ┃ property $ do
65 ┃ input :: S.R 3 <- forAll randomVector
┃ │ (vector55046569861,0.7353593240191039,0.10220169420456582] :: R 3)
66 ┃ cell :: S.R 2 <- forAll randomVector
┃ │ (vector94712659202,8.446059985294041e-2] :: R 2)
67 ┃ net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
┃ │ LSTM
68 ┃ let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
69 ┃ = runRecurrentForwards net (S1D cell) (S1D input)
70 ┃ actualBacks = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
71 ┃ case actualBacks of
72 ┃ (actualGradients, _, _ :: S ('D1 3)) ->
73 ┃ let refNet = Reference.lstmToReference lstmWeights
74 ┃ refCell = Reference.Vector . H.toList . S.extract $ cell
75 ┃ refInput = Reference.Vector . H.toList . S.extract $ input
76 ┃ refGradients = Reference.runLSTMback refCell refInput refNet
77 ┃ in toList refGradients ~~~ toList (Reference.lstmToReference actualGradients)
┃ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
┃ │ Failed (- lhs ~/~ + rhs)
┃ │ [
┃ │ - -0.2833762365900625
┃ │ + -0.2772291966623182
┃ │ - , 0.2792387920721928
┃ │ + , 0.273181502212966
┃ │ - , 3.8809159964732e-2
┃ │ + , 3.796730583209162e-2
┃ │ - , -1.8789963351874522e-2
┃ │ + , -1.9046776736865983e-2
┃ │ - , 1.851561984376432e-2
┃ │ + , 1.8768683616069006e-2
┃ │ - , 2.5733374902189056e-3
┃ │ + , 2.60850879413327e-3
┃ │ - , 0.3522269593369693
┃ │ + , 0.34458639918017514
┃ │ - , 3.2072315003399895e-2
┃ │ + , 3.1376597524498194e-2
┃ │ - , 2.3355281081871578e-2
┃ │ + , 2.36744913261781e-2
┃ │ - , 2.1266342964228386e-3
┃ │ + , 2.1557002473284216e-3
┃ │ - , 0.3797310824129001
┃ │ + , 0.3714938986832906
┃ │ - , 2.51790100961354e-2
┃ │ + , 2.5523146308241297e-2
┃ │ - , -0.14407943439166543
┃ │ + , -0.14095404163950748
┃ │ - , 0.14197579763956925
┃ │ + , 0.13889603729210484
┃ │ - , 1.9732077340779888e-2
┃ │ + , 1.930404615252405e-2
┃ │ - , 0.10233133248392495
┃ │ + , 0.10372995447129899
┃ │ - , -0.10083724033390323
┃ │ + , -0.1022154416927028
┃ │ - , -1.401455922896567e-2
┃ │ + , -1.4206104381415103e-2
┃ │ - , 0.17908580369842605
┃ │ + , 0.1752010475203034
┃ │ - , 1.6306804906884104e-2
┃ │ + , 1.5953075243230058e-2
┃ │ - , -0.12719434247358946
┃ │ + , -0.12893278171536413
┃ │ - , -1.1581785295885263e-2
┃ │ + , -1.1740080308514117e-2
┃ │ - , 0.19306996321689512
┃ │ + , 0.1888818605480774
┃ │ - , -0.13712648638597202
┃ │ + , -0.13900067402973101
┃ │ - , -0.11252340904695016
┃ │ + , -0.131574427655387
┃ │ - , 0.11088050713147728
┃ │ + , 0.12965337068552493
┃ │ - , 1.5410392325159424e-2
┃ │ + , 1.80194820553455e-2
┃ │ - , 4.787291093695013e-2
┃ │ + , 4.6151157876618276e-2
┃ │ - , -4.71739408493592e-2
┃ │ + , -4.5477326303979465e-2
┃ │ - , -6.556327661366858e-3
┃ │ + , -6.320528814073785e-3
┃ │ - , 0.1398627446668094
┃ │ + , 0.16354250849410923
┃ │ - , 1.2735317059880666e-2
┃ │ + , 1.489149739912809e-2
┃ │ - , -5.9504389135933686e-2
┃ │ + , -5.736430903441441e-2
┃ │ - , -5.418221013079177e-3
┃ │ + , -5.22335426217051e-3
┃ │ - , 0.1507841180627999
┃ │ + , 0.17631294858261248
┃ │ - , -6.415087061319925e-2
┃ │ + , -6.184367943473307e-2
┃ │ - , -0.6893313089947273
┃ │ + , -0.6743782306038787
┃ │ - , 0.6792667034381152
┃ │ + , 0.6645319479841436
┃ │ - , 9.440583078310466e-2
┃ │ + , 9.235796530850202e-2
┃ │ - , -0.7491175754191286
┃ │ + , -0.7593562021102653
┃ │ - , 0.7381800584171601
┃ │ + , 0.7482691956860943
┃ │ - , 0.10259372545373369
┃ │ + , 0.10399593371881949
┃ │ - , 0.9237208005000677
┃ │ + , 0.9036833100206666
┃ │ - , 1.00383585861486
┃ │ + , 1.0175558686009876
┃ │ ]
This failure can be reproduced by running:
> recheck (Size 57) (Seed 15557724053009315116 9838685224472302247) prop_lstm_reference_backwards
✗ prop_lstm_reference_backwards_input failed after 73 tests and 51 shrinks.
┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
79 ┃ prop_lstm_reference_backwards_input =
80 ┃ property $ do
81 ┃ input :: S.R 3 <- forAll randomVector
┃ │ (vector132148291512,0.2850784935453341,-1.40887764348131e-2] :: R 3)
82 ┃ cell :: S.R 2 <- forAll randomVector
┃ │ (vector429500642426,0.7481729899291754] :: R 2)
83 ┃ net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
┃ │ LSTM
84 ┃ let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
85 ┃ = runRecurrentForwards net (S1D cell) (S1D input)
86 ┃ actualBacks = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
87 ┃ case actualBacks of
88 ┃ (_, _, S1D actualGradients :: S ('D1 3)) ->
89 ┃ let refNet = Reference.lstmToReference lstmWeights
90 ┃ refCell = Reference.Vector . H.toList . S.extract $ cell
91 ┃ refInput = Reference.Vector . H.toList . S.extract $ input
92 ┃ refGradients = Reference.runLSTMbackOnInput refCell refNet refInput
93 ┃ in toList refGradients ~~~ H.toList (S.extract actualGradients)
┃ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
┃ │ Failed (- lhs ~/~ + rhs)
┃ │ [
┃ │ - 0.41783498961490245
┃ │ + -7.144093770897443e-2
┃ │ - , -0.643583384088483
┃ │ + , -0.5033482252382088
┃ │ - , 0.24741239536602957
┃ │ + , 0.2196259366479876
┃ │ ]
This failure can be reproduced by running:
> recheck (Size 72) (Seed 15511439570864369243 14061565670218007919) prop_lstm_reference_backwards_input
✗ prop_lstm_reference_backwards_cell failed after 45 tests and 31 shrinks.
┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
95 ┃ prop_lstm_reference_backwards_cell =
96 ┃ property $ do
97 ┃ input :: S.R 3 <- forAll randomVector
┃ │ (vector957622892669,-0.26370740740732634,-0.8739678849810585] :: R 3)
98 ┃ cell :: S.R 2 <- forAll randomVector
┃ │ (vector544704658233,-0.82774989764567] :: R 2)
99 ┃ net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
┃ │ LSTM
100 ┃ let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
101 ┃ = runRecurrentForwards net (S1D cell) (S1D input)
102 ┃ actualBacks = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
103 ┃ case actualBacks of
104 ┃ (_, S1D actualGradients, _ :: S ('D1 3)) ->
105 ┃ let refNet = Reference.lstmToReference lstmWeights
106 ┃ refCell = Reference.Vector . H.toList . S.extract $ cell
107 ┃ refInput = Reference.Vector . H.toList . S.extract $ input
108 ┃ refGradients = Reference.runLSTMbackOnCell refInput refNet refCell
109 ┃ in toList refGradients ~~~ H.toList (S.extract actualGradients)
┃ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
┃ │ Failed (- lhs ~/~ + rhs)
┃ │ - [ 0.3555130556015041 , 4.6592664606872156e-2 ]
┃ │ + [ 0.4057801473816573 , -1.403610494537344e-2 ]
110 ┃
111 ┃ (~~~) :: (Monad m, Eq a, Ord a, Num a, Fractional a, Show a, HasCallStack) => [a] -> [a] -> PropertyT m ()
112 ┃ (~~~) x y =
113 ┃ if all (< 1e-8) (zipWith (-) x y) then
114 ┃ success
115 ┃ else
116 ┃ case valueDiff <$> mkValue x <*> mkValue y of
117 ┃ Nothing ->
118 ┃ withFrozenCallStack $
119 ┃ failWith Nothing $ unlines [
120 ┃ "━━━ Not Simliar ━━━"
121 ┃ , showPretty x
122 ┃ , showPretty y
123 ┃ ]
124 ┃ Just diff ->
125 ┃ withFrozenCallStack $
126 ┃ failWith (Just $ Diff "Failed (" "- lhs" "~/~" "+ rhs" ")" diff) ""
This failure can be reproduced by running:
> recheck (Size 44) (Seed 17734744653642261044 4653266027254713753) prop_lstm_reference_backwards_cell
✗ 4 failed.
Test suite test: FAIL
Test suite logged to: dist/test/grenade-0.1.0-test.log
0 of 1 test suites (0 of 1 test cases) passed.
builder for '/nix/store/g2ishp6kxmakfxpxxl0l58m0n6w7dfzi-grenade-0.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv' failed
make[1]: *** [makefile:70: enter] Fout 100
make[1]: Map '/home/jappie/projects/raster' wordt verlaten
I used cabal2nix to just grab the latest version from github, this one to be precise:
{ mkDerivation, ad, base, bytestring, cereal, constraints
, containers, criterion, deepseq, fetchgit, hedgehog, hmatrix
, MonadRandom, mtl, primitive, random, reflection, singletons
, stdenv, text, transformers, typelits-witnesses, vector
}:
mkDerivation {
pname = "grenade";
version = "0.1.0";
src = fetchgit {
url = "https://github.com/HuwCampbell/grenade";
sha256 = "1nzzacpiz86z3svvi2lpczzkv4algjfw3m2h2pqsw5n44xp8b320";
rev = "f729b291e40a9044d95d49d31bce3149a5b9d599";
fetchSubmodules = true;
};
libraryHaskellDepends = [
base bytestring cereal containers deepseq hmatrix MonadRandom
primitive singletons vector
];
testHaskellDepends = [
ad base constraints hedgehog hmatrix MonadRandom mtl random
reflection singletons text transformers typelits-witnesses vector
];
benchmarkHaskellDepends = [ base bytestring criterion hmatrix ];
description = "Practical Deep Learning in Haskell";
license = stdenv.lib.licenses.bsd2;
}I'll disable the tests for now so it's not a big deal, but I just wanted to let you know.
erikd commented
I don't use Nix, but i just tested building this with ghc-8.6.3 and cabal-install 2.0.0.1.
I built it and ran the tests using:
./mafia build -w
./mafia test
Unless this can be reproduced outside Nix, I can't work on it.
HuwCampbell commented
I’m running NixOS at home at the moment, I’ll try running it there. Do you
have a pure derivation I can test against? (That is, one including all
dependencies)
…On Thu, 7 Mar 2019 at 12:06 pm, Erik de Castro Lopo < ***@***.***> wrote:
I don't use Nix, but i just tested building this with ghc-8.6.3 and
cabal-install 2.0.0.1.
I built it and ran the tests using:
./mafia build -w
./mafia test
Unless this can be reproduced outside Nix, I can't work on it.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#87 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE9t0e5-q86Ae2wukXOcgME527fov_UKks5vUGYOgaJpZM4bhqww>
.
jappeace commented
This project fails reproducibly: https://github.com/jappeace/awesome-project-name/tree/show-grenade-fails
You can't enter the shell unless you disable the tests.
Enter shell with:
nix-shell -A shells.ghcjappeace commented
I'm not sure how to make a pure derivation, but since the project I linked is pinned, that should be good enough?