ethereum/webthree-umbrella

Debian - Add support in install_deps.sh

bobsummerwill opened this issue · 56 comments

Hey @programonauta,

(moving here from Gitter, and let's chat about this as comments on this issue).

I have been working on an infrastructure-as-code alternative to these tedious per-distro sets of instructions, which will "go live" when we switch back to cpp-ethereum.

You can see my work-in-progress on that here:

I would like to expand this install_deps.sh pattern pattern to cover more distros, so that our ethdocs.org instructions for the distros which we have bothered to do that can be simplified to, with the steps being executed directly and committed in with the code itself.

  • git clone --recursive
  • ./install_deps.sh
  • cmake
  • make

So maybe for Debian we can take that approach instead, rather than bothering with written instructions?

Please could help me out and copy https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh locally and run it, does it go into the right conditional?

I think it does, and then we would be looking to add further command-lines into the shell script which replicate the process which was already documented by you. Thanks!

OK, @programonauta!

I have had a go at writing these steps "blind". Please could you re-sync and try these?

See https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh#L142.

You could even just copy https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh locally and try running it and report back, if the git flow is impeding you? Thanks!

Question for you, @programonauta.
Do we really need to install gcc and g++, as per your instructions?
Those aren't part of a vanilla Debian Jesse 8.5 install?

Hey @phonikg,

Please could you check out https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh?

Looking at your EthEmbedded scripts, I think that the Debian and Ubuntu steps which I have here are within a gnats-cock-width distance of what you have in:

With the differences likely boiling down to ensuring that the correct conditional paths are being followed for Raspbian, Ubuntu MATE, etc. on the SBCs which you have things working for, plus some tiny deltas on the packages being installed.

ie. you also have packages like ... dphys-swapfile, python, automake, binfmt-support, libffi-dev, libobjc-4.9-dev, libobjc4 ... which we may need to add into my script.

I've also not bothered getting LLVM built, so that EVMJIT can be enabled. We'll come back to that.

Anyway - would you like to join in the fun here? It will be WAY better than the tedious documentation work which is the alternative.

You've got some for C.H.I.P. too, right?

Please just grab https://github.com/bobsummerwill/cpp-ethereum, flip to the merge_repos branch and try install_deps.sh out on whatever you have, and hopefully we can have a grand fusion going in the near future.

NOTE - I'm only including packages need to BUILD for right now. We might well need more to RUN, especially for your use-cases. And I see you've also got some "development tools" in there too, ie. nano, which may or may not make sense in there.

You could even just copy https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh locally and try running it and report back, if the git flow is impeding you? Thanks!

@bobsummerwill, I'm justing working on it. I was based on old script, but I'll adjust to the new one.

Here my current script.

I'm planning some improvements, like test each step, giving a message and exit in case of error, like lin 99 here: https://gist.github.com/programonauta/89d96499815fe7430e4c34c8b146ca04. I'm getting an error here, but after I tested whole script I'll come back here to fix it.

Btw, is it possible have a direct link to a line or set of lines on Gist? Like you did on your comment?

Do we really need to install gcc and g++, as per your instructions?
Those aren't part of a vanilla Debian Jesse 8.5 install?

No @bobsummerwill, every time I start a test, I launch a new Debian from scratch on Virtual Box, and it doesn't have gcc and g++ installed.

Right. I've just updated https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh to add g++, gcc and unzip, then!

For the testing each step, conditionals checking existing versions, building/running tests for packages, etc, I would recommend that you don't bother with any of that at all. It will just be an unnecessary maintenance burden, and doesn't add any real value.

I went through that with the scripts which were build up for http://github.com/doublethinkco/cpp-ethereum-cpp and it's just a waste of time, IMHO.

We just need clean, simple, steps, each assumed to work.

So what I have there right now on that latest version of Debian is pretty clean and simple and I think that it will work. Install packages for everything except the two build-from-source.

I've not bothered with LLVM, because Pawel changed the way that worked in the EVMJIT package, so it builds-from-source within the package's own CMake files, rather than using any distro-specific package server.

So you should actually be able to build with -DEVMJIT=1 and it should work, I think, without needing to do anything to install LLVM.

For CMake, looking at the package versions for the distros, I think the built-in version should actually be OK, again after Pawel fixed the behavior which required 3.1.

Or not, actually, we DO need a newer one, don't we? That was that path expansion crap in the tests, wasn't it?

Right - we that is something which would need readding them, but just unconditionally, like the other build-from-source ones.

For CMake, looking at the package versions for the distros, I think the built-in version should actually be OK, again after Pawel fixed the behavior which required 3.1.

When we do apt-get install cmake the version installed is 3.0.2. Anyway we need to install a new one.

I've not bothered with LLVM, because Pawel changed the way that worked in the EVMJIT package, so it builds-from-source within the package's own CMake files, rather than using any distro-specific package server.

So as I understood, it's not necessary install LLVM. Correct?

For the testing each step, conditionals checking existing versions, building/running tests for packages, etc, I would recommend that you don't bother with any of that at all. It will just be an unnecessary maintenance burden, and doesn't add any real value.

Correct, I'll remove conditionals checking.

@bobsummerwill

I've finished Debian section to install_deps.sh

I started a Debian from scratch, and didn't install LLVM.

I tried to run cmake .. -DEVMJIT=1 but I got this error

Trying to run cmake .. I got the same error, so I remove build directory and started again.

The cmake run fine, but on make I got an old error. You can see in this link.

I got solve this error, with the same changing EthCompilerSettings.cmake commenting line 27 (add_compile_options(-Werror)), and including add_compile_options(-Wno-array-bounds) on line 104.

I know this is an workaround, but I really don't remember what we did to solve this.

Many thanks!

So I have just added an extra suppression for the CryptoPP unused function one. We fixed something like that before, but it wasn't exactly the same. I guess some code changed in the meantime to make this appear, or maybe just slightly different versions of something?

https://github.com/bobsummerwill/cpp-ethereum/commit/a832bcdc93f4b5dbf9c1650dc2f2dcb6184c21b7

The EVMJIT=1 error will be because you didn't do --recursive on your git clone. The repository is almost standalone now, but we have two optional ones left - evmjit (for this!) and deps (for Windows dependencies). You should just be able to do `git submodule update --init", I think, to fix that in-place.

Ok @bobsummerwill

Let's combine: I'll save my install_deps.sh, and start again with a new Debian. I have an image that I copied when the installation was finished and git installed.

After this:

  • delete my repo and fork it again
  • clone recursively
  • bring my install_deps.sh
  • run it
  • cmake .. -DEVMJIT=1
  • make
  • make install

After that I let you know what happens.
So, I will be sure we're on the same page.

It's a plan, thanks for all your help, @programonauta!

Hi @bobsummerwill,

EVMJIT=1 error vanished, but the error listed here remains.

It's a plan, thanks for all your help, @programonauta!

It's a pleasure for me @bobsummerwill. I'm learning a lot! 😄

Did you only do the git submodule update --init? That will have pulled in evmjit, but not pulled my edit for that warning.

That will need you to do a git pull upstream develop and then git submodule update. You did add the remote for the upstream already, right?

Getting Debian going will be really helpful for single board computers, where many of them are (or can) run Debian.

Hummm,
I'm still a bit confused with git stuffs.

Did you only do the git submodule update --init? That will have pulled in evmjit, but not pulled my edit for that warning.

No, I just did the steps I described on my previous message. I imagined if I just clone your repo I would be up to date.
But, as I could noticed, the code that fix the problem is on develop branch, correct?

What do you mean upstream? the main repo from ethereum? or from you?

Anyway, I have to work a bit more on my script. The Cmake installation is not working smoothly, the first time it runs, I received an error. So I needed to source ~/.profile to system understands the new CMake version and run CMake with no problems.
Or, consider that script must run with Version 3.5.2 or greater of CMake. What do you think about?

OK - So I've just looked at your Github setup.

So you forked your https://github.com/programonauta/cpp-ethereum from my https://github.com/bobsummerwill/cpp-ethereum. So my repository there is your "upstream".

When you do push and pull operations, those are between your clone of your fork, which is on your machine. You can commit on your machine as much as you like, and then when you git push, you are moving those changes up to your public version on Github.

What we need to do here is for you to pull new changes from ME, ie. from the upstream. From the grandparent :-) To do that, you add a "remote", which is a symbolic link to a remote repository. In this case, my repo.

So you will do this:

git remote add upstream https://github.com/bobsummerwill/cpp-ethereum

Which gives you a second remote to go with the one you have already called origin, which points to your Github repo.

Then you will do ...

git pull upstream merge_repos

... to pull my latest changes down onto your machine. Then if you git commit and git push then they would go back into your Github repo, where you could actually make a Pull Request which I could pull :-)

Hi @bobsummerwill

I did the pull upstream, in a new Debian. You can see the log here

The problem remains.

Does it have any relation with #pragma? If yes, there is nothing to do with install_deps.sh correct?

If yes, my Debian jessie section of install_deps.sh is ready.

Hey there!
So yeah, the commit which that shows includes my addition of that pragma in:

https://github.com/bobsummerwill/cpp-ethereum/commit/a832bcdc93f4b5dbf9c1650dc2f2dcb6184c21b7

So you are still seeing the warning with that code-base?

/home/ricardo/cpp-ethereum/cryptopp/misc.h:1074:20:
error: ‘std::string CryptoPP::StringNarrow(const wchar_t*, bool)’ defined but not used [-Werror=unused-function]

static std::string StringNarrow(const wchar_t *str, bool throwOnError = true)

I've just got an Ethereum Foundation Azure account set up, and can do some runs against other distros there myself now, so maybe I will try that myself tomorrow.

If you remove the -Werror in /cmake, do you get a clean build otherwise now? Using my vanilla install_deps.sh? Best wishes!

So you are still seeing the warning with that code-base?

I will try it at night. But seeing the messages and your commit, I suppose my last try was after your commit. My comment was July 10 15:12 GMT-3 and your commit on July 9 19:22 GMT-3.

If you remove the -Werror in /cmake, do you get a clean build otherwise now? Using my vanilla install_deps.sh? Best wishes!

I'm using your Cmake.txt with no changes. Anyway after my test at night I'll let you know.

OK - thanks!

Sorry! I see the issue. It looks like my PR for libweb3core for that warning suppression never got merged! AH! I was waiting for a code-review, then though it was already in :-)

It's just been merged now, and I've just updated the umbrella to pick that up as of ...

b863c40

Sorry about that. Now I understand why you were still seeing the issue.

So maybe, just maybe, a vanilla umbrella, using install_deps.sh will work for you now?

No, actually, because you are working directly from my cpp-ethereum branch, aren't you? Hmm.

Anyway - let me know what you do tonight and what you see, and we'll work it from there. I've been doing work in webthree-umbrella today "upstream" which I will then pull down into my cpp-ethereum branch.

PS ... see also for some broader context on where this change fits into the grand plan:

@bobsummerwill

I've just finished the test.

I pulled upstream and cmake .. -DEVMJIT=1 and finished with SUCCESS!! \o/

You can see the complete log here: https://gist.github.com/programonauta/2ce82337b56228d1c37483f4c92cb90d

By the way, after run make install I need to export LD_LIBRARY_PATH="/usr/local/lib to run the programs.

But I didn't see this instruction in the docs.

Hey @programonauta,

Thanks for creating the pull request! The happy surprise I got is that my TravisCI and Appveyor automation were both invoked for the PR :-) See https://github.com/bobsummerwill/cpp-ethereum/pull/1.

The bad news for you is that the macOS builds failed, and I can see why. If you look at your branch you will see that is it is 25 commits behind mine. You're missing a load of my changes, including my attempt at Debian support :-)

https://github.com/programonauta/cpp-ethereum/tree/merge_repos

So I think you'll need to do a git rebase to refresh from my upstream branch and then try to reapply your changes to top of mine. They WILL clash and they will need resolving. Some more git learning for you!

WRT to LD_LIBRARY_PATH, I wonder whether that is something Debian specific?

You're missing a load of my changes, including my attempt at Debian support :-)
Humm... I did a git pull upstream merge_repos. But git asked me to commit my install_deps.sh first and then pull upstream. I was thinking this let me up to date with your repo.

So I think you'll need to do a git rebase to refresh from my upstream branch and then try to reapply your changes to top of mine. They WILL clash and they will need resolving. Some more git learning for you!

Indeed, I didn't understand what is happening 😵. When I git rebase I got:

$ git rebase
First, rewinding head to replay your work on top of it...
Applying: Let's try building the Windows installer, and using variables for the config names.
Applying: Grr.   Tabs and spaces.
Applying: Stripped out old msbuild invocation.
Applying: More tabs and spaces.
Applying: Further cmake --build tweaks.
Applying: Do I need to specify the generator?
Applying: Let's just use msbuild directly.
Applying: Fixed working directory issue for Windows installer step.
Applying: Can't work out the correct msbuild invocation for the Windows installer, so will switch back to the indirect command-line via CMake which is in ethbinaries.sh.
Applying: Added generator as parameter for Windows installer call to cmake in appveyor.
Applying: Try msbuild again.
Applying: msbuild case-sensitive target name for PACKAGE?
Applying: Need to use PACKAGE.vcxproj as an entry-point?
Applying: Why do we need an installer anymore?   Forget it!
Applying: Updated Arch Linux detection.
Applying: No Arch Linux confirms.
Applying: Fix up repos for arch packages.
Applying: Replace your with git clone / makepkg / paceman sequence.
Applying: Stripped out unwanted sudo.
Applying: Removed unwanted sudo.
Applying: Cannot find opencl-headers.
Applying: Wants a trailing slash?
Applying: Build libjson-rpc-cpp from source on Arch.
Applying: Clean out old clone, if present.
Applying: Disabled test RUNS.   Want to get "green" and focus on finishing release flow tomorrow.

When I did git status:

$ git status
On branch merge_repos
Your branch is ahead of 'origin/merge_repos' by 25 commits.
  (use "git push" to publish your local commits)
Untracked files:
  (use "git add <file>..." to include in what will be committed)

    libjson-rpc-cpp/
    libmicrohttpd-0.9.37+dfsg/
    libmicrohttpd_0.9.37+dfsg-1.debian.tar.xz
    libmicrohttpd_0.9.37+dfsg-1.dsc
    libmicrohttpd_0.9.37+dfsg.orig.tar.gz

nothing added to commit but untracked files present (use "git add" to track)

Obviously I need to remove these temporary files before any git add, but as I could noticed, my install_deps.sh was preserved (remember I committed it before git pull upstream.

Please help me to clear my understanding :)

I'm going to a business trip tomorrow, but I'll try to understand it at night (it's late now and I must wake up early tomorrow).

Yeah - rebasing is at the edge of my understanding too.

What you should be able to do though is pull/merge, ie.

git checkout develop
git pull upstream develop
git push
git checkout yourbranch
git merge develop

But that will leave you with multiple commits in the PR.

So perhaps you do need to do all that and THEN do the rebase, and that will fix that up into a single commit again, by "replaying your changes" as though they had been made against the new HEAD?

That will definitely leave you with manual conflicts to resolve, between your edits and mine, but might at least have the correct history and "upstream is at head" that we want.

I must confess that I have often created fresh new branches when I get into such situations, rather than taking the time to really understand these git operations as deeply as I should.

Best wishes!

Bob, I was not getting do that, the number of commits didn't equal.

So, I closed my PR, delete my cpp-ethereum, forked it again. And now, I suppose is everything ok.

What I noticed is probably when I Pulled request didn't select correct branches. I only clicked on Pull Request button without pay attention on the branches.

Please verify if it's everything OK and I PR the install.sh. ;)

Getting close.

See programonauta/cpp-ethereum@62795c9, and look at the diff for your commit.

That has regressed out a chunk of my work on "Arch", so you'll just need to cut-and-paste the old revision back on top of that.

This - https://github.com/bobsummerwill/cpp-ethereum/compare/merge_repos...programonauta:merge_repos - is a compare between your merge_repos and mine. Or you can just make a new PR, and then just keep making further commits until it is correct. It's the same "diff" comparison.

Secondly, I moved where the Debian steps were happening to BELOW the version-specific checks within Debian, because I think the steps will be near identical for Jessie, Wheezy and Stretch. Your steps are still directly under the Jessie "case", which is why we haven't seen the expected clashes, but also means that both your steps and mine will be run :-)

So, after fixing the Arch, you are going to need to chop your steps out from that location entirely, and please can you just try mine, vanilla? And where they are broken, you would then tweak mine to add bits of yours as required.

You can see yours here:

https://github.com/programonauta/cpp-ethereum/blob/merge_repos/install_deps.sh#L174

And mine here:

https://github.com/programonauta/cpp-ethereum/blob/merge_repos/install_deps.sh#L261

OK - good to understand what's happening.

BTW - How are you doing your clean VMs? Which exact image are you using?

Because I'm starting to look at Docker images on CoreOS within Azure, so I can do some of this testing myself. And/or I might just use Azure images directly where available.

Are you just doing something with Vagrant/VirtualBox or similar? What is your host OS?

Debian Jessie on Debian Jessie? :-)

BTW - How are you doing your clean VMs? Which exact image are you using?

http://cdimage.debian.org/debian-cd/8.5.0/amd64/iso-cd/debian-8.5.0-amd64-netinst.iso

Are you just doing something with Vagrant/VirtualBox or similar? What is your host OS?

I'm using Debian on Yosemite and El Capitan with Virtual Box

please can you just try mine, vanilla?

Sorry, what do you means 'vanilla'?

I mean ... can you try fixing that Arch regression, and then REMOVING your chunk of code, and see what happens? It should drop through to my new alternative to yours, and I just want to see whether that works for you, or what happens. Thanks!

@bobsummerwill, very sorry.

Now I realized the mess I was doing. I was just copying my old install_deps.sh over your version.

Tomorrow I will fix this and test again in a new Debian.

Great - thanks!

BTW - If you aren't already using it, I would thoroughly recommend SourceTree, from Atlassian, as a GUI for git.

@bobsummerwill

I've cloned the repo again, using your install_deps.sh. I only inserted some code to install cmake 3.5.2 because the cmake installed with apt-get install cmake is 3.0.2

Then I tried to run it.

The first time I got an error logged here: https://gist.github.com/programonauta/ae3198828ef8f19237efaaf5d7ad57d0

Then I tried a second time and got success: https://gist.github.com/programonauta/ddeca4e8eec52ca82396156d276781a2

But when I tried cmake .. -DEVMIJ=1 and make I got an error: https://gist.github.com/programonauta/cdeb0fae48046c8ddcbd7daae7459c52

OK - got out a new release including a tweak to this.

Please could you try against https://github.com/bobsummerwill/cpp-ethereum/tree/merge_repos HEAD?

Which included this tweak to those warnings:

https://github.com/bobsummerwill/cpp-ethereum/commit/076831b0a1dea1903323b46d7731e81df95cf805

Thanks! Goodnight.

@bobsummerwill, sorry for my absence these days.

Did you removed install_deps.sh from your merge_repos branch?

Hi @bobsummerwill

I've updated my local clone and test all process in a fresh Debian. But first I inserted a cmake version test.

But at first time I run the install_deps.sh I got this error:

CMake Error: Could not find CMAKE_ROOT !!!
CMake has most likely not been installed correctly.
Modules directory not found in
/usr/local/share/cmake-3.0
CMake Error: Error executing cmake::LoadCache(). Aborting.

make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target 'install'.  Stop.

At second time everything run ok.

If I keep the install_deps.sh without this check, when the user runs cmake .. will get an error due the version installed (3.0.2)

And I have no idea on how to solve this error at first run.

Hmm - I'm guessing that is failing here?

https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/scripts/install_deps.sh#L269

Looks like my script is missing the "build CMake from source" step, eh?

Let me blindly re-add that from yours, and then you can try mine again.

@bobsummerwill, the problem remains. Probably inside the script we need to find a way to "inform" it that there is a new version of cmake, simulating a "finish and rerun" the script. I say that because if I just run the same script again everything works fine.

Look the error I got at first time (and only at first time) I run the script:

It has anything to do with environment variable CMAKE_ROOT.

Cloning into 'libjson-rpc-cpp'...
remote: Counting objects: 5159, done.
remote: Total 5159 (delta 0), reused 0 (delta 0), pack-reused 5159
Receiving objects: 100% (5159/5159), 1.36 MiB | 230.00 KiB/s, done.
Resolving deltas: 100% (3462/3462), done.
Checking connectivity... done.
Note: checking out 'v0.6.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at c6e3d71... Merge branch 'release/v0.6.0'
CMake Error: Could not find CMAKE_ROOT !!!
CMake has most likely not been installed correctly.
Modules directory not found in
/usr/local/share/cmake-3.0
CMake Error: Error executing cmake::LoadCache(). Aborting.

make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target 'install'.  Stop.

Hmm. Maybe the same as http://stackoverflow.com/questions/18615451/cmake-missing-modules-directory?

Where they suggest this flow:

cd cmake-3.4.3
./bootstrap --prefix=/usr
make
sudo make install

I wonder if it is that missing bootstrap step which is the problem? Want to try that as a local edit?

SUCCESS!
I've just pushed install_deps.sh. Please see if it's ok to PR.

Great - thanks!

I've just made that change :-)

https://github.com/bobsummerwill/cpp-ethereum/commit/d7b92ceeb01fe4805836fcf8234ff29c2904cc44

So, to confirm, with that change you can now do the plain

  • git clone recursive
  • cd cpp-ethereum
  • ./scripts/install_deps.sh
  • mkdir build && cd build
  • cmake ..
  • make

process on a fresh Debian Jessie box and it works? If so, hurrah!

And I will update the ethdocs.sh instructions for the pending v1.3.1 release with this code so that we say to just just that on Debian, and remove all the manual instructions, because they don't matter anymore!

You don't happen to have VM images for Wheezy and Stretch, do you? The flow which is being followed for those in install_deps.sh is the same, and I think that is correct, but we don't have those verified.

Or maybe @phonikg could help? How is your non-existent spare time at the moment, John?

Because I think that with these changes in place, building cpp-ethereum on Debian-based SBCs should be pretty close to "just work" now too. Some extras for swap-space and blah-bah, but the core of the build process should be identical.

Also paging @vgrade!

What SBCs do you have again, Martin? We've been working on an install_deps.sh script for cpp-ethereum which just "does the right thing" for a large range of distros. We are effectively just "infrastructure-as-coding" all our existing manual install steps.

So we now have macOS, Windows, Ubuntu and Debian working, with incremental progress on other distros. Looking to pull together all the different threads of "C++ build instructions" into scripts within the repo itself, to eliminate all that manual process.

OK - So I am going to close this specific Debian issue for now.

See also #639, which has a chunk of issues there related to install_deps.sh.

When cpp-ethereum-v1.3.1 is released (next few days), and the Homecoming is complete (we're back in develop on http://github.com/ethereum/cpp-ethereum), I will likely split that chunk of tasks into a new grouping meta-task.

So, to confirm, with that change you can now do the plain process on a fresh Debian Jessie box and it works?

Exactly

AWESOME! Thank you :-)

image