build: Cylon Summit recipe is failing
Opened this issue · 10 comments
Hello all,
We were trying to deploy Cylon on Summit following this recipe: https://github.com/cylondata/cylon/tree/main/summit (README). We tried two modes of installation on Summit:
- Following the recipe mentioned above and using compute nodes and the Cylon did not finish as it was failing with dependency installation.
- We switched to the login node, and we were able to install Cylon, and it was faster. We exported the necessary
arrow
andbuild
paths. Yet we keep getting this error:
(We ran a test from a compute node here usingjsrun
as recommended)
Any help or idea is appreciated it.
Hi @nirandaperera.
Which library path are you talking I tried with CYLON/lib, but I don't think it is working
![image](https://private-user-images.githubusercontent.com/20664913/239019288-c31f6d72-25a0-43a4-8187-88c8693410f3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU3MDE3NTAsIm5iZiI6MTcyNTcwMTQ1MCwicGF0aCI6Ii8yMDY2NDkxMy8yMzkwMTkyODgtYzMxZjZkNzItMjVhMC00M2E0LTgxODctODhjODY5MzQxMGYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA3VDA5MzA1MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg4OTkxMzkwZjNlODdlN2FlNzgxZWZhOGIxY2RjOGZjMjExMzFjZGMwZDY2ZDM2MjZlNDk3MzYyMmUwN2U0YWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.IDhgSQNJC2llBOwiYBqyq5YdAS8qO5R4OH9g99ILDD4)
I have run across a similar issue on Amazon Linux and it is related to cmake not finding openMPI. Niranda may have another suggestion but perhaps try installing from a OS repo such as yum or apt or build openmpi from source. I'm doing the same in parallel :-)
Hi @okilic1,
I think we are talking about two separate issues here. @AymenFJA's initial issue was related to running cylon after compilation. There, you need to add <cylon install dir>/lib
path to the LD_LIBRARY_PATH
.
I believe you are referring to building cylon. Do you have OpenMPI installed in your system? If not, you can either install it locally, or install it using conda. It should resolve the compilation problem.
Can documentation for this be improved. Do code changes need to be done?
Is this issue producing again? It should be fixed after executing
export LD_LIBRARY_PATH=$BUILD_PATH/arrow/install/lib64:$BUILD_PATH/glog/install/lib64:$BUILD_PATH/lib64:$BUILD_PATH/lib:$LD_LIBRARY_PATH
This is already in the documentation
@arupcsedu we tried this approach before opening the ticket, and it did not work. Did you test from your side recently with this approach on Summit? If you did, then happy to give it a try again. Maybe something has changed.
@AymenFJA , if you have access to the summit, let's have a joint session to make cylon work. The steps by steps guidelines mentioned in the https://github.com/cylondata/cylon/tree/main/summit, was working before in summit.
@arupcsedu Unfortunately I have no access.
@arupcsedu we all wait for approval.