This document gives step by step instructions to help you get started developing CWL workflows on Arvados.
- Install & set up ssh client on Windows
- Install vscode and relevant extensions
- Use vscode to connect to an arvados shell node for development
- Create an simple command line tool, run it, and view the log
- Upload input, create a command line tool to process it, and view the output
Linux and MacOS users can skip this step.
- Install git for windows https://git-scm.com/download/win
- Open git bash
- Run “ssh-keygen”
- Look for “Your public key has been saved in /c/Users/MyUsername/.ssh/id_rsa.pub”
- Run “cat /c/Users/MyUsername/.ssh/id_rsa.pub”
- Copy the lines starting with “ssh-rsa …”
- Open Arvados workbench
- Go to “SSH keys” in the user menu
- Click + Add new ssh key
- Paste the key into “Public key” and enter something for “name”
- At the git bash command line, run “ssh shell…” and say “yes” if it asks “do you want to continue connecting”
- You should be logged into the Arvados shell node.
- Log out by typing “exit”
- Install vscode https://code.visualstudio.com/ and start it up
- Vscode: go to “Extensions”
- search for “remote development” and install the Remote Development extension pack from Microsoft
- Vscode: On the left side bar, choose “Remote explorer”
- In the drop down, choose “SSH targets”
- Click “Add new”
- Enter the “ssh shell…” command line you used in step 1(j)
- Right click the ssh target in the list and select “connect to host in current window”
- Vscode: go to “Extensions”
- Search for “benten” and install “CWL (Rabix/Benten)”
- Choose “Install extension on the remote ssh host”
- Vscode: On the left side bar, choose “Explorer”
- Select “Clone Repository” and then enter “https://github.com/tetron/vscode-cwl-testing.git”
- Choose “Open”
- Go to Arvados Workbench
- Workbench: In the user menu, select “Current token”
- Vscode: Click on the “Terminal” menu
- Vscode: Click “Run Task…”
- Select “Configure Arvados”
- Workbench: Copy the string following “ARVADOS_API_HOST=”
- Vscode: Paste the string at the “Value for ARVADOS_API_HOST” prompt
- Workbench: Copy the string following “ARVADOS_API_TOKEN=”
- Vscode: Paste the string at the “Value for ARVADOS_API_TOKEN” prompt
- Vscode: Click on the “lesson1/main.cwl” file
- Click on the “Terminal” menu
- Click “Run Task…”
- Select “Register or update CWL workflow on Arvados Workbench”
- Workbench: Go to “+NEW” and select “New project”
- Enter a name for the project like “Lesson 1”
- You should arrive at the panel for the new project
- Workbench: With “Lesson 1” selected
- Click on “+NEW” and select “Run a process”
- Select “CWL training lesson 1” from the list and click “Next”
- Enter a name for this run like “First training run”
- Enter a message (under “#main/message”) like “Hello world”
- Click “Run process”
- This should take you to a panel showing the workflow run status
- Workbench: workflow run status panel
- Click on the three vertical dots in the top-right corner
- Choose “Log”
- Under “event type” choose “stdout”
- You should see your message
- Vscode: Click on the “lesson2/main.cwl” file
- Click on the “Terminal” menu
- Click “Run Task…”
- Select “Register or update CWL workflow on Arvados Workbench”
- Go to your desktop
- Using a text editor such as notepad, create a file “message.txt”
- Enter a message like “Hello world” and save
- Workbench: Go to “+NEW” and select “New project”
- Enter a name for the project like “Lesson 2”
- You should arrive at the panel for the new project
- Arvados workbench: With “Lesson 2” project selected
- Click on +NEW and select “New collection”
- Call the collection “my message”
- Drag and drop “message.txt” into the browser
- Click “Create a collection”
- The file should be uploaded and then you will be on the collection page
- Workbench: Select the “Lesson 2” project
- Click on “+NEW” and select “Run a process”
- Select “CWL training lesson 2” from the list and click “Next”
- Enter a name for this run like “Second training run”
- Click on “#main/message”
- A selection dialog box will appear
- Navigate to the collection you created in step (13) and choose “message.txt”
- Click “Run process”
- This should take you to a panel showing the workflow run status
- Workbench: workflow run status panel
- Wait for the process to complete
- Click on the dot menu
- Choose “Outputs”
- Right click on “reverse.txt”
- Click on “Download”
- Open the downloaded file. It should have your results.
The default value for the “message” parameter will taken from the “lesson3/defaults.yaml” file
- Workbench: Go to “+NEW” and select “New project”
- Enter a name for the project like “Lesson 3”
- You should arrive at the panel for the new project
- Workbench: With “Lesson 3” selected
- Click on “+NEW” and select “Run a process”
- Select “CWL training lesson 3” from the list and click “Next”
- Enter a name for this run like “Third training run”
- The “#main/message” parameter will be pre-filled with your default value. You can choose to change it or use the default.
- Click “Run process”
- This should take you to the status page for this workflow
The “message” parameter will be taken from the “lesson4/input.yaml” file. This is useful during development.
- Workbench: Go to “+NEW” and select “New project”
- Enter a name for the project like “Lesson 4”
- You should arrive at the panel for the new project
- Click on “Additional info” in the upper right to expand the “info” panel
- Under “Project UUID” click the “Copy to clipboard” button
- Vscode: Select the file “lesson4/main.cwl”
- Click on the “Terminal” menu
- Click “Run Task…”
- Select “Set Arvados project UUID”
- Paste the project UUID from workbench at the prompt
- Vscode: Select the file “lesson4/main.cwl”
- Click on the “Terminal” menu
- Click “Run Task…”
- Select “Run CWL workflow on Arvados”
- Vscode: In the bottom panel select the “Terminal” tab
- In the upper right corner of the Terminal tab select “Task - Run CWL Workflow” from the drop-down
- Look for logging text like “submitted container_request pirca-xvhdp-hlp14qfnc8eb189”
- Highlight and copy the workflow identifier (this the string containing -xvhdp- in the middle)
- The results of this run will appear in the terminal when the run completes.
- Workbench: Paste the workflow identifier into the search box
- This will take you to the status page for this workflow