rapier1/hpn-ssh

Accessibility, Packaging, and Distribution

zardini123 opened this issue · 10 comments

zardini123 commented

Hi!

I recently came across HPN-SSH when trying to understand why my sftp data transfer was so darn slow. I really want to try it out, but the steps that make install takes are unclear to me. I don't want to run make install, try out HPN-SSH and find some issue that makes it unusable for me, then find that my default installation of ssh in /usr/bin has been overwritten and I have no way to revert back.

This brings up two points:

  1. Providing accessible and explicit information and instruction to new and novice users like myself
  2. Provide package repositories for popular package managers for ease of experimentation

I will elaborate on each point:

1. Providing accessible and explicit information and instruction to new and novice users like myself

I found HPN-SSH via the HPN-SSH homepage. It took me two or three look overs to understand that everything "precompiled" is on the HPN-SSH SourceForge page, and the most recent up-to-date code changes is on this GitHub repo. It was difficult to tell by glancing that it was useful as it was under the News and Notes, even though these two page links were prefaced well with text explaining their reasoning.

Once I understood that the HPN-SSH was applied via "patching," I wanted to apply it myself. I went to the patch folder in SourceForge, and was greeted with 8 options for patches, with no explanation in the README about what each mean, and do. I assumed that the file openssh-8_3_P1-hpn-14.22.diff was the patch containing all the relevant modifications. Regardless of what I chose, I didn't even know how to apply those patches. I never knew watch patch files were before I found HPN-SSH!

I just realized a couple days ago that there is instructions at the very bottom of the HPN-SSH homepage how to apply patches to the source! That was great information to find, but certainly took many iterations looking over all the pages to come across that.

Now that I understand that everything new relating to HPN-SSH is happening in this GitHub, I search "hpn ssh GitHub" in google. The first repository I find is rapier1/hpn-ssh which hasn't been pushed to in 6 years! This repository (rapier1/openssh-portable) does not show up in the first page of search results in Google.

In conclusion for 1., condensing all information on how to use and install HPN-SSH into one location. Having a glossary section providing explanation of terminologies, separated by the processes of using/installing HPN-SSH. For example, having a section explaining what all the different versions/terms like "KitchenSink" and "ServerLog" mean and do. Have one of the top sections be the steps to apply the patch files, instead on the bottom of the page.

In terms of this Github Page, the README seems to be a carbon copy of OpenSSH's README. It be great to have the following information for HPN-SSH on there too. More importantly, provide information about how the HPN-SSH project modifies OpenSSH, and where make install installs stuff. Also, would renaming this repository to be hpn-ssh be viable way to increase foot traffic?

Does HPN-SSH require itself to be installed on the client and the server? I couldn't find any explicit information about this.

2. Provide package repositories for popular package managers for ease of experimentation

In my quest to try out HPN-SSH, I was looking for a Homebrew package for a quick way to install (and uninstall) HPN-SSH on my macOS system. I couldn't find an official brew repository that you all provide, so I turned to community-made ones. I came across this one, but I noticed the URL to the patch file is hard coded! They use the KitchenSink version (which I have no idea what that means)! There is also patch files specific to macOS that are applied! They also run make install too, without clarification where its actually installed!

Homebrew is fantastic as it usually manages the source and binaries inside their own little "kegs", and then symlink any necessary binaries into /usr/local/bin instead of overwriting /usr/bin. Being that this is a home-brew repository not sponsored by you all, I was critical of its functionality and did not use it. I still have yet to try HPN-SSH.

Same applies to linux. There are some third-party PPA's for installing HPN that are years old, each with the link to the path file in SourceForge hardcoded. One of my recommendations with using Github would be hosting the host files as releases here. Github has an API for downloading assets of any non-pre-release. Only one link would be needed for grabbing the most recent patch.

Here you all state you have Packaging and Distribution in mind. I would propose that providing package repositories for Homebrew (macOS), Chocolatey (Windows), apt (Debian), and pacman (Arch) would be important for the accessibility of this project.

I sincerely thank you for reading through all of this! I have no idea if I represent the majority user-base with my points. Regardless, I hope this perspective helps form the information surrounding HPN-SSH be as user accessible as possible. Providing easy ways to install HPN-SSH via package managers would be great for all types of users. I hope for the day to be able to install HPN-SSH without the concern of compromising the current state of my systems!

Thank you!

  • zardini123
rapier1 commented
  • 🎉4
ikozhukhov commented

Hi Chris,
i'd like propose DilOS platform with remote env for builds - please let me know what you can be interested in.
with DilOS we can setup separate lx zones with linux builds, but not at all platforms, just several.
also we are interested in builds and tests on native DilOS platform.
with DilOS we are using Intel and SPARC builds.
please let me know if it can be interested in.
-Igor

mej commented

We have recently received a new grant from the NSF and one of the major deliverables in the project is to address the packaging and other issues.

This is phenomenal news, Chris! I know how big a pain in the ass those funding proposals can be to put together, so your perseverance is greatly appreciated! :-)

zardini123 commented

@rapier1 Thank you so much for your reply! I am sincerely excited for the future for this project. Congratulations on the grant!!

I do have two quick questions:

  1. Is HPN-SSH required to be installed on both the server and the client? What happens if one only installs it on only the client, or only the server?
  2. Does this repository accept pull requests? If I manage to figure out how to install HPN-SSH, I wouldn't mind contributing some time to write up some documentation regarding that. It'd be great for this project to have contribution guidelines as well.

Thank you again!

rapier1 commented

@zardini123 My apologies for not answering sooner. Optimally HPN-SSH should be on both sides of the connection but you should still see a performance improvement (assuming that the receive buffer is limiting performance) as long as HPN-SSH is installed on the data destination side. If you want to make use of the None cipher then you must have HPN-SSH on both sides of the connection.

rapier1 commented

@zardini123

Also, I just released a new version of HPN-SSH that incorporates some new performance improvements. Taking your suggestion I've also started the process of customizing the packaging so that HPN-SSH can live along side of a standard OpenSSH installation. Currently I only have an RPM but it now installs HPN-SSH in /opt/hpnssh, creates a systemd init, and maintains separate keys and configs. It even works under SELinux (which was a huge pain in the butt and probably needs fine tuning). The only issue, and this is intentional, is that the default listen port is still TCP 22. So if you wanted to run and HPN-SSH server along side of OpenSSH you'll need to edit /opt/hpnssh/etc/ssh/sshd_config and change the listen port. If you are running SELinux you'll need to make some other changes but those are addressed in the /opt/hpnssh/usr/share/doc/openssh/HPN-README file.

I'll be working on setting up a PPA, debian packages, homebrew etc as time allows. Hopefully I'll be able to hire a student to do all the work on that :)

zardini123 commented

Hi @rapier1! Apologies for not responding earlier. I just found time and an opportunity to try installing HPN-SSH, but I am still in major confusion.

As a new user who wants to try out HPN SSH, what exactly should I download to get the best, most up-to-date experience? There are currently so many options, I feel very overwhelmed. Here are the options with their methods of installation I can find:

  • From default OpenSSH readme
    • Download zip of repository, run configure, and make
    • Clone repository, run configure, and make
    • Clone repository, run configure with a list of build time options, and make
  • Github individual releases (all of these end with 8_4_P1)
    • NoneSwitch?
    • DynamicWindow?
    • DynWinNoneSwitch?
    • ServerLog?
    • PeakTput?
    • KitchenSink
      • This one sounds amazing simply by the name, but I have no idea what it is (as per the others).
    • AES CTR?
    • hpn
      • Sounds basic and approachable, but how does it differ from the others?
    • V
      • what the...
    • Note: each of these releases only contain zip/tar.gz of the "source code." Downloading each of them, unzipping, and running a difference check using diff shows that there are differences in the code. I am assuming compiling these is identical to that of the main repository.
  • Sourceforge
    • Patches
      • Filled with version folders. Each version folder has .diff files. Every diff file has the same set of names as in the GitHub releases. The "V" naming type is absent from the folders though.
      • My assumption is these are applied to the OpenSSH source code using patch command line, as instructed at the bottom of the HPN-SSH website. Though, there is no direct instruction on how to do this in the Sourceforge README's.
    • Debian Packages
      • Packages for Debian Linux.
    • RPMS
      • I'm assuming these are .rpm packages, which are the packages used in Red Hat.
    • GSI-Openssh-HPN-SSH / CentOS 7
      • Filled with .tar.gz files, has a folder called "Source RPMs". How do these differ from RPMS folder?
    • OpenSSL-1.1 Compatibility
      • Filled with what I'm assuming is patch files? Last modified was 2018 so I'm assuming I shouldn't touch this :)
LuckyFellow commented

Please consider distribution via docker container at: https://hub.docker.com/

Docker containers can run on a wide range of different architectures and software versions.
You can preconfigure everything to work out of the box.

If hosting at hub.docker.com is not an option, then please provide a "Dockerfile" to automatically execute the proper steps to build a docker image locally: https://docs.docker.com/engine/reference/builder/

rapier1 commented

@LuckyFellow I have considered a docker container but the additional network overhead of a docker will have an impact on performance. That said, I do have an old container around here somewhere that I can use as a base. Just in case anyone wants it. I'll work on that in the next week.

LuckyFellow commented

According to the following post are the runtime performance cost of a Docker container relatively little and the network overhead (NAT) can be avoided by attaching the container to the host network stack (docker run --net=host):

https://stackoverflow.com/questions/21889053/what-is-the-runtime-performance-cost-of-a-docker-container

In my case, I don't need a setup with maximal performance possible. A Docker image would have allowed me to setup HPNSSH within minutes. Without the Docker image it took me a day to get HPNSSH compiled and running in a container. Since I build Docker images very infrequently, everytime it's reading through a lot of documentation to be sure to get things setup the right way.