das-developers/das2java

HTTP authentication mishandled in HttpFileSystem, KeyChain

Opened this issue · 4 comments

jbfaden commented

There's a bug that the FileSystem keychain is misimplemented for HttpFileSystems. This is because it was originally added to support FTPFileSystems, which are based on directory, and HTTP directories are not secured this way. The credentials should be paired with the "basic realm" specified in the 401 response. Any two directories using the same basic realm use the same credentials.

This has caused a number of problems. For one, the keychain file used to store the credentials does so based on the directory. So you might end up with credentials attached to http://autoplot.org/autoplot/data/agg/efi/1999/ and then not have credentials for http://autoplot.org/autoplot/data/agg/efi/2000/. (Note this folder doesn't require credentials.) This also means kludgy logic ascends through the folders looking for passwords in the keychain.

The das2server uses the same 401 auth mechanism, but it has it implemented properly. Perhaps the keychain for the das2server can also serve as the keychain for filesystems.

See https://datatracker.ietf.org/doc/html/rfc7617.

Further, there's a bug where it fails to realize that it could ask for credentials to get more data. See the procedure below which shows an error similar to what Kristoff is seeing.

jbfaden commented

Kristoff reminded me of this problem.

jbfaden commented

Here are some URLs to play with:

wget --server-response http://research.ssl.berkeley.edu/data/spp/data/sci/fields/l2/mag_RTN/2020/01
wget --server-response http://research.ssl.berkeley.edu/data/spp/data/sci/fields/l2/mag_RTN/2021/01
wget --server-response http://research.ssl.berkeley.edu/data/spp/data/sci/fields/l2/mag_RTN/2021/02
wget --server-response http://research.ssl.berkeley.edu/data/spp/data/sci/fields/l2/mag_SC/2021/04
jbfaden commented

To demo this bug in Autoplot,

  1. plot http://research.ssl.berkeley.edu/data/spp/data/sci/fields/l2/mag_SC/2020/12/psp_fld_l2_mag_SC_$Y$m$d$(H;delta=6)_v$v.cdf?psp_fld_l2_mag_SC[:,2]&timerange=2020-12-27
  2. hit next so many times to get to 2021.
  3. instead of asking for credentials, you will see "no files in interval"