Edit url and indexUrl after the igv browser creation (S3 signed URL expiration)

[PierreGabioud]

I am using S3 signed URLs for some of my track's files (cram and its index).
The problem is that those URLs expire after some time (that needs to be as short as possible) and I get 403 errors in the browser.

Is it possible to update the "url" and "indexUrl" of a track after the browser creation ?

I tried to change the values in the myIGVBrowser.config.tracks[0].url = "newURL"; but it's not this URL that is used. I can't find other places in the "browser" or in the code where this URL is set.

[jrobinso]

Yes I see the issue, igv.js isn't setup for signed URLs. Some possible solutions are proposed below, I'm unsure myself on what would work best just throwing them out for comment.

Its a bit of a hack, but you could "find" the track using the browser find function. Then set the url on the track object itself. It would look something like this. I have not tried it, but it should work

const tracksByID = browser.findTracks("id", "T2");
const track = tracksByName[0]
track.url = ...
track.indexURL = ...   // if needed
track.config.url = ...     // This is probably not neccessary
track.config.indexURL = ...     // This is probably not neccessary

Some other longer term options:

• The idea solution would be to authenticate in igv.js, then update S3 urls as required. I'm not sure this is possible with AWS, and would require a bit of time to investigate. I could not give an ETA on this at this time.

• Alternatively, authenticate in the enclosing application and pass IGV the oAuth token to fetch fresh S3 URLs. I'm still not sure this is possible, but is less work than the above option.

• Provide a function (which can be async) for IGV to pass "s3" urls to which will resolve them to https (signed) urls. All "s3" urls would be resolved by this function anytime they are used. It would be the responsibility of the function to decide if fetching a new one is required. This is the simplest thing to implement.

[PierreGabioud]

@jrobinso Thank you ! I've implemented the third solution. I manage the caching and refreshing of the signed URL myself in the frontend.

[jrobinso]

@PierreGabioud How did you implement it? I had thought some change to igv.js would be required. Did you implement it without changing igv.js?

[PierreGabioud]

@PierreGabioud How did you implement it? I had thought some change to igv.js would be required. Did you implement it without changing igv.js?

I didn't change igv.js. All I did was giving an async function to "url" and "indexUrl".

I've just looked at the igv.esm.js code that I use and saw this in the load function that seems to be called :

async load(url, options) {
  ...
  // Resolve functions, promises, and functions that return promises
  url = await (typeof url === 'function' ? url() : url);
  ...

[jrobinso]

Yes, that is the right direction, but from code inspection it looks like that is resolved once on track creation, but I could be wrong. I'm traveling, will look at this later.

[sjneph]

I'm certainly interested in a concrete way to make this work.

[jrobinso]

So is your current solution not working?

For a more general solution, I'm leaning towards allowing an async function to be registered with igv.js that would take s3:// urls as input and return https:// urls, which might be signed or not. This function would be called everytime a request is made to the s3:// url. This could be generalized for other schemes (other than s3://).

I'm really covered up right now but will get to this ASAP.

[PierreGabioud]

@jrobinso Maybe I'm missing something but I am not sure I see the benefit of another special function if the url and indexURL are already accepting async functions ? The function is called every time the url is needed, at least in my code with version igv 2.15.11. Would this special function catch expiration errors and make the necessary "refresh" of the URL itself ?

I'm using next/react and doing things a bit differently but basically the process can look like that :

// Store the signed URLs
const cachedSignedUrls = {
  cram: null,
  crai: null,
};

const urlExpirationTime = 60 * 60 * 1000; // The duration of your signed URLs

async function getSignedUrlCramFn() {
  if (cachedSignedUrls.cram) {
    return cachedSignedUrls.cram;
  }
  // No cached signed URL, so get a new one
  return myApi.getSignedUrl(someParams)
    .then((signedUrl) => {
      cachedSignedUrls.cram = signedUrl;
      // Set a timeout to clear the cached signed URL
      // Clear old timeouts if any: clearTimeout(cramTimeoutId)... etc
      cramTimeoutId = setTimeout(() => {
        cachedSignedUrls.cram = null; // Delete it from cache
      }, urlExpirationTime);
     return signedUrl;
    });
}

async function getSignedUrlCraiFn() {
  // same process as getSignedUrlCramFn
}

const igvOptions = {
  ...
  tracks: [
    {
      url: getSignedUrlCramFn,
      indexURL: getSignedUrlCraiFn,
      ...
    },
    {...}
  ],
  ...
}
igv.createBrowser(someDiv, igvOptions)
  .then((browser) => {
    // ..
  });

[jrobinso]

I'm surprised its called everytime its needed, that is not what it looks like it does from code inspection. I haven't had time to do tests. But if that's the case you are all set. That functionality is not going away.

My proposal was simple, just supply a global function that exchanges all "s3://" urls for something else. Then the url in the config can just be "s3://......". I think some people might find this simpler, but if your solution is working for you great.

[jrobinso]

Closing out old issues. After some code review I can confirm you are correct, if URLs are specified as a function or promise it is called/resolved everytime the URL is accessed. So there is no need for another mechanism. I thought the resolved URL might be cached somewhere but that is not the case.