Error when adding a score column with additional filters

Release number or git hash: f045a07
Web browser version and OS: Chrome 77
Environment (local or deployed): local and deployed

Steps to reproduce

Open https://ordino.caleydoapp.org
Select Cancer Gene Census
Select one or multiple genes
Open Copy Number view
Filter for Organ: bladder (or any other filter)
Apply filter

Same behavior applies when starting with cell lines and switching to genes.

Observed behavior

Shows an error message
The selected gene is not added as score column

Expected behavior

No error message and the score column for the filtered rows should be added

The error message appears for request like https://ordino.caleydoapp.org/api/tdp/db/publicdb/cellline_gene_single_score/score?table=copynumber&attribute=relativecopynumber&name=ENSG00000121410&species=human&filter_organ=bladder, i.e., requests with an additional filter_.

The error message started to appear with the merged PR datavisyn/tdp_core#177. More specific: the code was changed that if an invalid filter is passed it throws an error 400 (see https://github.com/datavisyn/tdp_core/pull/177/files#diff-1e0bc69eb888c71b05b1f80de7e2e635R155-R156). Before the invalid filters were just removed silently and not added to the SQL query.

Restoring the old version (i.e., before datavisyn/tdp_core#177) shows, that the organ filter is removed as well and the score is calculated for all cell lines.

api_1          | 09:33:45 WARNING tdp_core.sql_filter: invalid filter key detected for view "
api_1          |           SELECT d.celllinename AS id, d.{attribute} AS score
api_1          |           FROM cellline.tdp_{table} d
api_1          |           INNER JOIN public.tdp_gene s ON d.ensg = s.ensg
api_1          |           INNER JOIN cellline.tdp_cellline g ON d.celllinename = g.celllinename
api_1          |           {joins}  WHERE g.species = :species AND d.ensg = :name {and_where}" and key "organ"

It just worked before the PR, because LineUp was selecting the rows by id and skipped the ones that are not displayed. Hence, the filter never worked correctly and filtered the result set down.

It turns out that view.is_valid_filter(key): () was checking the filter and columns of the view.

https://github.com/datavisyn/tdp_core/blob/3657f28e77c3d6ec868dd5a96577859a3d530221/tdp_core/dbview.py#L69-L75

In this case the score view is defined in query_score.py:

tdp_publicdb/tdp_publicdb/query_score.py

Line 28 in 5b7d963

.filters(sample.columns) \

The problem, is that in our example the columns for the genes are set as filter. Correct would be the columns of the cell lines. Hence, the correct filter are applied from the variable gene. (Don't be confused: in case of the inverted score, like in our example, the samples variables contains the gene and the gene variables contains the cell lines entity.)

The result is that the correct filtered subset (and not all) scores are returned.

The server logs correctly state now that the organ filter was added to the SQL query:

api_1          | 11:25:30 INFO tdp_core.db:
           SELECT d.celllinename AS id, d.relativecopynumber AS score
           FROM cellline.tdp_copynumber d
           INNER JOIN public.tdp_gene s ON d.ensg = s.ensg
           INNER JOIN cellline.tdp_cellline g ON d.celllinename = g.celllinename        
           WHERE g.species = :species AND d.ensg = :name  AND organ = :organ ({'name': u'ENSG00000121410', 'organ': u'bladder', 'species': u'human'})