Issues with new study step templates in CSIS
DenoBeno opened this issue · 33 comments
UPDATE: the mechanism for making new studies is broken! Instead of cloning the GL-step template nodes it links the template nodes to the new study and then changes some fields in them as it should be changing in the new nodes. THIS IS EXTREMELY DANGEROUS
Update 2: The problem is caused by switch to beta 9. See full description on how the rule used to work and several reports indicating that this stopped working in beta9 here:https://www.drupal.org/project/business_rules/issues/3043752#comment-13504586
I have just tried to set up a new "advanced screening" study. This is what I got:
https://csis.myclimateservice.eu/study/118/view/context
That's new and not a good sign. What happened here?
Update: https://csis.myclimateservice.eu/maintenance/study_types clearly shows that something is wrong with the tempaltes used in advanced screening.
Apparently the problem is simply that the template names aren't set, see e.g. https://csis.myclimateservice.eu/node/934/edit/template
I changed this and now this step looks OK. So it's just a question of renaming the templates in whatever we want them to be called.
Originally posted by @DenoBeno in #104 (comment)
So, I'll try to fix this now, but I need to understand how this happened in the first place. Someone has apparently tried to improve our templates, but forgot some important steps. Mainly:
-
a "study step" template should be opened for editing in "edit as template" mode, like https://csis.myclimateservice.eu/node/934/edit/template
-
A template needs a name, template step label and a boolean field "template: is template?" must be set to true.
-
A template document should not be used in any studies.
It's possible to take any existing study step and turn it into a template, but one needs to be careful that the templates aren't deleted when a study is deleted later. I'll test what happens and report in a minute. I'll also fix the template documents that currently claim not to be templates. In fact, I'm surprised that it was even possible to link them to study types because the view that's used to include them should not even show the nodes where "is template" is not set to true. I'll test this too.
OK, I have double checked the study type edit form. E.g. this one
https://csis.myclimateservice.eu/taxonomy/term/1159/edit
When I click on "select templates", indeed only the nodes that are marked as templates can be chosen. I don't understand how these non-templates ended up here.
By the way, clicking on "edit" for each of the GL_template, editing and "updating" the template works fine and it's quite easy to use IMO. Update: except that it can't be saved afterwards. "This entity (node: 936) cannot be referenced." error. ARGH.
OK, I have updated the templates that weren't templates now. I gave them "urban screening (new)" in name so that they can be easily found. As you can see on https://csis.myclimateservice.eu/maintenance/study_types, they are used in several study types.
I have also tested and now I'm able to edit them all within https://csis.myclimateservice.eu/taxonomy/term/1159/edit. Not sure what was preventing this before.
----- somewhat related -----
I can see that the "step description" has been defined for GL steps already.
https://csis.myclimateservice.eu/admin/structure/types/manage/gl_step/fields/node.gl_step.field_step_description. However, this field wasn't shown in "template edit" mode. I have added it now. https://csis.myclimateservice.eu/admin/structure/types/manage/gl_step/form-display/template
Since the field was empty, I have coped the text from the GL step descriptions here. According to the field description this should have some impact on studies... And it does. I'm just not 100% sure if the PDF generation has been updated too.
After adding another new study, I must conclude that something is badly broken here. It would appear that the template files get overwritten every time a new study is added now. Oh, the templates are actually added to a study instead of making a copy????
See https://csis.myclimateservice.eu/group/118/nodes
Nodes linked to this study: 934, 935 etc. THIS ARE THE TEMPLATE NODES!!!! Something really bad happened with the rules that are supposed to clone the templates and produce new nodes for each study.
I've put the site in maintenance mode until we know how to resolve the problem.
Yesterday, I updated modules with composer update to try to resolve an issues with the quick edit module. So maybe this caused the problem.
This problem could also be related to the quick node clone module, which has been installed recently.
@DanielRodera Could you please check in you local instance if you can reproduce the problems discovered by Denis?
When I perform an update, I commit the changes to our private gitlab repo. I've attached the composer.lock patch so that you can reproduce which modules have been updated since my last update on December 10 2019. However, I don't know if somebody else updated modules in between.
Some updated modules that may be related to the issue:
- myclabs/deep-copy: 1.9.4 -> 1.9.5
- rupal/business_rules: 1.0.0-beta8 -> 1.0.0-beta9
- drupal/entity: 1.0.0-rc3 -> 1.0.0
- drupal/group: 1.0.0-rc4 -> 1.0.0-rc5
- drupal/views_bulk_operations: 3.3.0 -> 3.4.0
The problem is caused by switch to beta 9. See full description on how the rule used to work and several reports indicating that this stopped working in beta9 here:https://www.drupal.org/project/business_rules/issues/3043752#comment-13504586
The patch that previously made the whole thing work is not applicable to the new code anymore.
O.K. I've locked business_rules to 1.0.0-beta8 in composer.json and downgraded the module:
Gathering patches for dependencies. This might take a minute.
- Downgrading drupal/business_rules (1.0.0-beta9 => 1.0.0-beta8): Downloading (100%)
- Applying patches for drupal/business_rules
patches/business_rules_2994155.patch (Issue #2994155)
patches/business_rules_3043752.patch (Issue #3043752)- Downgrading drupal/br_group (1.0.0-beta9 => 1.0.0-beta8)
Patrick and I took a look at beta 9 and we don't understand what's going on.
- The patch is really simple and affects only one file in busines rules (https://www.drupal.org/files/issues/2019-03-29/view-result-loop-3043752-9.patch).
- This file has not changed between beta8 and beta9
- Actually the changes between b8/b9 are very minor - complete diff is just 2 pages and nost of it is in yaml files...
- However, the composer fails to apply the patch in beta 9 (??) and when Patrick applies it manually, a lot of failures are reported.
I'm reporting this upstream, hope someone will find a fix.
Some updated modules that may be related to the issue:
- myclabs/deep-copy: 1.9.4 -> 1.9.5
- rupal/business_rules: 1.0.0-beta8 -> 1.0.0-beta9
- drupal/entity: 1.0.0-rc3 -> 1.0.0
- drupal/group: 1.0.0-rc4 -> 1.0.0-rc5
- drupal/views_bulk_operations: 3.3.0 -> 3.4.0
You are right, it could be in some other element too. Basically, what this patch we are applying to rules relies on is the following:
- If someone changes the UUID while manipulating the in-memory copy of an entity, then a new entity should be generated.
- To do so, a new UUID should be generated before saving.
- This can be done by calling:
$uuid_service = \Drupal::service('uuid');
$uuid_new = $uuid_service->generate();
- And a new entity is generated on save if the nid is unset and the uuid is given the new vaue that we just generated.
If any of this fails, things will not work.
diff --git a/src/Plugin/BusinessRulesAction/SaveEntityVariableAction.php b/src/Plugin/BusinessRulesAction/SaveEntityVariableAction.php
index 66ad8bc..a5b4664 100644
--- a/src/Plugin/BusinessRulesAction/SaveEntityVariableAction.php
+++ b/src/Plugin/BusinessRulesAction/SaveEntityVariableAction.php
@@ -11,6 +11,7 @@ use Drupal\business_rules\Plugin\BusinessRulesActionPlugin;
use Drupal\business_rules\VariableObject;
use Drupal\Core\Entity\Entity;
use Drupal\Core\Form\FormStateInterface;
+use Drupal\Component\Uuid\Uuid;
/**
* Class SaveEntityVariableAction.
@@ -112,7 +113,17 @@ class SaveEntityVariableAction extends BusinessRulesActionPlugin {
if ($entity->uuid->getValue() !== $saved_uuid) {
$key_value->set($uuid, $uuid);
- $entity->save();
+
+ if ($entity->isNew()){
+ $entity->save();
+ } else {
+ $uuid_service = \Drupal::service('uuid');
+ $uuid_new = $uuid_service->generate();
+ if(Uuid::isValid($uuid_new) == TRUE){
+ $entity->uuid->value = $uuid_new;
+ $entity->nid->value = NULL;
+ $entity->save();
+ }
+ }
$result = [
'#type' => 'markup',
Is everything working OK now, after downgrade to beta9?
I've created a new study, but that doesn't look good:
Is everything working OK now, after downgrade to beta9?
I doubt it, since the patch doesn't get applied (event though thecommand line says it's being applied). Not sure what the reason for this is.
I've created a backup before applying the updates (yesterday around noon). I can restore it.
I've created a backup before applying the updates (yesterday around noon). I can restore it.
Then this would probably be the best solution. I wasn't working on the system yesterday, so from my side nothing would be lost. Not sure how it is for the others, but AFAIK no major development took place yesterday and also no DPs and Resources got updated or added in the meantime.
OK. now I'm able to create new studies and steps are copied. However, there are a few more issues:
- Vulnerability Step is created but the respective tab is missing here. I assume @patrickkaleta implemented clarity-h2020/docker-drupal#141 (comment) ?
EMIKAT calculation is not triggered when changing the bounding box here. See log. Perhaps @therter or @patrickkaleta already worked on clarity-h2020/emikat#34 or clarity-h2020/map-component#87 ? If not, this is a new problem.-> OK, no Data Package selected. :/ As already mentioned, its too easy for the user to get stuck.- Apart form the calculation not being triggered, EMIKAT doesn't show any results. See clarity-h2020/emikat#33 (comment)
I find it really spooky that the patch isn't getting applied by the composer anymore although the file on which it should be applied didn't change.
Once we have this working again, someone should try applying these updates ine by one in a test environment to see what breaks it. Is it really the latest rules or maybe the entities update?
I have just re-calculate the study 37. https://csis.myclimateservice.eu/study/37/step/1644/view/table is making a call to emikat that returns the results, but no results are shown in the table.
I don't get it. has the DB been re-set or not?
Btw, the map calls are just returning a "black" screen. And surprisingly, there is four of them every time when I change the (EMIKAT) layer. I expected two, one for the map legend and one for the layer.
I don't get it. has the DB been re-set or not?
no, db has not been restored so we didn't loose any data. but i deleted some test studies and obsolete steps.
If the db has not been restored, then templates need to be fixed again. I'll do this, just a moment.
- Heinrich has fixed the issue with maps not showing anything, a re-load of geoserver was needed after changes.
- Why the map calls the EMIKAT geoserver 4x on every change is not clear. Should be 1x, 2x at most IMO.
- table is still not shown, although the EMIKAT REST call seems to be OK and returns results.
I have just re-calculate the study 37. https://csis.myclimateservice.eu/study/37/step/1644/view/table is making a call to emikat that returns the results, but no results are shown in the table.
When I use this request, emikat does only return the column names, which are also shown in the table, but no data for the columns.
Templates for advanced screening are OK now. The only thing that needed fixing was https://csis.myclimateservice.eu/node/937/edit/template where EU-GL step was not set.
I have just re-calculate the study 37. https://csis.myclimateservice.eu/study/37/step/1644/view/table is making a call to emikat that returns the results, but no results are shown in the table.
This is what it calls: https://service.emikat.at/EmiKatTst/api/scenarios/3207/feature/view.2974/table/data?rownum=100&filter=SZ_ID=3207&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Frequent%27&column=GRID_ID&column=STUDY_VARIANT&column=TIME_PERIOD&column=EMISSIONS_SCENARIO&column=EVENT_FREQUENCY&column=T_MRT&column=T_UTCI&column=T_A&column=DISCOMFORT_LEVELWhen I use this request, emikat does only return the column names, which are also shown in the table, but no data for the columns.
Maybe it doesn't like you (@patrickkaleta , see if the user the table widget uses is blocked or something?)? I get this:
// 20200311145516
// https://service.emikat.at/EmiKatTst/api/scenarios/3207/feature/view.2974/table/data?rownum=100&filter=SZ_ID=3207&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Frequent%27&column=GRID_ID&column=STUDY_VARIANT&column=TIME_PERIOD&column=EMISSIONS_SCENARIO&column=EVENT_FREQUENCY&column=T_MRT&column=T_UTCI&column=T_A&column=DISCOMFORT_LEVEL
{
"name": "MAP Local Effects- Mean, UTCI and Apperent Temperature, Discomfort level",
"description": "Display of Mean, UTCI and Apperent Temperature and Discomfort level for different Scenarios",
"columnnames": [
"GRID_ID",
"STUDY_VARIANT",
"TIME_PERIOD",
"EMISSIONS_SCENARIO",
"EVENT_FREQUENCY",
"T_MRT",
"T_UTCI",
"T_A",
"DISCOMFORT_LEVEL"
],
"rows": [
{
"rownum": 0,
"values": [
"500mE46385N28030",
"BASELINE",
"Baseline",
"Baseline",
"Frequent",
42.08397989280555,
39.39331065564088,
26.725610429438873,
4.0
]
},
{
"rownum": 1,
"values": [
"500mE46385N28035",
"BASELINE",
"Baseline",
"Baseline",
"Frequent",
44.39060240696659,
39.96050913187309,
27.13172453842113,
4.0
]
},
{
"rownum": 2,
"values": [
"500mE46385N28040",
"BASELINE",
"Baseline",
"Baseline",
"Frequent",
44.70946898553909,
40.03891842354406,
27.187865591257548,
4.0
]
},
{
"rownum": 3,
"values": [
"500mE46385N28045",
"BASELINE",
"Baseline",
I have just re-calculate the study 37. https://csis.myclimateservice.eu/study/37/step/1644/view/table is making a call to emikat that returns the results, but no results are shown in the table.
This is what it calls: https://service.emikat.at/EmiKatTst/api/scenarios/3207/feature/view.2974/table/data?rownum=100&filter=SZ_ID=3207&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Frequent%27&column=GRID_ID&column=STUDY_VARIANT&column=TIME_PERIOD&column=EMISSIONS_SCENARIO&column=EVENT_FREQUENCY&column=T_MRT&column=T_UTCI&column=T_A&column=DISCOMFORT_LEVELWhen I use this request, emikat does only return the column names, which are also shown in the table, but no data for the columns.
Yes, it seems to be a permission problem on Emikat-side. The "clarity GAST" user (which is used by the Table component) doesn't get any results, only the column names are returned. I talked to Heinrich, he is now investigating this.
* Why the map calls the EMIKAT geoserver 4x on every change is not clear. Should be 1x, 2x at most IMO.
leaflet that is used to draw the map, uses tiles. Although large tiles are used (1536px), up to 4 requests per layer can be made. This depends on the position on the map
OK. now I'm able to create new studies and steps are copied. However, there are a few more issues:
- Vulnerability Step is created but the respective tab is missing here. I assume @patrickkaleta implemented clarity-h2020/docker-drupal#141 (comment) ?
No, I haven't done anything related to that. Turned out that the Vulnerability GL-Step in that Study didn't have a EU-GL taxonomy term assigned to it. That's why it wasn't shown in the Study navigation. I added the taxonomy term reference by hand and now this step is shown in the Study.
Not sure, why the reference field was empty, because the template for this Study type has the right taxonomy term assigned to it.
As I said above, this step was indeed missing the link to gl taxonomy and I fixed that. Probably the one study by Pascal was made in the meantime.
Another thing: the rules didn't work yesterday. After talking to @fgeyer16 , I cleaned the cache and everything worked as it should. I already forgot this issue but here it is again.
@fgeyer16 will take a look at beta9 next week. In the meantime, I would still recommend cleaning the cache occasionally with a cron job...
@fgeyer16 will take a look at beta9 next week. In the meantime, I would still recommend cleaning the cache occasionally with a cron job...
What I was able to find out last week was that this patch (fixing BR issue 3043752) does not get applied anymore in BR beta9.
However, the file affected by this patch (SaveEntityVariableAction.php) didn't change in BR beta9, so it's unclear to my, why the patch was applied successfully in the past but not anymore.
Furthermore, Composer claims that during an update this patch gets applied, even though it doesn't. Might be somehow related to this composer-patches issue with silent fails, although we are using a newer version, which apparently should've fix that problem. So, this remains as another mystery...
Has this been resolved now in clarity-h2020/docker-drupal#148 ?
Has this been resolved now in clarity-h2020/docker-drupal#148 ?
Yes, this has been resolved.