Reinstatement of Check for Duplicate Plans on OpenSRP Case Triggered Plans
Closed this issue · 2 comments
Recently we removed the checks for duplicates on OpenSRP. This was after the validation we had made prevent re-use of deleted index case IDs. There is a likely hood that duplicates would occur in the event OpenSRP timeouts on NiFi take longer to recover and case detail events are resubmitted. To curb this, CHAI has requested we update the logic to check for duplicates based on a query they use on their end that considers plans generated after the insertion of the index case as opposed to using the caseId no, which is what we previously did. Below is the query CHAI has used which we should consider in reinstating this functionality.
select *
from (
select identifier, title, date, dateCreated, status, p.fiReason, caseNum, last_data_date, row_number() over(partition by title order by last_data_date desc) rn
from holding.[plan] p
left join plans_with_data_2_v pwd on pwd.planId = identifier
left join focus_masterlist fm on fm.opensrp_id = jurisdiction
where p.fiReason = 'Case Triggered' and province_externalid <> '12'
) f
where rn > 1
order by date desc
TESTING
TEST PREVENTION OF NIFI GENERATED DUPLICATES {scenario 1}
Sample Case Details Event
{
"type": "Event",
"dateCreated": "2021-06-08T10:38:04.411+07:00",
"serverVersion": 1601985074968,
"identifiers": {},
"baseEntityId": "abe875a2-92f0-11ec-b909-0242ac120002",
"locationId": "8cf6788d-5c7b-467a-99e5-987ba6adbc15",
"eventDate": "2020-12-06T07:00:00.000+07:00",
"eventType": "Case_Details",
"formSubmissionId": "a142f8b6-92f0-11ec-b909-0242ac120002",
"providerId": "nifi-user",
"duration": 0,
"obs": [],
"entityType": "Case_Details",
"details": {
"id": "2206",
"age": "23",
"bfid": "7107060601",
"flag": "Site",
"species": "V",
"surname": "-",
"focus_id": "8cf6788d-5c7b-467a-99e5-987ba6adbc15",
"first_name": "เช่อ",
"focus_name": "บ้านไร่ (7107060601)",
"case_number": "1411170000065902106071700319889",
"family_name": "เช่อ",
"focus_reason": "Investigation",
"focus_status": "A1",
"house_number": "m13",
"ep1_create_date": "2021-06-07T00:00:00.000+0000",
"ep3_create_date": "2021-06-08T00:00:00.000+0000",
"investigtion_date": "2020-12-06T00:00:00.000+0000",
"case_classification": "A"
},
"version": 1557860282617,
"teamId": " ",
"_id": "d357eb36-92f0-11ec-b909-0242ac120002",
"_rev": "v1"
}
Sample Blood Screening Event
{
"_id": "77c50cd6-92f0-11ec-b909-0242ac120002",
"obs": [
{
"set": [],
"values": [
"1"
],
"fieldCode": "eligiblePerson",
"fieldType": "formsubmissionField",
"parentCode": "",
"fieldDataType": "text",
"saveObsAsArray": false,
"formSubmissionField": "eligiblePerson",
"humanReadableValues": []
}
],
"_rev": "v1",
"team": "raitest",
"type": "Event",
"teamId": "a3bf8a41-3753-4b1d-816b-3999a30250c9",
"details": {
"taskStatus": "READY",
"location_id": "eb3855da-c302-435b-b85e-2bef522d5d78",
"form_version": "0.0.1",
"locationUUID": "ce870295-b803-4185-a3af-719ea4c4b008",
"appVersionName": "5.3.2",
"planIdentifier": "daa88a0b-a0e0-4a53-a0d6-36366781eac3",
"taskIdentifier": "f28b33db-5034-40f7-a70b-3978e472b413",
"locationVersion": "0",
"taskBusinessStatus": "Not Visited"
},
"version": 1604042574985,
"duration": 0,
"eventDate": "2020-10-30T00:00:00.000+07:00",
"eventType": "blood_screening",
"entityType": "ec_family_member",
"locationId": "48cd7455-a6fb-40df-8bfe-43250f37cc16",
"providerId": "raitest",
"dateCreated": "2021-06-30T10:53:07.138+07:00",
"identifiers": {},
"baseEntityId": "538bbbdc-796d-47d6-b939-c14facb6bd47",
"serverVersion": 1607066371611,
"formSubmissionId": "8c2bb24c-92f0-11ec-b909-0242ac120002"
}
How to Add an event via the API
Make a post request to {baseUrl}/opensrp/rest/event/add with a payload in the following format. The event goes into the events Array
{
"clients": [
],
"events": [
]
}
How to retrieve an event from the API using the id
Make a GET request to {base_url}/opensrp/rest/event/sync?baseEntityId={baseEntityIdValue} .
How to generate a UUID
Use the Online UUID Generator
TESTS
A. Test that it's possible to add any other type of event via the API
- Replace the
_id,formSubmissionIdandbaseEntityIdvalues in the Sample blood screening events with generated UUIds - Fetch the event from the API using the generated id. There should be no event returned.
- Add the event via the API
- Retrieve the event via the api. You should get the event.
B. Test that it's possible to add a case details event
- Replace the
_id,formSubmissionId,baseEntityIdanddetails.case_numberwith randomly generated UUIdS. Thedetails.flagvalue should remain asSite. Replace any other values as you wish. - Fetch the event from the API using the generated id. There should be no event returned.
- Add the event via the API
- Retrieve the event via the API. You should get the event.
C. Test that it's not possible to add a case details event with the same case number and flag values
- Replace the
_id,baseEntityIdandformSubmissionIdwith randomly generated UUId In the case details event used in test B. Thedetails.flaganddetails.case_numbervalues should remain the same. Replace any other values as you wish. - Fetch the event from the API using the generated id. There should be no event returned.
- Add the event via the API.
- Retrieve the event via the API. You should not get an event.
D. Test that it's possible to add a case details event with the same case number but different flag value
- Replace the
_id,baseEntityIdandformSubmissionIdwith randomly generated UUIdS in the event used in test B. Thedetails.flagvalue should be set toSource. Thedetails.idanddetails.case_numbershould remain as is .Replace any other values as you wish. - Fetch the event from the API using the generated id. There should be no event returned.
- Add the event via the API
- Retrieve the event via the API. You should get the event.
TEST THAT RE-INSERTED CASE DETAILS GENERATE PLANS {scenario 2}
-
Test that newly added index case generate plans
a. Insert an index case
b. Using the Web UI confirm that a plan was generated -
Test that duplicate plans are not generated
a. Insert an index case record with the same details as in step 1. a) above
b. Confirm that a plan is not generated for this -
Test that it is possible to update index case details and new plan is generated when the original details failed to create a plan
a. Insert an index case. Use a value that does not exist in opensrp for thebfidfield. This will ensure that not plan is generated for this particular index case.
b.Confirm that no plan is generated for this index case
c. Insert an index case record with the same details as in step 3. a) above (note that since this is a new record, a different case ID value will be generated ), however update thebfidfield with a value that exists in opensrp
d. Confirm that a plan is generated for this -
Test that re-inserted index case details without edits do not create duplicates
a. Delete the index case record created in 3c.
b. Insert an index case record with the same details as 3c (no changes to be made on the index case details)
c. Confirm that no new plan is generated for this index case
@Rkareko @samkanga @ciremusyoka
This has been tested.
see below findings and details.
Scenario 1
Followed the steps that have been highlighted above.
Test A
- before adding the event, there's no event returned with that ID.
- after adding the event there is an event returned with that id
base entitiy ID used1476150f-738e-4755-b75a-bed6f6a5aad7
Test B
- no event fetched with that id
- after adding case details event, an event was returned.
base entity used22ee8be3-c523-4afe-b6e2-0d3003448d4d
Test C
There is an error response when you try to add an event with the same Case ID and same flag value
tested using
base entity id b345b1e4-366e-4513-93ce-f63f0ef96cbe
Case ID `613e4af3-dbf5-4a7f-a720-63a8385281fd'
see error response in the screenshot below.
Test D
Same Case ID with a different flag value can be added.
used source for this.
Case ID used: 613e4af3-dbf5-4a7f-a720-63a8385281fd
base entity used: 6191597f-4528-47b1-aad8-8cdaea20c9cb
scenario 2
Test 1
-inserted index case generated plans (a site and a source plan)
Test 2.( FAILS QA)
-With the same details and a different Case ID , plan does not generate.
However, with the same details but different Case ID and and a different plan name a plan generates. Example
3d8e19b0-1fbf-4b76-9647-924a980f3ca3 (site).
Test 3
b) - Used a plan with a new Case ID and a bfid that does not exist in opensrp.
- the plan above did not generate.
c) - used plan used in a and b above with a new Case ID and an existing opnsrp bfid.
- the plan generated.
Test 4.
Passes QA . NO plan is generated when an index case is inserted with the same index case details.
Screenshot:
scenario 1 (Test C)
