Bank the DataCache object in the transactionHistory collections
Closed this issue · 0 comments
Justus-at-Tazama commented
Missed requirement/impact, now a bug, related to #154.
As per #154, the keys that are used to store and address the components of the graph related to a transaction (entity, account_holder, account, transaction) are now composed out of a number of data fields in the incoming message to ensure uniqueness across operators in a switching network. The keys are composed as follows:
- debtorKey = debtorId + debtorIdType
- debtorAccountKey = debtorAccountId + debtorAccountType + debtorAgentId
- creditorKey = creditorId + creditorIdType
- creditorAccountKey = creditorAccountId + debtorAccountType + creditorAgentId
In an ISO20022 pacs.008 message, for example, the terms above are mapped from:
Debtor:
- debtorId:
FIToFICstmrCdtTrf.CdtTrfTxInf.Dbtr.Id.PrvtId.Othr[0].Id - debtorIdType:
FIToFICstmrCdtTrf.CdtTrfTxInf.Dbtr.Id.PrvtId.Othr[0].SchmeNm.Prtry - debtorAccountId:
FIToFICstmrCdtTrf.CdtTrfTxInf.DbtrAcct.Id.Othr[0].Id - debtorAccountType:
FIToFICstmrCdtTrf.CdtTrfTxInf.DbtrAcct.Id.Othr[0].SchmeNm.Prtry - debtorAgentId:
FIToFICstmrCdtTrf.CdtTrfTxInf.DbtrAgt.FinInstnId.ClrSysMmbId.MmbId
Creditor:
- creditorId:
FIToFICstmrCdtTrf.CdtTrfTxInf.Cdtr.Id.PrvtId.Othr[0].Id - creditorIdType:
FIToFICstmrCdtTrf.CdtTrfTxInf.Cdtr.Id.PrvtId.Othr[0].SchmeNm.Prtry - creditorAccountId:
FIToFICstmrCdtTrf.CdtTrfTxInf.CdtrAcct.Id.Othr[0].Id - debtorAccountType:
FIToFICstmrCdtTrf.CdtTrfTxInf.CdtrAcct.Id.Othr[0].SchmeNm.Prtry - creditorAgentId:
FIToFICstmrCdtTrf.CdtTrfTxInf.CdtrAgt.FinInstnId.ClrSysMmbId.MmbId
These keys are then packed into the DataCache object and transmitted to the downstream processors for evaluation.
Example DataCache:
{
"DataCache": {
"dbtrId": "dbtr_9c58300b7c924ef28f685aa68efeb699TAZAMA_EID",
"cdtrId": "cdtr_c79e05f2932047a8b6ab53236afbb13eTAZAMA_EID",
"dbtrAcctId": "dbtrAcct_c08f2262448e4798b8c1b734a1bd1974MSISDNfsp001",
"cdtrAcctId": "cdtrAcct_c622e7853335471bbcc8ed376dbbb073MSISDNfsp002",
"creDtTm": "2024-07-14T06:42:16.920Z",
"amt": {
"amt": 1000,
"ccy": "XTS"
}
}
}Rule processors use the keys from the DataCache to retrieve transaction history from the database. Each node and edge in the pseudonyms graph database is identified directly by the keys from the DataCache:
entities node collection:
account_holder edge collection:
accounts node collection:
transactionRelationship edge collection:
pacs.008
pacs.002
For performance reasons, look-ups are done directly via the already indexed _key, _from, _to fields in the graph, whenever possible.
PROBLEM
For provenance and auditability reasons, the documents in the transactionHistory collection are written to the database "as received", without modification. The documents do not have a composite reference that can be addressed from the keys in the DataCache. Look-ups to the transactionHistory collections out of the rule processors using the dbtrId, cdtrId, dbtrAcctId and cdtrAcctId are currently failing. Once composed, the key cannot be decomposed for look-ups using its component parts.
Solution options
- Add the
DataCacheobject to the root of each document written to the transactionHistory collections, index the keys, and use this object to look up the documents. - Integrate the keys from the
DataCacheobject into the various (Othr) data structures inside the documents as the new element [0], with a "TAZAMA_ID" scheme. I'm disqualifying this solution out of hand because the pacs.002 message doesn't have any of the ID information and this feels like too invasive an update compared to the provenance/auditability requirement. - Move the data that is required for rule processors from the transactionHistory collections into the pseudonyms database and the
transactionRelationshipedge collection to obviate look-ups in the transactionHistory collections. This feels cleaner, at first glance, but creates a dependency in the future where we won't be able to add any rules that rely on data that's still only in the transactionHistory collections until development work is done to first transform that data into thetransactionRelationshipedge collection. There is, however, a chance that we may realise a performance improvement since we'll be eliminating cross-database multi-stage reads for some of the rule processors. - Add the decomposed key components to the
DataCacheas well, and use these to look up documents in the transactionHistory collections. - Change the lookup approach to only use
endToEndIdvalues discovered in the pseudonyms DB to look up documents in the transactionHistory collections.
Short term (release 2.0.0)
Option 1 above would be the quickest and least invasive change.
Acceptance criteria
- Out of the data preparation, after the
DataCacheis composed, add theDataCacheobject to the pain.001, pain.013, pacs.008 and pacs.002 messages before writing the messages to the transactionHistory database and the associated collections, adjacent to the root object, and along with theTxTpobject that we are also inserting. - Add the
DataCachekeys to the indexes for each of the collections. - Update the rule queries for the rule processors that interact with the
transactionHistory.transactionHistoryPacs008collection to use theDataCachefrom the document for look-ups, and not the data from theOthrobjects.
Rules impacted
Example pacs.008
pacs.008.001.10 message with embedded `DataCache` and `TxTp` objects.
{
"TxTp": "pacs.008.001.10",
"FIToFICstmrCdtTrf": {
"GrpHdr": {
"MsgId": "693cd691efac4bc79e49d5ddab66e1c3",
"CreDtTm": "2024-07-13T18:41:03.367Z",
"NbOfTxs": 1,
"SttlmInf": {
"SttlmMtd": "CLRG"
}
},
"CdtTrfTxInf": {
"PmtId": {
"InstrId": "5ab4fc7355de4ef8a75b78b00a681ed2",
"EndToEndId": "acc819be1a3447898bf8c00199b3029e"
},
"IntrBkSttlmAmt": {
"Amt": {
"Amt": 301.04,
"Ccy": "XTS"
}
},
"InstdAmt": {
"Amt": {
"Amt": 301.04,
"Ccy": "XTS"
}
},
"ChrgBr": "DEBT",
"ChrgsInf": {
"Amt": {
"Amt": 0,
"Ccy": "XTS"
},
"Agt": {
"FinInstnId": {
"ClrSysMmbId": {
"MmbId": "fsp001"
}
}
}
},
"InitgPty": {
"Nm": "April Blake Grant",
"Id": {
"PrvtId": {
"DtAndPlcOfBirth": {
"BirthDt": "1968-02-01",
"CityOfBirth": "Unknown",
"CtryOfBirth": "ZZ"
},
"Othr": [
{
"Id": "+27730975224",
"SchmeNm": {
"Prtry": "MSISDN"
}
}
]
}
},
"CtctDtls": {
"MobNb": "+27-730975224"
}
},
"Dbtr": {
"Nm": "April Blake Grant",
"Id": {
"PrvtId": {
"DtAndPlcOfBirth": {
"BirthDt": "1999-06-25",
"CityOfBirth": "Unknown",
"CtryOfBirth": "ZZ"
},
"Othr": [
{
"Id": "dbtr_9c58300b7c924ef28f685aa68efeb699",
"SchmeNm": {
"Prtry": "TAZAMA_EID"
}
}
]
}
},
"CtctDtls": {
"MobNb": "+27-730975224"
}
},
"DbtrAcct": {
"Id": {
"Othr": [
{
"Id": "dbtrAcct_c08f2262448e4798b8c1b734a1bd1974",
"SchmeNm": {
"Prtry": "MSISDN"
}
}
]
},
"Nm": "April Grant"
},
"DbtrAgt": {
"FinInstnId": {
"ClrSysMmbId": {
"MmbId": "fsp001"
}
}
},
"CdtrAgt": {
"FinInstnId": {
"ClrSysMmbId": {
"MmbId": "fsp002"
}
}
},
"Cdtr": {
"Nm": "Felicia Easton Quill",
"Id": {
"PrvtId": {
"DtAndPlcOfBirth": {
"BirthDt": "1935-05-08",
"CityOfBirth": "Unknown",
"CtryOfBirth": "ZZ"
},
"Othr": [
{
"Id": "cdtr_c79e05f2932047a8b6ab53236afbb13e",
"SchmeNm": {
"Prtry": "TAZAMA_EID"
}
}
]
}
},
"CtctDtls": {
"MobNb": "+27-707650428"
}
},
"CdtrAcct": {
"Id": {
"Othr": [
{
"Id": "cdtrAcct_c622e7853335471bbcc8ed376dbbb073",
"SchmeNm": {
"Prtry": "MSISDN"
}
}
]
},
"Nm": "Felicia Quill"
},
"Purp": {
"Cd": "MP2P"
}
},
"RgltryRptg": {
"Dtls": {
"Tp": "BALANCE OF PAYMENTS",
"Cd": "100"
}
},
"RmtInf": {
"Ustrd": "Generic payment description"
},
"SplmtryData": {
"Envlp": {
"Doc": {
"Xprtn": "2021-11-30T10:38:56.000Z",
"InitgPty": {
"Glctn": {
"Lat": "-3.1609",
"Long": "38.3588"
}
}
}
}
}
},
"DataCache": {
"dbtrId": "dbtr_9c58300b7c924ef28f685aa68efeb699TAZAMA_EID",
"cdtrId": "cdtr_c79e05f2932047a8b6ab53236afbb13eTAZAMA_EID",
"dbtrAcctId": "dbtrAcct_c08f2262448e4798b8c1b734a1bd1974MSISDNfsp001",
"cdtrAcctId": "cdtrAcct_c622e7853335471bbcc8ed376dbbb073MSISDNfsp002",
"creDtTm": "2024-07-13T18:41:03.367Z",
"amt": {
"amt": 301.04,
"ccy": "XTS"
}
}
}Long term (not this ticket)
Evaluate the potential of option 3.