Idempotency Concern of the ProcessAppendQueueItem Function

[NullPointer4096]

I would like to kindly bring attention to a potential issue in the ProcessAppendQueueItem, which is invoked by messages that enter the %TableNamespace%-ingestion queue. For each queue item, function ProcessPasswordEntry is called to first increment the pwned passwords’ prevalence stored in the table entries and then increment the prevalence in the blob storage. However, this method is not idempotent. Suppose the function is crashed after incrementing table entries; when the function retries, the table entries and blob entries will be incremented again. The values stored in the table entries hence deviate from the blob entries and the ground truth. If the retry happens after incrementing both the table entries and the blob entries, both counters will deviate from the ground truth due to duplicate updates.

Though the deviation of the prevalence counter is not a huge problem, such an issue can be resolved by maintaining a LastRequestId field in each table entry and blob entry, which stores the invocation id of the last ProcessAppendQueueItem function call. The invocation id is constant across Azure Function retries. Before updating a table/blob entry, if the comparison finds item.LastRequestId == FunctionContext.InvocationId, this means the update has happened before, so continue to iterate the next table entry or blob. Otherwise, if item.LastRequestId != FunctionContext.InvocationId, update the prevalence counter and the LastRequestId field with FunctionContext.InvocationId.

Thank you for considering this potential issue. I hope this suggestion can help improve the idempotency of the ProcessAppendQueueItem. Please feel free to reach out if you have any questions or concerns.