Received a successful response to the tx request, but tx is not stored in mempool.

[egonspace]

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):
latest

ABCI app (name for built-in, URL for self-written if it's publicly available):
cosmos-sdk

Environment:

• OS (e.g. from /etc/os-release):
• Install tools:
• Others:

What happened:
With multiple clients sending tx requests at the same time, some txs have been found to be lost without being stacked in mempools. Of course, this doesn't seem to cause a fatal flaw, but from the client's point of view, they has been successfully answered to the tx request, which is likely to cause inconvenience in usability if later found that it wasn't stored in blocks and even in mempools.

After investigating the cause, I think this scenario is possible. Suppose the mempool is almost full and processing two txs requests simultaneously. When the following code is met in the function that processes callback after successfully passing CheckTx and sending a successful response, the second thread returns an error because it is determined that the mempool is full when the first thread is filled with tx.

https://github.com/tendermint/tendermint/blob/master/mempool/clist_mempool.go#L416

What you expected to happen:

Have you tried the latest version: yes/no

How to reproduce it (as minimally and precisely as possible):

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):

Config (you can paste only the changes you've made):

node command runtime flags:

/dump_consensus_state output for consensus bugs

Anything else we need to know:

[egonspace]

I should have made this issue on tendermint/tendermint#5675.