Truncate fields too large for indexes.
Opened this issue · 9 comments
The following error occurred when generating a table row for a field containing a long string:
Exception MongoCursorException
Error
insertDocument :: caused by :: 17280 Btree::insert: key too large to index, failing kent.table_rows.$value_title_1 1085 { : "..." }
Tripod should check the length of fields being inserted and truncate where necessary.
Today we had another failure on the queue, so increasing the related PagerDuty alert to 2 items on the failed queue.
@malcyL I think my preferred approach to fixing this is when the exception happens, we look at which fields are indexed on table rows (this is in the config) and truncate all those strings to less than < 1024 bytes
@Roj this is a public repo, your comment feels like it should live on our private repo as it is specific to our use of tripod and not relevant to anyone outside Talis ;-)
Do we know the exception that gets thrown in this instance? We need to be sure we're only catching the most specific exception we can.
I think it's a MongoCursorException, message shown above
Oh, duh - right. Sorry. Bah, that's not terribly granular.
Nope, but I guess a strpos($msg,"Btree::insert: key too large to index")!==FALSE would do it.
Yep, seems like a reasonable approach.
Looking further into this, this is only failing in Mongo 2.6 and above.
See https://jira.mongodb.org/browse/SERVER-12982 - stricter enforcement of the handling of the index key limit.
However, see https://jira.mongodb.org/browse/SERVER-12834 - which adds a flag which if set allows mongo to ignore this error. This was added for people who already had indexes containing keys which exceed the limit.