PIT support for Opensearch 2.7/2.8
Closed this issue · 2 comments
Hi gilles,
This issue is to exchange thoughts on how to support the above. I would like to check a few things with you before I create a PR.
In the release-notes for 2.0.4 you write:
_pit API seems no longer supported in the latest Opensearch 2.7 release. I've restricted searchAfter to not allow OS1/OS2 anymore. If somebody needs this, please create a PR to add support for the new APIs in Opensearch. Or you can use scrolling searches
This surprised me because it appears in the docs for both versions 2.7 and 2.8
I also checked the docs of the Elastic counterpart, and I see no differences between the two.
What am I missing? Because from the looks of it, it seems to me it would 'just' work?
Full disclosure: In the end, what I will need is PIT-based slicing support :)
A few things I've learned: the restful-url paths for opensearch are different, creating a pit is at /my-index-pattern/_search/point_in_time
as opposed to elastic's /my-index-pattern/_pit
.
object OpensearchPit {
suspend fun createPointInTime(index: String, keepAlive: Duration, searchClient: SearchClient): String {
val result = searchClient.restClient.post {
path(index, "_search", "point_in_time")
parameter("keep_alive", "${keepAlive.inWholeSeconds}s")
}
val id = searchClient.json.parseToJsonElement(result.getOrThrow().text).jsonObject["pit_id"]?.jsonPrimitive?.content
require(id != null, lazyMessage = { "opensearch API did not contain pit id. Payload was ${result.getOrNull()}" })
return id
}
suspend fun deletePointInTime(id: String, searchClient: SearchClient): JsonObject {
val body = mapOf("pit_id" to listOf(id))
return searchClient.restClient.delete {
path("_search", "point_in_time")
rawBody(searchClient.json.encodeToString(body))
}.parse(JsonObject.serializer())
}
}
So as far as I can see if we make the existing method for creating a PID aware of this difference, it seems fairly simple to add the support for creating them.
As far as using the PIT is concerned, I used something that looks like the 'hard way' as explained in the kt-search manual.