evadne/etso

Is Etso compartible with DETS?

Enuzo opened this issue ยท 10 comments

Enuzo commented

Hello,
Can Etso be used with DETS? If yes, is Ecto Changeset/Update operation available? Only the Insert, Get, Delete features appears to have been listed.

Laymer commented

Hi :)

I believe @evadne is probably the best person to answer this question, but I hope I can at least provide a little help :)

Please correct me if I am mistaken, but if I understand well your question, you would like to use a DETS-based storage backend instead of the current ETS one ?

To my knowledge, there is no currently implemented feature in Etso that allows switching to DETS storage. But regarding the Changeset/Update operations, I believe they are implemented and tested, as in the case below :

test "Promote to Customer" do
    Model.Employee
    |> where([x], x.title == "Vice President Sales" and x.first_name == "Andrew")
    |> Repo.one()
    |> Model.Employee.changeset(%{title: "Customer"})
    |> Repo.update()
  end

Technically it is probably possible to implement an alternative adapter that relies on DETS. The adapter behavior might even be extracted into a higher level behavior with ETS configured by default and additional backends could be plugged instead if they implement a common API.

The dets and ets OTP modules have a very similar API, which seems to make it rather simple to implement a DETS-based module. And that might even actually be the case and work properly. But my personal concern is that dets comes with limitations that might be hard to handle in some cases.

As pointed out in the Erlang docs, there is a 2 GB size limit per DETS file :

The size of Dets files cannot exceed 2 GB. If larger tables are needed, table > fragmentation in Mnesia can be used.

Also, and probably what would be my biggest concern, I am not sure how safe traversal could be guaranteed using DETS. Looking at the ways it can be achieved in ETS docs :

A table traversal is safe if either :

  • the table is of type ordered_set.

  • the entire table traversal is done within one ETS function call.

  • function safe_fixtable/2 is used to keep the table fixated during the entire
    traversal.

For all 3 ways to ensure safety with ETS, there is a DETS counterpart that could make it difficult :

  • There is no ordered_set type for DETS tables
  • If more than 2 GB of data needs to be traversed, a single dets:traverse/2 function call can not go through all the values
  • dets:safe_fixtable/2 documentation indicates that dets only prevents the hash list of a table to be resized when it is fixed. If elements are added concurrently while a table is fixed, there are no guarantees that traversing will work properly

One alternative would be to keep DETS tables in RAM using the {ram_file, boolean()} when opening the files. But fragmentation could arise, so one way to avoid it would be to set {repair, force} as well. An update would then behave more or less like a transaction, opening a file, performing CRUD operations and then closing the file so that it is actually flushed from RAM to disk.

But I am not sure if implementing something like that would not be reinventing the wheel : persistence transactions for CRUD operations can be provided by DBs such as PostgreSQL, and Ecto can wrap around it out-of-the-box. So I think that would be my solution, except if there is a specific requirement for DETS, which would require it to be implemented. But from what I know, the main goal of Etso is to enable in-memory storage with ETS's numerous advantages in terms of safety and performance. DETS on the other hand is mostly used by Mnesia, so it would perhaps be another solution to actually go directly that way and use Ecto as an adapter for Mnesia databases ?

Hope this helps @Enuzo ๐Ÿ™‚

Best,
Igor

evadne commented

Remember this? #3

evadne commented

@Laymer I suppose we should put out a documentation update on the topic of DETS usage. If you agree, I shall aggregate your very wise words.

  • ๐Ÿ‘1
  • ๐Ÿ˜„1
Laymer commented

Remember this? #3

Damn, my memory is definitely not persistent ๐Ÿ˜‚

I'd certainly be very glad to contribute for documentation, I just hope you are right about the wiseness of my input ๐Ÿ˜

  • ๐Ÿ˜„1
evadne commented

@Enuzo yo

  • ๐Ÿ‘€1
evadne commented

I shall close this for now. Please reopen if required.

mhsdef commented

FWIW, simple DETS usage would be great for no dependency test situations.

  • Without DETS, I'll need to spin a Postgres or MySQL backend container to use Ecto.
  • If etso had disk persistence (DETS, or if somehow it's simpler for you, Mnesia), I could stay all Elixir. No container sides.

Yeah, 2 GB limit, but I can live with that. I'm just doing dozens, hundreds, of simple records and I know it up front.

Thanks for your work here btw. Very cool stuff.

evadne commented
mhsdef commented

Yep, certainly could, shutdown...dump to flat file, startup, load from flat file. Just some flag that does that "magically" -- disk_persist: true.

I'll try to PR if I can manage to unload some other stuff elsewhere off my plate.

evadne commented

all operations performed by Dets are disk operations

Thinking it would be out of scope for etso to gain 1st class dets support, I think the most that I will be ok with would be some kind of startup, checkpoint, and shutdown hooks. Then there can be a dets plug-in to work with these hooks.

Essentially, use ETS as write-through cache via etso