Run namespace and Request serialization
aaronflorey opened this issue · 2 comments
Would you accept a PR that adds a namespace to a run?
My use case is that i want to write a RequestSchedulerInterface
that runs in Redis so i can resume runs, but there's no way to make the run consistent across runs without passing through something from the spider. So my solution is to pass the Spider class name into the Run and then i can pass that to the scheduler.
Another thing that i need to support is serialization of requests which isn't possible currently since the callable is stored as a closure. So i propose we store the callable and convert it to a closure later, that way i can serialize the request for later use.
As i said in my other #102, i assume there's reasoning behind the way things are done currently so i won't make a PR until i get your go ahead to make these outlandish changes!
I've thought about adding some kind of namespace or identifier for a run before since that would also help in solving #36. It would definitely be a breaking change, however, since it would at least require changing the Run
class to accept an additional parameter. This is something that I would want to implement myself, but I'm open for ideas on what the API could look like.
About serialization: storing the callable wouldn't actually solve the problem because Closure
satisfies the callable
type, too. So it would still be possible to pass a Closure
to the $parseMethod
parameter of the Request
class. I think the best way to deal with this would be implementing the __sleep
and __wakeup
magic methods for the Request
class so we can properly handle serialization. We could then use opis/serializable-closure
or laravel/serializable-closure
to deal with the parse callback.
Awesome, thanks for the reply. I'll look into implementing it!