Solve the numerous Access Denied problems (shared filesystems)
Closed this issue · 3 comments
GoogleCodeExporter commented
The adfi-hub logs show a variety of areas where external file locks are causing
problems, and most of the are handled badly resulting in files not transforming
or (at worst) not even getting indexed.
Given that most of these scenarios occur in the tool chain, here's the start of
an idea for brainstorming and evolution:
* Build a new queue consumer to be used as a holding area waiting on file locks.
* Each part of the tool chain tests incoming files for a lock. If something else has a lock on the file, instead of going into the conveyor belt it forwards the message to the holding area.
* The holding area keeps a list of files we are waiting on and which part of the toolchain sent them to the holding area.
* At configurable time intervals the holding areas attempts to access each file on the list. If access is secured, the message is bounced back to the toolchain where it came from.
* If items remain in the holding area for too long (again configurable) it will spawn a message to the email queue. In our workflow it would probably notify adfi_dev that particular files have been locked for X amount of time.
* After the situation is resolved outside the system the holding area will notice during its next 'timeout' and proceed as usual.
The idea is basically trying to address three concerns:
* Failures aren't handled well, and without file access TF can't do much with the file.
* Failures aren't 'broadcast' so some files can remain locked for days and just not appear in the system.
* We don't want an email based solution spamming. So the holding area ensure a) that it's not a shot term lock, b) we only send emails after/every significant time period (likely daily).
Original issue reported on code.google.com by greg.pen...@gmail.com on 1 Aug 2011 at 5:17
GoogleCodeExporter commented
Original comment by greg.pen...@gmail.com on 11 May 2012 at 3:36
- Changed title: Solve the numerous Access Denied problems (shared filesystems)
GoogleCodeExporter commented
Original comment by greg.pen...@gmail.com on 23 May 2012 at 1:09
- Added labels: Type-Task
GoogleCodeExporter commented
Migrated to https://github.com/the-fascinator/the-fascinator/issues/25
Original comment by duncan.q...@gmail.com on 14 Jan 2013 at 5:44
- Changed state: MovedToGitHub