data-dot-all/dataall

Metadata Forms Enhancements and Development

Opened this issue · 3 comments

SofiaSazonova commented

  • MF Enforcement. Description

  • MF attachement:
    - [x] Organizations
    - [ ] Organization Teams
    - [x] Environments
    - [ ] Environment Teams
    - [x] Datasets
    - [ ] Worksheets
    - [ ] Dashboards
    - [ ] Consumption roles
    - [ ] Notebooks
    - [ ] ML Studio entities
    - [ ] Pipelines

  • Search by MF fields

  • Target Entities. E.g. MF A can be attached only to Datasets.

  • Rollback all MF changes when fields are not created

  • Draft mode?

  • UI
    - [ ] Display value of glossary node
    - [ ] view attached entities

  • Integration tests

noah-paige commented

Some additional enhancements / fixes requested for Metadata Forms (cc: @rbernotas):

Enhancement/ Fix Requests:

  • [Nice to have] "Add Field" button also at bottom in the MF Create form, for MF's with many fields to enter
  • [Nice to have] "Attach" button at bottom of the MF attach Form, for MF’s with many values to enter
  • Need to be able to audit: For a given MF, which Datasets have attached it, and which Datasets have not?
  • We should be able to export/download metadata form data, either from a single attached metadata form, or in bulk (like from all datasets, for a given metadata form)
  • Catalog Search (mentinoed above but in more detail)
    • Filter to see only which datasets have a particular metadata form attached
    • Broadly search for some text that might appear in any metadata form that is attached to any dataset
    • Filter on a specific metadata form field, presence of that field in a dataset, or value of it.
  • Input Validation and UI View
    • (BUG) value entry in the MF fields isn’t checked for string length. Tried entering 37k chars in a field, got a network error. String input should also be sanitized in some way to check for injection attacks, etc.
    • (BUG) a long string in a string entry field, once attached and rendered, is truncated. I saw this starting to happen around 68 characters.
    • (BUG) when attaching a MF, the name of the field can be truncated if the Name is long. Seems like the name should be outside of the value entry box, and line wrapped. Also seems like we should be checking and/or limiting number of characters for the Name of a field.
    • (BUG) after a MF is attached, if the name of the field is very long, in the metadata table the value of the field will be truncated.
  • Required boolean field isn't useful, have to always pick True (bug?) or double-click UX is confusing
  • Currently there is no enforcement, and the MF form tab for Enforcement reads "Under Construction"
    • Let's remove this Tab and add back when we can support Enforcement
  • If a MF is already attached to a dataset - Need to be able to edit existing values of already attached MF:
    • I can edit the MF fields as the owner of the MF
    • Adding a field does not require the dataset to enter a value for the new field
      • It's not shown, there is also no way to edit the data in the attached form on the Dataset
      • Seems like we should be able to add a field to the MF, and then there should be a process to update the attached MF on Datasets, with the new field and value.
    • Deleting a field DOES remove it from any attached metadata forms on the Datasets.
      • This seems fine, but also feels inconsistent if I can't also add fields to already attached MF's.

@SofiaSazonova - I think we already have some parts of the above around:

  • enforcement strategy
  • View attached entities / audit enhancement
  • catalog search
  • attach metadata form editability
    ... all as part of future release but please confirm if that is the case.

Otherwise, I see a couple of other requests in the above, mainly:

  • Input Validation / UI Fixes for String Length + Boolean Field
  • Export/Download MF data
  • Additional Buttons

Maybe we can figure out if/how we plan to prioritize the other pieces?

SofiaSazonova commented

@noah-paige thanks for the list.
I can confirm, that editability, enforcement and audit are planned as the next steps.
All other features and bugs I will prioritise as we come to the next sprint

rbernotas commented

Our top priorities would be in this order:

  1. Enforcement - ability to enforce a given metadata form across all datasets in data.all
  2. Edit attached form - ability for user to edit values in a metadata form that has been attached to a dataset already, without having to detach and reattach the form.
    1. UI view enhancements for attached metadata forms (fix things like truncation of values and input fields, etc.).
  4. Discoverability using metadata form information in Catalog search
  5. Ability to audit what has been attached to a metadata form

The rest are also valuable but these would be our top 5, with particular emphasis on 1 & 2. Thanks!