difi/dcat-harvester

Høste frå data.norge.no

Closed this issue · 7 comments

hoyum commented

http://demo.difi.no/datanorge/api/dcat2/991825827/data.jsonld

Har vi gjort rett her eller er vi heilt på viddene

hmottestad commented 
ValidationError{className='Category', ruleId=167, ruleSeverity=warning, ruleDescription='skos:Concept does not exist.', message='null', s=null, p=null, o=null}
ValidationError{className='Category scheme', ruleId=168, ruleSeverity=warning, ruleDescription='skos:ConceptScheme  does not exist.', message='null', s=null, p=null, o=null}
-------------
ValidationError{className='Catalog', ruleId=25, ruleSeverity=error, ruleDescription='dct:issued should be a literal typed as date or dateTime.', message='null', s=http://data.norge.no/dcat/991825827, p=http://purl.org/dc/terms/issued, o=2014-09-25}
ValidationError{className='Catalog', ruleId=32, ruleSeverity=error, ruleDescription='dct:modified should be a literal typed as date or dateTime.', message='null', s=http://data.norge.no/dcat/991825827, p=http://purl.org/dc/terms/modified, o=2014-09-25}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1276, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/215, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/583, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/584, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1152, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/elma@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1272, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1264, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1268, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1237, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/oystein.asnes@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/813, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/kristian.kjollesdal@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/580, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1357, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/john.nonseid@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1233, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/ibj@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1175, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/mettelouise.mannsaker@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1274, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1278, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/166, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/kristian.kjollesdal@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/581, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/183, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/martin.standley@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/430, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/john.nonseid@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/966, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/jon.hakon.odd@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/579, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/811, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/kristian.kjollesdal@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1270, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/126, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/Jakob.Andre.Sandal@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1266, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/anr@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/95, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/jon.hakon.odd@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/1231, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/ibj@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/585, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
ValidationError{className='Dataset', ruleId=43, ruleSeverity=error, ruleDescription='dcat:contactPoint should be a vcard:Kind.', message='null', s=http://data.norge.no/dcat/dataset/991825827/582, p=http://www.w3.org/ns/dcat#contactPoint, o=http://data.norge.no/contacts/msc@difi.no}
hmottestad commented

Jeg har prøvd med RDFS resonering (lastet inn vcard ontologien) og da blir det følgende output:

ValidationError{className='Category', ruleId=167, ruleSeverity=warning, ruleDescription='skos:Concept does not exist.', message='null', s=null, p=null, o=null}
ValidationError{className='Category scheme', ruleId=168, ruleSeverity=warning, ruleDescription='skos:ConceptScheme  does not exist.', message='null', s=null, p=null, o=null}
-------------
ValidationError{className='Catalog', ruleId=25, ruleSeverity=error, ruleDescription='dct:issued should be a literal typed as date or dateTime.', message='null', s=http://data.norge.no/dcat/991825827, p=http://purl.org/dc/terms/issued, o=2014-09-25}
ValidationError{className='Catalog', ruleId=32, ruleSeverity=error, ruleDescription='dct:modified should be a literal typed as date or dateTime.', message='null', s=http://data.norge.no/dcat/991825827, p=http://purl.org/dc/terms/modified, o=2014-09-25}

Så da er det kun de to literalene som trenger typing.

hmottestad commented

Slik ser contact point ut med resonering:

<http://data.norge.no/contacts/martin.standley@difi.no>
        a               vcard:Individual , rdfs:Resource , vcard:Kind ;
        vcard:fn        "Martin Standley" ;
        vcard:hasEmail  "martin.standley@difi.no" .
hmottestad commented

For å fikse datatype så kan man enten legge det til i kontekst:

"issued":"2014-09-25"

--- MED ---

 "@context" : {
    "issued" : {
      "@id" : "http://purl.org/dc/terms/issued",
      "@type" : "http://www.w3.org/2001/XMLSchema#date"
    }
}

Eller rett i dataene:

--- FRA ---
"dct:issued":"2014-09-25"
--- TIL ---
"dct:issued":{"@value":"2014-09-25", "@type":"http://www.w3.org/2001/XMLSchema#date"}
hmottestad commented

Her er uansett måten jeg gjør resonering på.

        Model model = FileManager.get().loadModel("http://demo.difi.no/datanorge/api/dcat2/991825827/data.jsonld");

        Model vcard = FileManager.get().loadModel("https://www.w3.org/2006/vcard/ns");

        Model schema = ModelFactory.createDefaultModel();
        schema.add(vcard);

        InfModel schemaInf = ModelFactory.createInfModel(ReasonerRegistry.getRDFSReasoner(), schema);

        InfModel infModel = ModelFactory.createInfModel(ReasonerRegistry.getRDFSReasoner(), schema, model);


        final Model finalModel = model;
        ExtendedIterator<Statement> stmts = infModel.listStatements().filterDrop((o) -> {
                    return finalModel.contains(o) ||
                            schema.contains(o) ||
                            schemaInf.contains(o) ||
                            o.getObject().equals(RDFS.Resource) ||
                            o.getObject().equals(RDFS.Class)
                            ;
                }
        );

        model.add(new StmtIteratorImpl(stmts));
hoyum commented

Alt som førte til feil er fiksa. No skal hausting fungere

hmottestad commented

Ser bra ut.