COPC Validator

Introduction
1. Getting Started
Usage
1. CLI
  1. Options
2. Import
  1. Options
Scans
Details
1. Checks
2. Report
  1. Report schema
Future Plans

Introduction

COPC Validator is a library & command-line application for validating the header and content of a Cloud-Optimized Point Cloud (COPC) LAS file. Extending the copc.js library, it accepts either a (relative) file path or COPC url, and runs a series of checks against the values parsed by copc.js.

Getting Started

Install from npm
```
npm i -g copc-validator
```
Global install is recommended for CLI usage
Scan copc.laz file with copcc CLI

Examples:

Default
```
copcc ./path/to/example.copc.laz
```

Deep scan, output to <pwd>/output.json

copcc --deep path/to/example.copc.laz --output=output.json

Deep & Minified scan with worker count = 64, showing a progress bar
```
copcc path/to/example.copc.laz -dmpw 64
```

Usage

COPC Validator has two main usages: via the copcc Command-Line Interface (CLI), or imported as the generateReport() function

CLI

copcc [options] <path>

The usage and implementation of COPC Validator is meant to be as simple as possible. The CLI will only need one file path and will automatically run a shallow scan by default, or a deep scan if provided with the --deep option. All other functionality is completely optional.

Options

Option	Alias	Description	Type	Default
`deep`	`d`	Read all points of each node; Otherwise, read only root point	`boolean`	`false`
`name`	`n`	Replace `name` in Report with provided string	`string`	`<path>`
`mini`	`m`	Omit Copc or Las from Report, leaving `checks` and `scan` info	`boolean`	`false`
`pdal`	`P`	Output a `pdal.metadata` object containing header & vlr data in `pdal info` format	`boolean`	`false`
`workers`	`w`	Number of Workers to create - Use at own (performance) risk	`number`	CPU-count
`queue`	`q`	Queue size limit for reading PDR data. Useful for very high node counts (>10000)	`number`	Unlimited
`sample`	`s`	Select a random sample of nodes to read & validate	`number`	All nodes
`progress`	`p`	Show a progress bar while reading the point data	`boolean`	`false`
`output`	`o`	Writes the Report out to provided filepath; Otherwise, writes to `stdout`	`string`	N/A
`help`	`h`	Displays help information for the `copcc` command; Overwrites all other options	`boolean`	N/A
`version`	`v`	Displays `copc-validator` version (from `package.json`)	`boolean`	N/A

Import

Add to project:

yarn add copc-validator
  # or
npm i copc-validator

Import generateReport():

import { generateReport } from 'copc-validator'

Example:

async function printReport() {
  const report = await generateReport({
    source: 'path/to/example.copc.laz',
    options: {} // default options
  })
  console.log(report)
}

Copy laz-perf.wasm to /public (for browser usage)

Options

generateReport accepts most* of the same options as the CLI through the options property of the first parameter:

TypeScript:

const generateReport = ({
  source: string | File
  options?: {
    name?: string          //default: source | 'COPC Validator Report'
    mini?: boolean         //default: false
    pdal?: boolean         //default: false
    deep?: boolean         //default: false
    workers?: number       //default: CPU Thread Count
    queueLimit?: number    //default: Infinity
    sampleSize?: number    //default: All nodes
    showProgress?: boolean //default: false
  },
  collections?: {copc, las, fallback}
})

See below for collections information

* Key option differences:

No output, help, or version options

queue is renamed to queueLimit

sample is renamed to sampleSize

progress is renamed to showProgress

Not usable in a browser

Any Alias (listed above) will not work

Scans

COPC Validator comes with two scan types, shallow and deep

(see requirements.md for more details)

The report output supports a custom scan type, intended to be used by other developers that may extend the base functionality of COPC Validator. It is not currently used anywhere in this library.

Shallow scan

The shallow scan checks the LAS Public Header Block and various Variable Length Records (VLRs) to ensure the values adhere to the COPC specificiations (found here)

This scan will also check the root (first) point of every node (in the COPC Hierarchy) to ensure those points are valid according to the contents of the Las Header and COPC Info VLR

Deep scan

The deep scan performs the same checks as a shallow scan, but scans every point of each node rather than just the root point, in order to validate the full contents of the Point Data Records (PDRs) against the COPC specs and Header info

Output

COPC Validator outputs a JSON report according to the Report Schema, intended to be translated into a more human-readable format (such as a PDF or Webpage summary)

Details

Checks

A Check ultimately refers to the Object created by calling a Check.Function with performCheck(), which uses the Check.Suite property name to build the returned Check.Status into a complete Check.Check. This already feels like a bit much, without even mentioning Check.Parsers or Check.Collections, so we'll break it down piece-by-piece here

Pseudo-TypeScript:

namespace Check {
  type Status = {
    status: 'pass' | 'fail' | 'warn'
    description: string
    info?: string
  }
  type Check = Status & { id: string }

  type Function<T> =
    | (c: T) => Status
    | (c: T) => Promise<Status>

  type Suite<T> = { [id: string]: Check.Function<T> }
  type SuiteWithSource<T> = { source: T, suite: Suite<T>}

  type Parser<Source, Parsed> = (s: Source) => Promise<SuiteWithSource<Parsed>>

  type Collection = (SuiteWithSource<any> | Promise<SuiteWithSource<any>>)[]
}
type Check = Check.Check

See ./src/types/check.ts for the actual TypeScript code

Status & Check Objects

A Check.Status Object contains a status property with a value of "pass", "fail", or "warn", and optionally contains an info property with a string value.

A Check Object is the same as a Status Object with an additional string property named id

pass means file definitely matches COPC specificiations
fail means file does not match any COPC specifications
warn means file may not match current COPC specifications or recommendations

Functions

Check.Functions maintain the following properties:

Single (Object) parameter
Syncronous or Asyncronous
Output: Check.Status (or a Promise)
Pure function

Suites

A Check.Suite is a map of string ids to Check.Functions, where each Function uses the same Object as its parameter (such as the Copc Object, for ./src/suites/copc.ts). The id of a Function becomes the id value of the Check Object when a Check.Suite invokes its Functions

The purpose of this type of grouping is to limit the number of Getter calls for the same section of a file, like the 375 byte Header

All Suites (with their Check.Functions) are located under src/suites

Parsers

Check.Parsers are functions that take a source Object and return a Check.SuiteWithSource Object. Their main purpose is to parse a section of the given file into a usable object, and then return that object with its corrosponding Suite to be invoked from within a Collection.

All Parsers are located under src/parsers (ex: nodeParser)

`nodes.ts`

src/parsers/nodes.ts is unique among Parsers, in that it's actually running a Suite repeatedly as it parses. However, the data is not returned from the multithreaded Workers like a regular Check.Suite, so nodes.ts then gives the output data to the (new) pointDataSuite for sorting into Check.Statuses

worker.js

src/utils/worker.js essentially matches the structure of a Suite because it used to be the src/suites/point-data.ts Suite. To increase speed, the pointDataSuite became per-Node instead of per-File, which maximizes multi-threading, but creates quite a mess since worker.js must be (nearly) entirely self-contained for Worker/Web Worker threading. So src/suites/point-data.ts now parses the output of src/utils/worker.js, all of which is controlled by the src/parsers/nodes.ts Parser

Collections

Check.Collections are arrays of Check.Suites with their respective source Object (Check.SuiteWithSource above). They allow Promises in order to use Check.Parsers internally without having to await them.

All Collections are located under src/collections (ex: CopcCollection)

Replacing Collections is the primary way of generating custom reports through generateReport, as you can supply different Check.Suites to perform different Check.Functions per source object.

Custom scan

generateReport has functionality to build customized reports by overwriting the Check.Collections used within:

Pseudo-Type:

import type {Copc, Getter, Las} from 'copc'
type Collections = {
  copc: ({
    filepath: string,
    copc: Copc,
    get: Getter,
    deep: boolean,
    workerCount?: number
  }) => Promise<Check.Collection>,
  las: ({
    get: Getter,
    header: Las.Header,
    vlrs: Las.Vlr[]
  }) => Promise<Check.Collection>,
  fallback: (get: Getter) => Promise<Check.Collection>
}

const generateReport = async ({
  source: string | File,
  options?: {...},
  collections?: Collections
}) => Promise<Report>

All Checks

ID	Description	Scan	Suite
`minorVersion`	`copc.header.minorVersion` is `4`	Shallow	`Header`
`pointDataRecordFormat`	`copc.header.pointDataRecordFormat` is `6`, `7`, or `8`	Shallow	`Header`
`headerLength`	`copc.header.headerLength` is `375`	Shallow	`Header`
`pointCountByReturn`	Sum of `copc.header.pointCountByReturn` equals `copc.header.pointCount`	Shallow	`Header`
`legacyPointCount`	`header.legacyPointCount` follows COPC/LAS specs	Shallow	`manualHeader`
`legacyPointCountByReturn`	`header.legacyPointCountByReturn` follows COPC/LAS specs	Shallow	`manualHeader`
`vlrCount`	Number of VLRs in `copc.vlrs` matches `copc.header.vlrCount`	Shallow	`Vlr`
`evlrCount`	Number of EVLRs in `copc.vlrs` matches `copc.header.evlrCount`	Shallow	`Vlr`
`copc-info`	Exactly 1 copc `info` VLR exists with size of `160`	Shallow	`Vlr`
`copc-hierarchy`	Exactly 1 copc `hierarchy` VLR exists	Shallow	`Vlr`
`laszip-encoded`	Checks for existance of LasZIP compression VLR, warns if not found	Shallow	`Vlr`
`wkt`	Ensures `wkt` string can initialize `proj4`	Shallow	`manualVlr`
`bounds within cube`	Copc `cube` envelops Las bounds (`min` & `max`)	Shallow	`Copc`
`rgb`	RGB channels are used in PDR, if present	Shallow	`PointData`
`rgbi`	Checks for 16-bit scaling of RGBI values, warns if 8-bit	Shallow	`PointData`
`xyz`	Each point exists within Las and Copc bounds, per node	Shallow	`PointData`
`gpsTime`	Each point has `GpsTime` value within Las bounds	Shallow	`PointData`
`sortedGpsTime`	The points in each node are sorted by `GpsTime` value, warns if not	Deep	`PointData`
`returnNumber`	Each point has `ReturnNumber <= NumberOfReturns`	Shallow	`PointData`
`zeroPoint`	Warns with list of all `pointCount: 0` nodes in the Hierarchy	Deep*	`PointData`
`nodesReachable`	Every `Node` (`'D-X-Y-Z'`) in the `Hierarchy` is reachable	Shallow	`PointData`
`pointsReachable`	Each Node pageOffset + pageLength leads into another Node page	Shallow	`PointData`
`...ID`	...Description	Shallow	`...`

Checks and their IDs are subject to change as I see fit

Report

Report schema

See JSON Schema

TypeScript pseudo-type Report:

import * as Copc from 'copc'

type Report = {
  name: string
  scan: {
    type: 'shallow' | 'deep' | 'custom' | string  //| 'shallow-X/N' | 'deep-X/N'
    filetype: 'COPC' | 'LAS' | 'Unknown'
    start: Date
    end: Date
    time: number
  }
  checks: ({
    id: string
    status: 'pass' | 'fail' | 'warn'
    info?: string
  })[]

  // When scan.filetype === 'COPC'
  copc?: {
    header: Copc.Las.Header
    vlrs: Copc.Las.Vlr[]
    info: Copc.Info
    wkt: string
    eb: Copc.Las.ExtraBytes
  }

  // When scan.filetype === 'LAS'
  las?: {
    header: Copc.Las.Header
    vlrs: Copc.Las.Vlr[]
  }
  error: {
    message: string
    stack?: string
  }

  // When scan.filetype === 'Unknown'
  error: {
    message: string
    stack?: string
  }
  copcError?: {
    message: string
    stack?: string
  } // only used if Copc.create() and Las.*.parse() fail for different reasons
}

Future Plans

Add more Check.Functions - waiting on laz-perf chunk table
Rewrite LAS Check.Collection to validate LAS 1.4 specifications
Continue to optimize for speed, especially large (1.5GB+) files

hobuinc/copc-validator

COPC Validator

Table of Contents

Introduction

Getting Started

Usage

CLI

Options

Import

Options

Scans

Shallow scan

Deep scan

Output

Details

Checks

Status & Check Objects

Functions

Suites

Parsers

`nodes.ts`

Collections

Custom scan

All Checks

Report

Report schema

Future Plans