Fix & stabilize command line, CLI, script

Question

Fix & stabilize command line, CLI, script

Closed this issue 8 years ago · 10 comments

I would like to use/help develop/ the CLI

If I try:
node src/index.js ./sample-datasets/silk-road-arrests.csv
I'm asked about the optional suites to apply, then get various errors.

If I require fs, and _ at the beginning of index.js, I make a little progress, but still lots of errors: first with the filename, then another error if I tweak the filename, and so on.

All of which makes me think I'm probably missing something obvious. Can you help me get on the right track? Thanks

Answer 1 · 2016-03-30T19:37:43.000Z

For sure @Fil! @enjalot built out the initial groundwork for the CLI. We have to get the desktop app portion up and running first, but I went ahead and made a cli branch that'll take care of those dependency issues and give you a better starting pointing.

Possible Next Steps

If you'd like to help out, we'll need to fix the addResult method in src/rendering.js to make sure it recognizes the suite we're trying to run. Feel free to work on cli and PRs are much appreciated.
If you'd want to test against data today, you can clone any suite and require it like an npm module. We'll be publishing more stabilized suites as separate npm modules around mid-April.

Answer 2 · 2016-03-30T19:56:36.000Z

I've had to npm install inquirer

Then:

isotype:Dataproofer fil$ node src/index.js ./sample-datasets/silk-road-arrests.csv
? Select optional test suites to run against your dataset. Info, Core, Mapping & Geographic, Statistics
readline.js:924
            throw err;
            ^

TypeError: path must be a string
    at TypeError (native)
    at Object.fs.readFile (fs.js:262:11)
    at null.completed (/Users/fil/Source/Dataproofer/Dataproofer/src/index.js:53:8)
    at PromptUI.onCompletion (/Users/fil/Source/node_modules/inquirer/lib/ui/prompt.js:57:10)
    at AnonymousObserver.Rx.AnonymousObserver.AnonymousObserver.completed (/Users/fil/Source/node_modules/rx-lite/rx.lite.js:1550:12)
    at AnonymousObserver.Rx.internals.AbstractObserver.AbstractObserver.onCompleted (/Users/fil/Source/node_modules/rx-lite/rx.lite.js:1489:14)
    at Subject.Rx.Subject.addProperties.onCompleted (/Users/fil/Source/node_modules/rx-lite/rx.lite.js:5871:19)
    at Subject.tryCatcher (/Users/fil/Source/node_modules/rx-lite/rx.lite.js:63:31)
    at AutoDetachObserverPrototype.completed (/Users/fil/Source/node_modules/rx-lite/rx.lite.js:5796:56)
    at AutoDetachObserver.Rx.internals.AbstractObserver.AbstractObserver.onCompleted (/Users/fil/Source/node_modules/rx-lite/rx.lite.js:1489:14)

so filename doesn't work when passed as a cli arg; let's try to type it:

isotype:Dataproofer fil$ node src/index.js
? Please enter the filename of the dataset you would like to proof ./sample-datasets/silk-road-arrests.csv
? Select optional test suites to run against your dataset. Info, Core, Mapping & Geographic, Statistics
suite name dataproofer-info-suite
suite name dataproofer-info-suite
suite name dataproofer-geo-suite
suite name dataproofer-stats-suite
add result dataproofer-info-suite { _name: 'Missing or duplicate column headers',
  _description: 'Check for errors in the header of the spreadsheet',
  _summary: '',
  _methodology: [Function],
  active: true } { passed: true,
  summary: 'No errors found in the header of the spreadsheet',
  badColumnHeads: [] }
results { undefined: {} }
/Users/fil/Source/Dataproofer/Dataproofer/src/rendering.js:36
  this.results[suite][test] = result;
                            ^

TypeError: Cannot set property '#<Object>' of undefined
    at Renderer.addResult (/Users/fil/Source/Dataproofer/Dataproofer/src/rendering.js:36:29)
    at Object.exports.run (/Users/fil/Source/Dataproofer/Dataproofer/src/processing.js:95:12)
    at /Users/fil/Source/Dataproofer/Dataproofer/src/index.js:59:18
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:404:3)

I'm blocked at about the same spot as before :(

Answer 3 · 2016-04-21T19:07:54.000Z

Quick update: just launched Dataproofer as a desktop app yesterday after going through six months as a grantee of the Knight Foundation's Prototype Fund. We want to take a second to get input from people who are currently interested in Dataproofer. If you'd value a CLI tool, click on the table below and upvote.

Answer 4 · 2016-05-12T00:48:50.000Z

@Fil we made some progress & wanted your feedback. I'll be adding options to select which tests or suites to run, but right now it runs them all. What were some other commands and options you were thinking of though?

Answer 5 · 2016-05-12T11:18:45.000Z

What I love with the CLI is that one can embed dataproofer as a step in an automated process. For this it's better if the outcome is in an easily readable format that doesn't change over time. So maybe an option to export the result as JSON would be handy. Other than that I have no great suggestions to offer :)

Not related to this issue: I can't give precise feedback, as I'm having this install problem—just spent two hours trying to install this but it keeps failing for no explicit reason, with messages such as:

pm ERR! Darwin 15.4.0
npm ERR! argv "/usr/local/Cellar/node/5.1.1/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v5.1.1
npm ERR! npm  v3.3.12
npm ERR! path /Users/fil/Source/Dataproofer/electron/node_modules/dataproofer-core-suite/node_modules/es5-ext
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename

and I don't have a clue if it's my node installation that went wrong or something else. I've tried several combinations of npm install etc, and I keep getting those errors.

Answer 6 · 2016-05-17T17:00:34.000Z

Hi I am having problem running the CLI tools from CLI branch I got the following error.

node src/index.js

? Please enter the filename of the dataset you would like to proof test.csv

? Select optional test suites to run against your dataset. Info

C:\Users\rifat\Desktop\Data Proofer\dataprooferCLI3\Dataproofer-cli\src\processi
ng.js:70
var columnHeads = loaded.columnHeads;
^

TypeError: Cannot read property 'columnHeads' of undefined
at Object.exports.run (C:\Users\rifat\Desktop\Data Proofer\dataprooferCLI3\D
ataproofer-cli\src\processing.js:70:27)
at C:\Users\rifat\Desktop\Data Proofer\dataprooferCLI3\Dataproofer-cli\src\i
ndex.js:59:18
at FSReqWrap.readFileAfterClose as oncomplete

I check I am confused with the problem-
is it module dependency issue?
can u give me some solution how to run the CLI ?

Answer 7 · 2016-05-17T18:11:36.000Z

hi @rifat963. apologies any confusion earlier. i just pushed the latest d5a9096 for you. that should fix the error you were previously seeing. FYI we're currently experiencing issues with larger files causing memory allocation errors, but we're working on improving this.

let us know if you run into any more errors or if you'd like to join our slack group if you'd like to follow the development more closely and potentially contribute! cheers

Answer 8 · 2016-05-20T12:39:15.000Z

@geraldarthur thank you for quick reply it worked; I love to join the slack group. Currently, I am analyzing the tool. My main focus is to analyze larger data set using CLI tool.

Answer 9 · 2016-09-12T17:43:29.000Z

@geraldarthur Is this issue taken care of? I think that the large file issue is a separate thing.

Answer 10 · 2016-10-30T14:28:10.000Z

This has been taken care of with the latest release. Thank you all for your correspondence and input. Further issues with large files should be filed as separate bugs (please and thank you!) to help us keep our Github issues tidy.