Running CBBuild on Autism Data

Question

Running CBBuild on Autism Data

itskhurrams opened this issue 3 years ago · 7 comments

Hi,
I have successfully build and run "Mini" and its working fine.

Now i download Autism Data place it in a folder. but when i try to run cbbuild on "Autism" it gives me following error. Please help. because i have to build mutiple datasets.

PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild --init
INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\cellbrowser.conf
INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\desc.conf
PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild -o D:\CellBrowser\cellBrowser\src\public_html -p 8888
INFO:root:dataRoot is not set in ~/.cellbrowser.conf or via $CBDATAROOT. Dataset hierarchies are not supported.
INFO:root:Determining if D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz needs to be created
INFO:root:D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz does not exist. Must build matrix now.
INFO:root:Checking and reordering meta data to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv
INFO:root:Reading sample names from D:\CellBrowser\cellBrowser\src\Autism\meta.tsv
INFO:root:Reading headers from file D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz
INFO:root:Data contains 104559 samples/cells
INFO:root:Converting to numbers and compressing meta data fields
WARNING:root:File colors.tsv does not exist
WARNING:root:D:\CellBrowser\cellBrowser\src\Autism\acronyms.tsv specified in config file, but does not exist, skipping
INFO:root:Field cell: type uniqueString, 104559 different values
INFO:root:Field cluster: type enum, 17 different values
INFO:root:Field sample: type enum, 41 different values
INFO:root:Field individual: type int, 31 different values
INFO:root:Field region: type enum, 2 different values
INFO:root:Field age: type int, 14 different values
INFO:root:Field sex: type enum, 2 different values
INFO:root:Field diagnosis: type enum, 2 different values
INFO:root:Field Capbatch: type enum, 8 different values
INFO:root:Field Seqbatch: type enum, 3 different values
INFO:root:Field postMinusmortemintervalhours: type int, 18 different values
INFO:root:Field RNAIntegrityNumber: type float, 18 different values
INFO:root:Field genes: type int, 8202 different values
INFO:root:Field UMIs: type int, 20938 different values
INFO:root:Field RNAmitochondrpercent: type float, 64073 different values
INFO:root:Field RNAribosomalpercent: type float, 62056 different values
INFO:root:Indexing meta file D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.index
INFO:root:Kept 104559 cells present in both meta data file and expression matrix
WARNING:root:Gunzip not in PATH, falling back to Python's built-in
INFO:root:Auto-detecting number type of D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz
INFO:root:Auto-detect: Numbers in matrix are of type 'float'
INFO:root:Auto-detected gene IDs type: gencode-human
INFO:root:Reading gene,symbol mapping from D:\CellBrowser\cellBrowser\cellbrowserData\genes\gencode-human.symbols.tsv.gz
INFO:root:Copying/compressing D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz to D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz
'cp' is not recognized as an internal or external command,
operable program or batch file.
ERROR:root:Could not run: cp "D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz" "D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz"
ERROR:root:Unexpected error: (<class 'SystemExit'>, SystemExit(1), <traceback object at 0x000001E5A624A900>)
Traceback (most recent call last):
File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4783, in cbBuildCli
build(confFnames, outDir, port, redo=options.redo)
File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4598, in build
convertDataset(inDir, inConf, outConf, datasetDir, redo)
File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3955, in convertDataset
convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix)
File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3285, in convertExprMatrix
matType = copyMatrixTrim(matrixFname, outMatrixFname, metaSampleNames, needFilterMatrix, geneToSym, outConf, matType)
File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2525, in copyMatrixTrim
ret = runCommand(cmd)
File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2459, in runCommand
errAbort("Could not run: %s" % cmd)
File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 168, in errAbort
sys.exit(1)
SystemExit: 1

Thanks in Advance.
Khurram

Answer 1 · 2021-10-25T13:19:37.000Z

Hi Khurram, thank you for finding this bug. I see that you're using Windows and while the output is missing a piece, I think I can guess from this output what the problem is. I forgot that the copy command is not "cp" on Windows. This is exactly why I ask in the documentation to contact us if Windows problem appear, I didn't test this enough on Windows. I tried to fix this now, by using the "copy" command on Windows, can you re-install the software again using the "--pre" option? pip install cellbrowser --pre Should give you version 1.2.0a3. Then, if you re-run the cbBuild command, the file copy operation should not fail anymore. Thanks again and let us know if this solved the problem, Max

…

On Sat, Oct 23, 2021 at 10:43 AM Khurram Shahzad ***@***.***> wrote: Hi, I have successfully build and run "Mini" and its working fine. Now i download Autism Data place it in a folder. but when i try to run cbbuild on "Autism" it gives me following error. Please help. because i have to build mutiple datasets. PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild --init INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\cellbrowser.conf INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\desc.conf PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild -o D:\CellBrowser\cellBrowser\src\public_html -p 8888 INFO:root:dataRoot is not set in ~/.cellbrowser.conf or via $CBDATAROOT. Dataset hierarchies are not supported. INFO:root:Determining if D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz needs to be created INFO:root:D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz does not exist. Must build matrix now. INFO:root:Checking and reordering meta data to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv INFO:root:Reading sample names from D:\CellBrowser\cellBrowser\src\Autism\meta.tsv INFO:root:Reading headers from file D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz INFO:root:Data contains 104559 samples/cells INFO:root:Converting to numbers and compressing meta data fields WARNING:root:File colors.tsv does not exist WARNING:root:D:\CellBrowser\cellBrowser\src\Autism\acronyms.tsv specified in config file, but does not exist, skipping INFO:root:Field cell: type uniqueString, 104559 different values INFO:root:Field cluster: type enum, 17 different values INFO:root:Field sample: type enum, 41 different values INFO:root:Field individual: type int, 31 different values INFO:root:Field region: type enum, 2 different values INFO:root:Field age: type int, 14 different values INFO:root:Field sex: type enum, 2 different values INFO:root:Field diagnosis: type enum, 2 different values INFO:root:Field Capbatch: type enum, 8 different values INFO:root:Field Seqbatch: type enum, 3 different values INFO:root:Field postMinusmortemintervalhours: type int, 18 different values INFO:root:Field RNAIntegrityNumber: type float, 18 different values INFO:root:Field genes: type int, 8202 different values INFO:root:Field UMIs: type int, 20938 different values INFO:root:Field RNAmitochondrpercent: type float, 64073 different values INFO:root:Field RNAribosomalpercent: type float, 62056 different values INFO:root:Indexing meta file D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.index INFO:root:Kept 104559 cells present in both meta data file and expression matrix WARNING:root:Gunzip not in PATH, falling back to Python's built-in INFO:root:Auto-detecting number type of D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz INFO:root:Auto-detect: Numbers in matrix are of type 'float' INFO:root:Auto-detected gene IDs type: gencode-human INFO:root:Reading gene,symbol mapping from D:\CellBrowser\cellBrowser\cellbrowserData\genes\gencode-human.symbols.tsv.gz INFO:root:Copying/compressing D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz to D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz 'cp' is not recognized as an internal or external command, operable program or batch file. ERROR:root:Could not run: cp "D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz" "D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz" ERROR:root:Unexpected error: (<class 'SystemExit'>, SystemExit(1), <traceback object at 0x000001E5A624A900>) Traceback (most recent call last): File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4783, in cbBuildCli build(confFnames, outDir, port, redo=options.redo) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4598, in build convertDataset(inDir, inConf, outConf, datasetDir, redo) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3955, in convertDataset convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3285, in convertExprMatrix matType = copyMatrixTrim(matrixFname, outMatrixFname, metaSampleNames, needFilterMatrix, geneToSym, outConf, matType) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2525, in copyMatrixTrim ret = runCommand(cmd) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2459, in runCommand errAbort("Could not run: %s" % cmd) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 168, in errAbort sys.exit(1) SystemExit: 1 Thanks in Advance. Khurram — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#229>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACL4TPMDHYE2XNSHHJ4O23UIJYTJANCNFSM5GSAY5SA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Answer 2 · 2021-10-26T07:31:56.000Z

Hi Max,
Thank you for your support i really appreciate. Actually i am running it from code so should i download the developer branch in order to get this fixed ?

Answer 3 · 2021-10-26T07:37:01.000Z

Yes, if you use the git repo, just do a "git pull" and then "git checkout develop". Then, re-run the cbBuild command. Let me know if there is still a problem. I don't have a Windows machine, so my options of testing this are limited.

…

On Tue, Oct 26, 2021 at 9:32 AM Khurram Shahzad ***@***.***> wrote: Hi Max, Thank you for your support i really appreciate. Actually i am running it from code so should i download the developer branch in order to get this fixed ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACL4TMXRJ74U6TMTVPN7K3UIZKPPANCNFSM5GSAY5SA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Answer 4 · 2021-10-26T08:06:51.000Z

No Luck.
I downloaded the developer repo and run the command and i can see CP is still showing.

PS D:\CellBrowser\cellBrowserdevelop\src> cd .\Autism
PS D:\CellBrowser\cellBrowserdevelop\src\Autism> python ../cbbuild --init
INFO:root:Wrote D:\CellBrowser\cellBrowserdevelop\src\Autism\cellbrowser.conf
INFO:root:Wrote D:\CellBrowser\cellBrowserdevelop\src\Autism\desc.conf
PS D:\CellBrowser\cellBrowserdevelop\src\Autism> python ../cbbuild -o D:\CellBrowser\cellBrowserdevelop\src\public_html -p 8888
WARNING:root:The directory D:\CellBrowser\cellBrowserdevelop\src\public_html does not exist. Making a new directory now.
INFO:root:dataRoot is not set in ~/.cellbrowser.conf or via $CBDATAROOT. Dataset hierarchies are not supported.
INFO:root:Creating D:\CellBrowser\cellBrowserdevelop\src\public_html\sample
INFO:root:Determining if D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz needs to be created
INFO:root:D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz does not exist. Must build matrix now.
INFO:root:Creating D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\metaFields
INFO:root:Checking and reordering meta data to D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\meta.tsv
INFO:root:Reading sample names from D:\CellBrowser\cellBrowserdevelop\src\Autism\meta.tsv
INFO:root:Reading headers from file D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz
INFO:root:Data contains 104559 samples/cells
INFO:root:Converting to numbers and compressing meta data fields
WARNING:root:File colors.tsv does not exist
WARNING:root:D:\CellBrowser\cellBrowserdevelop\src\Autism\acronyms.tsv specified in config file, but does not exist, skipping
INFO:root:Field cell: type uniqueString, 104559 different values
INFO:root:Field cluster: type enum, 17 different values
INFO:root:Field sample: type enum, 41 different values
INFO:root:Field individual: type int, 31 different values
INFO:root:Field region: type enum, 2 different values
INFO:root:Field age: type int, 14 different values
INFO:root:Field sex: type enum, 2 different values
INFO:root:Field diagnosis: type enum, 2 different values
INFO:root:Field Capbatch: type enum, 8 different values
INFO:root:Field Seqbatch: type enum, 3 different values
INFO:root:Field postMinusmortemintervalhours: type int, 18 different values
INFO:root:Field RNAIntegrityNumber: type float, 18 different values
INFO:root:Field genes: type int, 8202 different values
INFO:root:Field UMIs: type int, 20938 different values
INFO:root:Field RNAmitochondrpercent: type float, 64073 different values
INFO:root:Field RNAribosomalpercent: type float, 62056 different values
INFO:root:Indexing meta file D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\meta.tsv to D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\meta.index
INFO:root:Kept 104559 cells present in both meta data file and expression matrix
WARNING:root:Gunzip not in PATH, falling back to Python's built-in
INFO:root:Auto-detecting number type of D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz
INFO:root:Auto-detect: Numbers in matrix are of type 'float'
INFO:root:Auto-detected gene IDs type: gencode-human
INFO:root:Reading gene,symbol mapping from D:\CellBrowser\cellBrowserdevelop\cellbrowserData\genes\gencode-human.symbols.tsv.gz
INFO:root:Copying/compressing D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz to D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz
'cp' is not recognized as an internal or external command,
operable program or batch file.
ERROR:root:Could not run: cp "D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz" "D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz"
ERROR:root:Unexpected error: (<class 'SystemExit'>, SystemExit(1), <traceback object at 0x000001B5EDCC6DC0>)
Traceback (most recent call last):
File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 4783, in cbBuildCli
build(confFnames, outDir, port, redo=options.redo)
File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 4598, in build
convertDataset(inDir, inConf, outConf, datasetDir, redo)
File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 3955, in convertDataset
convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix)
File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 3285, in convertExprMatrix
matType = copyMatrixTrim(matrixFname, outMatrixFname, metaSampleNames, needFilterMatrix, geneToSym, outConf, matType)
File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 2525, in copyMatrixTrim
ret = runCommand(cmd)
File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 2459, in runCommand
errAbort("Could not run: %s" % cmd)
File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 168, in errAbort
sys.exit(1)
SystemExit: 1
PS D:\CellBrowser\cellBrowserdevelop\src\Autism>

Answer 5 · 2021-10-26T08:08:20.000Z

itskhurrams commented 3 years ago

Answer 6 · 2021-10-26T08:22:53.000Z

Yes, somehow you still have the old version of the file. Line 2525 of cellbrowser.py on your disk has the old line, ret = runCommand(cmd), but the version of the file has these lines: if inFname.endswith(".gz"): shutil.copyfile(inFname, outFname) else: if platform.system()=="Windows": # slow but quick hack for Github #229 tmpFname = outFname+".tmp" shutil.copyfile(inFname, tmpFname) runGzip(tmpFname, outFname) Somehow D:\CellBrowser/cellBrowserdevelop is not the "develop" branch or didn't get updated from Github. Can you try git pull in that directory, "git checkout develop", check that you're on develop using "git branch" and then check line 2525 in cellbrowser.py ?

…

On Tue, Oct 26, 2021 at 10:08 AM Khurram Shahzad ***@***.***> wrote: [image: image] <https://user-images.githubusercontent.com/5083361/138835286-b67550ff-927e-4e60-8267-1ea452e63a37.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACL4TNSZA2JZ6QCFLLMM7LUIZOX7ANCNFSM5GSAY5SA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Answer 7 · 2021-10-28T12:55:29.000Z

Thanks for closing the ticket. I assume that this means that my last commit solved your problem? Let us know if you run into something else. I admit that I haven't tested on windows, but that was a long time ago.

…

On Thu, Oct 28, 2021 at 2:37 PM Khurram Shahzad ***@***.***> wrote: Closed #229 <#229>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACL4TLF7LO4MZWG7PPKOF3UJE7W3ANCNFSM5GSAY5SA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.