600 feature limit?
aeshaw1 opened this issue · 8 comments
This program is such a break-through!!! Thank you so much for sharing it! I only encountered two issues:
- it only downloaded the first 600 features. If there is a 600 feature limit, is there a way to just request the 2nd group of 600? The service I tried was http://sigagis.conagua.gob.mx/ArcGIS/rest/services/Acuiferos_2014/MapServer/0 if you want to test it. Thanks in advance for any tips on this.
- it didn't apply the name I assigned after the | in services.txt, but no big deal, I just renamed the output.
Looking at the service, there only seems to be 653 features total. Are you only getting 600 in your final dataset? You can also incorporate your own query parameters in your url, referenced in #9. I'll look into the name designation. It's been naming my json and geojson correctly, but packages up the shapefile with a generic name.
Yes, here is what happens when I run it:
C:\Users\Allison.Shaw\AGStoShapefile-master>node AGStoSHP.js
(node:9284) fs: re-evaluating native module sources is not supported. If
you are
using the graceful-fs module, please update it to a more recent version.
(node:9284) DeprecationWarning: process.EventEmitter is deprecated. Use
require(
'events') instead.
info: Number of features for service: 653
info: Getting chunks of 100 features...
info: query -> 0 out of 653
info: query -> 100 out of 653
info: query -> 200 out of 653
info: query -> 300 out of 653
info: query -> 400 out of 653
info: query -> 500 out of 653
info: wait for requests to settle...
info: all requests settled
info: creating Acuiferos_2014_600 json
info: Creating Esri JSON
info: Creating GeoJSON
info: Creating Shapefile
So it stops when it hits the 600th feature, even now that I updated
services.txt to query just the ones after that:
http://sigagis.conagua.gob.mx/ArcGIS/rest/services/Acuiferos_2014/MapServer/0?whereFID
599|Acuiferos_2014_600.
What's going wrong?
And yes, it's just the shapefile with a generic name, no biggie.
On Thu, Aug 11, 2016 at 1:59 PM, Joshua Tanner notifications@github.com
wrote:
Looking at the service, there only seems to be 653 features total. Are you
only getting 600 in your final dataset? You can also incorporate your own
query parameters in your url, referenced in #9
#9. I'll look into the
name designation. It's been naming my json and geojson correctly, but
packages up the shapefile with a generic name.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#15 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUAU9JGGjO6lnBVrztL0usU7UhoHwiV2ks5qe38-gaJpZM4JidAa
.
FYI, the > sign didn't copy right for "whereFID>599"
Any other advice to get the last 63 features?
I have been missing features from larger datasets, its been a consistent 100 for the first 5000 features queried. I've been using services with no FID, rather just OBJECTID, and found that OBJECTID 91-99 ("?where=OBJECTID>90 and OBJECTID<100") and 910-999 were almost always missing.
Check your output shp file table for missing FID ranges.
I copy the column of interest (OBJECTID for me) from the DBF into a blank excel document and use a quick vlookup to find missing features.
Copy all the current job numbers into column A of a worksheet and sort it, ascending.
On a second worksheet, create a list of all possible numbers:
Enter the text JOB in A1 and the text FOUND in B1
in A2 enter the lowest possible value, 7500
in A3 enter A2 + 1
Copy A3 down over however many cells you need to include all the possible values, (in this case A2002, i.e. 7500 through 9500 is 2,001 values plus the header in Row 1 gives 2,002).
Now in column B, enter the formula
=VLOOKUP(A1,Sheet1!$A$2:Sheet1!$A$2002,2,FALSE)
on the first row that contains a job number.
I hope this helped.
Can you pull the latest commit and test? I've made a couple changes that should hopefully solve your issues. I tested with your dataset and got all 653 features.
Thanks so much tannerjt, it works perfectly now!!! You have saved me so much work!
P.S. mdusch, you were right to check, it wasn't just the last OBJECTIDs that were missing, it was some here and there...