Have you ever encountered a limitation of 1000 or 2000 features when you extracted data from REST service endpoint, either through ArcGIS Online, Server or Portal?
import urllib.request, urllib.error, urllib.parse
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt
import io, sys
import json, requests
from json import loads
from shapely.geometry import Point, Polygon
import numpy as np
from shapely.geometry import shape
from pandas.io.json import json_normalize
from collections import OrderedDict
# remove warnings
# set url
baseURL = "https://services.arcgis.com/V6ZHFr6zdgNZuVG0/ArcGIS/rest/services/Alternative_Fuel_Stations/FeatureServer/0"
fields = "*"
# Get record extract limit
urlstring = baseURL + "?f=json"
j = urllib.request.urlopen(urlstring)
js = json.load(j)
maxrcn = int(js["maxRecordCount"])
print(("Record extract limit: %s" % maxrcn))
ArcGIS Server map services limit the number of records that can be returned by a query to 1,000 or 2000 by default to provide optimal performance from the ArcGIS server to the client. The max record count determines the maximum number of features that can be returned in a single request. Specifying a large number of records to be returned by the server can slow the performance of client applications consuming your map service, such as web browsers, and your GIS server.
There are some available ArcGIS REST Service export online tools, Github repositories, and useful scripts about getting all datasets if there is server query limitation. However, sometimes it is not very convenient to use these tools in your script. Today I would like to share an easy trick to get a whole dataset with a single query. The following script loops through an ArcGIS Server REST API query in order to retrieve all features contained within the layer. It can be tested without importing arcpy library that is required for geoprocessing in ArcGIS software.
Let’s inspect the downloaded Geopandas DataFrame.
The above script only works for geospatial data with geojson or json format. If you want to use it with different data formats (or with non-geospatial data), it needs to be updated based on the various data output format. Please note that sometimes you need to set up proxy or user name and password parameters to get REST service endpoint data. The notebook can be accessed on Google Colab.
I hope you find the above script useful. Please don’t hesitate to write comments and questions.