Easy Way to Bypass REST API Query Limits

Record Extract Limit

Have you ever encountered a limitation of 1000 or 2000 features when you extracted data from REST service endpoint, either through ArcGIS Online, Server or Portal?

ArcGIS Server map services limit the number of records that can be returned by a query to 1,000 or 2000 by default to provide optimal performance from the ArcGIS server to the client. The max record count determines the maximum number of features that can be returned in a single request. Specifying a large number of records to be returned by the server can slow the performance of client applications consuming your map service, such as web browsers, and your GIS server.

There are some available ArcGIS REST Service export online tools, Github repositories, and useful scripts about getting all datasets if there is server query limitation. However, sometimes it is not very convenient to use these tools in your script. Today I would like to share an easy trick to get a whole dataset with a single query. The following script loops through an ArcGIS Server REST API query in order to retrieve all features contained within the layer. It can be tested without importing arcpy library that is required for geoprocessing in ArcGIS software.

Let’s inspect the downloaded Geopandas DataFrame.

The above script only works for geospatial data with geojson or json format. If you want to use it with different data formats (or with non-geospatial data), it needs to be updated based on the various data output format. Please note that sometimes you need to set up proxy or user name and password parameters to get REST service endpoint data. The notebook can be accessed on Google Colab.

I hope you find the above script useful. Please don’t hesitate to write comments and questions.

References:

  1. Server configuration details — ArcGIS Enterprise Sites | Documentation for ArcGIS Enterprise
  2. https://help.arcgis.com/en/arcims/10.0/mainhelp/topics/security_featurelimits.html
  3. ArcGIS REST Service Export (cartong.github.io)
  4. Extracting MORE Features from Map Services | SoCalGIS.org

Senior Geospatial Specialist in Toronto

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store