Reading and writing spatial data
Session 11
2024-09-05
1 Spatial data is everywhere
- Shapefiles, GeoJSON files, and other spatial data files
- Spatial data editing and hosting services, e.g. ArcGIS Online or Felt)
- Open data portals serving spatial and non-spatial data, e.g. data.gov
2 Spatial data is everywhere
- Google Maps, Apple Maps, Waze, and navigation apps (on your phone or in your vehicle)
- Biking and running apps (Strava publishes a global user activity heatmap)
- Web browsers on your phone and desktop application
3 Spatial data is everywhere
- Phone camera photos (stored in the EXIF metadata)
- Calendars (iCal), contacts (vCard), and emails (using IP address locations)
4 Creating spatial data is “easy”
- GIS Desktop applications, e.g. QGIS
- Web mapping applications (e.g. geojson.io, Felt)
- Spreadsheet applications (e.g. Google Sheets)
- Database builders (e.g. Airtable or Microsoft Lists)
5 What can we do with it all?
- Read and write spatial data files
- Read and write related data files
- Read from and write to web services
6 Common spatial file formats
- Shapefile (
.shp) - GeoJSON (
.json,.geojson) - GeoPackage (
.gpkg) - KML - Keyhole Markup Language (
.kml) - Esri File Geodatabase (
.gdb)
7 Common spatial file formats
These file formats vary in:
- Types of geometry that they support
- Constraints or limits (e.g. shapefiles limit the lenghth of column names, GeoJSON can only use one coordinate reference system)
- File size
- Speed of input or output
- Compatibility with different tools or systems
- And more!
8 Read spatial data files with R
The dsn or data source name for sf::read_sf() can include:
- a file name
- a URL
- a folder
- the name and access credentials of a database
- a character string with the contents of a GeoJSON file
9 Read spatial data files with R
You can use the query parameter of sf::read_sf() to use a SQL query to select records from a file (or select a specific geographical extent).
10 Many file formats can include spatial data
- Excel files (
xlsxorxls)
- vCard or VCF (Virtual Contact File)
- EXIF metadata in a JPEG, HEIC, PNG, TIFF or other media file
11 Reading data into R
- Read data using
{sf} - Read data using a package that returns a
sfobject - Read data and use
{sf}to convert to asfobject
12 Writing data from R
- Write data to a spatial file using
{sf} - Write data a a non-spatial file using a different package
When you write spatial data to a non-spatial format, you should drop the geometry or convert the geometry to coordinates or well-known text.
13 Read data using {sf}
First load {sf}:
14 Read data using {sf}
15 Read data using {sf}
16 Read data using {sf}
{sf} can use any available drivers to read data:
name
ESRIC ESRIC
PCIDSK PCIDSK
netCDF netCDF
PDS4 PDS4
VICAR VICAR
JP2OpenJPEG JP2OpenJPEG
PDF PDF
MBTiles MBTiles
BAG BAG
EEDA EEDA
OGCAPI OGCAPI
ESRI Shapefile ESRI Shapefile
MapInfo File MapInfo File
UK .NTF UK .NTF
LVBAG LVBAG
OGR_SDTS OGR_SDTS
S57 S57
DGN DGN
OGR_VRT OGR_VRT
Memory Memory
CSV CSV
GML GML
GPX GPX
KML KML
GeoJSON GeoJSON
GeoJSONSeq GeoJSONSeq
ESRIJSON ESRIJSON
TopoJSON TopoJSON
OGR_GMT OGR_GMT
GPKG GPKG
SQLite SQLite
ODBC ODBC
WAsP WAsP
PGeo PGeo
MSSQLSpatial MSSQLSpatial
PostgreSQL PostgreSQL
OpenFileGDB OpenFileGDB
DXF DXF
CAD CAD
FlatGeobuf FlatGeobuf
Geoconcept Geoconcept
GeoRSS GeoRSS
VFK VFK
PGDUMP PGDUMP
OSM OSM
GPSBabel GPSBabel
OGR_PDS OGR_PDS
WFS WFS
OAPIF OAPIF
EDIGEO EDIGEO
SVG SVG
Idrisi Idrisi
XLS XLS
ODS ODS
XLSX XLSX
Elasticsearch Elasticsearch
Carto Carto
AmigoCloud AmigoCloud
SXF SXF
Selafin Selafin
JML JML
PLSCENES PLSCENES
CSW CSW
VDV VDV
MVT MVT
NGW NGW
MapML MapML
TIGER TIGER
AVCBin AVCBin
AVCE00 AVCE00
HTTP HTTP
long_name write copy
ESRIC Esri Compact Cache FALSE FALSE
PCIDSK PCIDSK Database File TRUE FALSE
netCDF Network Common Data Format TRUE TRUE
PDS4 NASA Planetary Data System 4 TRUE TRUE
VICAR MIPL VICAR file TRUE TRUE
JP2OpenJPEG JPEG-2000 driver based on OpenJPEG library FALSE TRUE
PDF Geospatial PDF TRUE TRUE
MBTiles MBTiles TRUE TRUE
BAG Bathymetry Attributed Grid TRUE TRUE
EEDA Earth Engine Data API FALSE FALSE
OGCAPI OGCAPI FALSE FALSE
ESRI Shapefile ESRI Shapefile TRUE FALSE
MapInfo File MapInfo File TRUE FALSE
UK .NTF UK .NTF FALSE FALSE
LVBAG Kadaster LV BAG Extract 2.0 FALSE FALSE
OGR_SDTS SDTS FALSE FALSE
S57 IHO S-57 (ENC) TRUE FALSE
DGN Microstation DGN TRUE FALSE
OGR_VRT VRT - Virtual Datasource FALSE FALSE
Memory Memory TRUE FALSE
CSV Comma Separated Value (.csv) TRUE FALSE
GML Geography Markup Language (GML) TRUE FALSE
GPX GPX TRUE FALSE
KML Keyhole Markup Language (KML) TRUE FALSE
GeoJSON GeoJSON TRUE FALSE
GeoJSONSeq GeoJSON Sequence TRUE FALSE
ESRIJSON ESRIJSON FALSE FALSE
TopoJSON TopoJSON FALSE FALSE
OGR_GMT GMT ASCII Vectors (.gmt) TRUE FALSE
GPKG GeoPackage TRUE TRUE
SQLite SQLite / Spatialite TRUE FALSE
ODBC FALSE FALSE
WAsP WAsP .map format TRUE FALSE
PGeo ESRI Personal GeoDatabase FALSE FALSE
MSSQLSpatial Microsoft SQL Server Spatial Database TRUE FALSE
PostgreSQL PostgreSQL/PostGIS TRUE FALSE
OpenFileGDB ESRI FileGDB FALSE FALSE
DXF AutoCAD DXF TRUE FALSE
CAD AutoCAD Driver FALSE FALSE
FlatGeobuf FlatGeobuf TRUE FALSE
Geoconcept Geoconcept TRUE FALSE
GeoRSS GeoRSS TRUE FALSE
VFK Czech Cadastral Exchange Data Format FALSE FALSE
PGDUMP PostgreSQL SQL dump TRUE FALSE
OSM OpenStreetMap XML and PBF FALSE FALSE
GPSBabel GPSBabel TRUE FALSE
OGR_PDS Planetary Data Systems TABLE FALSE FALSE
WFS OGC WFS (Web Feature Service) FALSE FALSE
OAPIF OGC API - Features FALSE FALSE
EDIGEO French EDIGEO exchange format FALSE FALSE
SVG Scalable Vector Graphics FALSE FALSE
Idrisi Idrisi Vector (.vct) FALSE FALSE
XLS MS Excel format FALSE FALSE
ODS Open Document/ LibreOffice / OpenOffice Spreadsheet TRUE FALSE
XLSX MS Office Open XML spreadsheet TRUE FALSE
Elasticsearch Elastic Search TRUE FALSE
Carto Carto TRUE FALSE
AmigoCloud AmigoCloud TRUE FALSE
SXF Storage and eXchange Format FALSE FALSE
Selafin Selafin TRUE FALSE
JML OpenJUMP JML TRUE FALSE
PLSCENES Planet Labs Scenes API FALSE FALSE
CSW OGC CSW (Catalog Service for the Web) FALSE FALSE
VDV VDV-451/VDV-452/INTREST Data Format TRUE FALSE
MVT Mapbox Vector Tiles TRUE FALSE
NGW NextGIS Web TRUE TRUE
MapML MapML TRUE FALSE
TIGER U.S. Census TIGER/Line FALSE FALSE
AVCBin Arc/Info Binary Coverage FALSE FALSE
AVCE00 Arc/Info E00 (ASCII) Coverage FALSE FALSE
HTTP HTTP Fetching Wrapper FALSE FALSE
is_raster is_vector vsi
ESRIC TRUE TRUE TRUE
PCIDSK TRUE TRUE TRUE
netCDF TRUE TRUE FALSE
PDS4 TRUE TRUE TRUE
VICAR TRUE TRUE TRUE
JP2OpenJPEG TRUE TRUE TRUE
PDF TRUE TRUE FALSE
MBTiles TRUE TRUE TRUE
BAG TRUE TRUE TRUE
EEDA FALSE TRUE FALSE
OGCAPI TRUE TRUE TRUE
ESRI Shapefile FALSE TRUE TRUE
MapInfo File FALSE TRUE TRUE
UK .NTF FALSE TRUE TRUE
LVBAG FALSE TRUE TRUE
OGR_SDTS FALSE TRUE TRUE
S57 FALSE TRUE TRUE
DGN FALSE TRUE TRUE
OGR_VRT FALSE TRUE TRUE
Memory FALSE TRUE FALSE
CSV FALSE TRUE TRUE
GML FALSE TRUE TRUE
GPX FALSE TRUE TRUE
KML FALSE TRUE TRUE
GeoJSON FALSE TRUE TRUE
GeoJSONSeq FALSE TRUE TRUE
ESRIJSON FALSE TRUE TRUE
TopoJSON FALSE TRUE TRUE
OGR_GMT FALSE TRUE TRUE
GPKG TRUE TRUE TRUE
SQLite FALSE TRUE TRUE
ODBC FALSE TRUE FALSE
WAsP FALSE TRUE TRUE
PGeo FALSE TRUE FALSE
MSSQLSpatial FALSE TRUE FALSE
PostgreSQL FALSE TRUE FALSE
OpenFileGDB FALSE TRUE TRUE
DXF FALSE TRUE TRUE
CAD TRUE TRUE TRUE
FlatGeobuf FALSE TRUE TRUE
Geoconcept FALSE TRUE TRUE
GeoRSS FALSE TRUE TRUE
VFK FALSE TRUE FALSE
PGDUMP FALSE TRUE TRUE
OSM FALSE TRUE TRUE
GPSBabel FALSE TRUE FALSE
OGR_PDS FALSE TRUE TRUE
WFS FALSE TRUE TRUE
OAPIF FALSE TRUE FALSE
EDIGEO FALSE TRUE TRUE
SVG FALSE TRUE TRUE
Idrisi FALSE TRUE TRUE
XLS FALSE TRUE FALSE
ODS FALSE TRUE TRUE
XLSX FALSE TRUE TRUE
Elasticsearch FALSE TRUE FALSE
Carto FALSE TRUE FALSE
AmigoCloud FALSE TRUE FALSE
SXF FALSE TRUE TRUE
Selafin FALSE TRUE TRUE
JML FALSE TRUE TRUE
PLSCENES TRUE TRUE FALSE
CSW FALSE TRUE FALSE
VDV FALSE TRUE TRUE
MVT FALSE TRUE TRUE
NGW TRUE TRUE FALSE
MapML FALSE TRUE TRUE
TIGER FALSE TRUE TRUE
AVCBin FALSE TRUE TRUE
AVCE00 FALSE TRUE TRUE
HTTP TRUE TRUE FALSE17 Read data from a URL using {sf}
read_sf() works well with a direct link to a spatial data file:
# GeoJSON (geojson)
read_sf("https://github.com/ropensci/geojsonio/raw/main/inst/examples/california.geojson")Simple feature collection with 1 feature and 11 fields
Geometry type: MULTIPOLYGON
Dimension: XY
Bounding box: xmin: -124.4096 ymin: 32.53416 xmax: -114.1315 ymax: 42.00952
Geodetic CRS: WGS 84
# A tibble: 1 × 12
name abbreviation capital city population area waterarea landarea
<chr> <chr> <chr> <chr> <int> <int> <int> <int>
1 California CA Sacramento Los A… 38332521 423968 20502 403466
# ℹ 4 more variables: houseseats <int>, statehood <date>, group <chr>,
# geometry <MULTIPOLYGON [°]># Keyhole Markup Language (kml)
read_sf("https://github.com/ropensci/geojsonio/raw/main/inst/examples/poa_annua.kml")Simple feature collection with 161 features and 2 fields
Geometry type: POINT
Dimension: XY
Bounding box: xmin: -167.15 ymin: 0 xmax: 0 ymax: 69.3333
Geodetic CRS: WGS 84
# A tibble: 161 × 3
Name Description geometry
<chr> <chr> <POINT [°]>
1 "Patterson's bluegrass, Patterson blue… "<table wi… (-113.9553 43.39448)
2 "Patterson's bluegrass, Patterson blue… "<table wi… (-114.1191 46.0793)
3 "Patterson's bluegrass, Patterson blue… "<table wi… (-114.1191 46.0793)
4 "Patterson's bluegrass, Patterson blue… "<table wi… (-114.1191 46.0793)
5 "Patterson's bluegrass, Patterson blue… "<table wi… (-109.5936 44.49239)
6 "Patterson's bluegrass, Patterson blue… "<table wi… (-114.1191 46.0793)
7 "Patterson's bluegrass, Patterson blue… "<table wi… (-109.817 40.7667)
8 "Patterson's bluegrass, Patterson blue… "<table wi… (-105 39)
9 "Patterson's bluegrass, Patterson blue… "<table wi… (-106.1376 39.62102)
10 "Patterson's bluegrass, Patterson blue… "<table wi… (-111.183 40.6167)
# ℹ 151 more rowsSimple feature collection with 100 features and 14 fields
Geometry type: MULTIPOLYGON
Dimension: XY
Bounding box: xmin: -84.32385 ymin: 33.88199 xmax: -75.45698 ymax: 36.58965
Geodetic CRS: NAD27
# A tibble: 100 × 15
AREA PERIMETER CNTY_ CNTY_ID NAME FIPS FIPSNO CRESS_ID BIR74 SID74 NWBIR74
<dbl> <dbl> <dbl> <dbl> <chr> <chr> <dbl> <int> <dbl> <dbl> <dbl>
1 0.114 1.44 1825 1825 Ashe 37009 37009 5 1091 1 10
2 0.061 1.23 1827 1827 Alle… 37005 37005 3 487 0 10
3 0.143 1.63 1828 1828 Surry 37171 37171 86 3188 5 208
4 0.07 2.97 1831 1831 Curr… 37053 37053 27 508 1 123
5 0.153 2.21 1832 1832 Nort… 37131 37131 66 1421 9 1066
6 0.097 1.67 1833 1833 Hert… 37091 37091 46 1452 7 954
7 0.062 1.55 1834 1834 Camd… 37029 37029 15 286 0 115
8 0.091 1.28 1835 1835 Gates 37073 37073 37 420 0 254
9 0.118 1.42 1836 1836 Warr… 37185 37185 93 968 4 748
10 0.124 1.43 1837 1837 Stok… 37169 37169 85 1612 1 160
# ℹ 90 more rows
# ℹ 4 more variables: BIR79 <dbl>, SID79 <dbl>, NWBIR79 <dbl>,
# geom <MULTIPOLYGON [°]>Even non-spatial data works if the data includes a well-known text column:
Simple feature collection with 3 features and 4 fields
Geometry type: POINT
Dimension: XY
Bounding box: xmin: 0 ymin: 0 xmax: 44 ymax: 55
CRS: NA
# A tibble: 3 × 5
Id Int64 Int32 WKT geometry
<chr> <chr> <chr> <chr> <POINT>
1 1 "0" "33" POINT(0 1) (0 1)
2 2 "4294967297" "44" POINT(44 55) (44 55)
3 3 "" "" POINT(0 0) (0 0)If the non-spatial data does not include a well-known text column, read_sf() still returns a data frame:
read_sf("https://github.com/elipousson/marylandedu/raw/main/data-raw/extdata/Enrollment_By_Grade_2023.csv")# A tibble: 9,766 × 8
Year LEA `LEA Name` School `School Name` Grade `Enrolled Count`
<chr> <chr> <chr> <chr> <chr> <chr> <chr>
1 2023 01 Allegany 0301 Flintstone Elementary Prekind… 19
2 2023 01 Allegany 0301 Flintstone Elementary Kinderg… 42
3 2023 01 Allegany 0301 Flintstone Elementary Grade 1 33
4 2023 01 Allegany 0301 Flintstone Elementary Grade 2 29
5 2023 01 Allegany 0301 Flintstone Elementary Grade 3 26
6 2023 01 Allegany 0301 Flintstone Elementary Grade 4 31
7 2023 01 Allegany 0301 Flintstone Elementary Grade 5 36
8 2023 01 Allegany 0301 Flintstone Elementary All Gra… 216
9 2023 01 Allegany 0401 South Penn Elementary Prekind… 77
10 2023 01 Allegany 0401 South Penn Elementary Kinderg… 79
# ℹ 9,756 more rows
# ℹ 1 more variable: `Create Date` <chr>However, multifile format like a shapefile does not work as a URL:
Error: Cannot open "https://github.com/r-spatial/sf/raw/main/inst/shape/nc.shp"; The file doesn't seem to exist.But it does work if you provide a local file path (system.file() returns a file path in this case):
Simple feature collection with 100 features and 14 fields
Geometry type: MULTIPOLYGON
Dimension: XY
Bounding box: xmin: -84.32385 ymin: 33.88199 xmax: -75.45698 ymax: 36.58965
Geodetic CRS: NAD27
# A tibble: 100 × 15
AREA PERIMETER CNTY_ CNTY_ID NAME FIPS FIPSNO CRESS_ID BIR74 SID74 NWBIR74
<dbl> <dbl> <dbl> <dbl> <chr> <chr> <dbl> <int> <dbl> <dbl> <dbl>
1 0.114 1.44 1825 1825 Ashe 37009 37009 5 1091 1 10
2 0.061 1.23 1827 1827 Alle… 37005 37005 3 487 0 10
3 0.143 1.63 1828 1828 Surry 37171 37171 86 3188 5 208
4 0.07 2.97 1831 1831 Curr… 37053 37053 27 508 1 123
5 0.153 2.21 1832 1832 Nort… 37131 37131 66 1421 9 1066
6 0.097 1.67 1833 1833 Hert… 37091 37091 46 1452 7 954
7 0.062 1.55 1834 1834 Camd… 37029 37029 15 286 0 115
8 0.091 1.28 1835 1835 Gates 37073 37073 37 420 0 254
9 0.118 1.42 1836 1836 Warr… 37185 37185 93 968 4 748
10 0.124 1.43 1837 1837 Stok… 37169 37169 85 1612 1 160
# ℹ 90 more rows
# ℹ 4 more variables: BIR79 <dbl>, SID79 <dbl>, NWBIR79 <dbl>,
# geometry <MULTIPOLYGON [°]>An ArcGIS FeatureServer URL does not work:
read_sf("https://services.arcgis.com/njFNhDsUCentVYJW/arcgis/rest/services/MDOT_SHA_County_Boundaries/FeatureServer/0")Error: Cannot open "https://services.arcgis.com/njFNhDsUCentVYJW/arcgis/rest/services/MDOT_SHA_County_Boundaries/FeatureServer/0"; The file doesn't seem to exist.But a FeatureServer URL that includes the output format does work:
read_sf("https://services.arcgis.com/njFNhDsUCentVYJW/arcgis/rest/services/MDOT_SHA_County_Boundaries/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson")Simple feature collection with 24 features and 17 fields
Geometry type: POLYGON
Dimension: XY
Bounding box: xmin: -79.48731 ymin: 37.88654 xmax: -75.0492 ymax: 39.72288
Geodetic CRS: WGS 84
# A tibble: 24 × 18
OBJECTID COUNTY_NAME COUNTY_INTEGER COUNTY_ABBREVIATION COUNTY_DIGITCAPS
<int> <chr> <int> <chr> <int>
1 1 Prince George's 16 PG 16
2 2 Cecil 7 CE 7
3 3 Howard 13 HO 13
4 4 Talbot 20 TA 20
5 5 Garrett 11 GA 11
6 6 Baltimore City 24 BC 24
7 7 Carroll 6 CL 6
8 8 Washington 21 WA 21
9 9 Worcester 23 WO 23
10 10 Frederick 10 FR 10
# ℹ 14 more rows
# ℹ 13 more variables: COUNTY_2DIGITCAPS <chr>, DISTRICT_NAME <chr>,
# DISTRICT_NUMBER <int>, COUNTY_FIP <int>, COUNTYNUM <int>, TSD_ID <int>,
# COUNTY <chr>, DISTRICT <int>, Shape__Area <dbl>, Shape__Length <dbl>,
# COUNTY_ALLCAPS <chr>, Region <chr>, geometry <POLYGON [°]>18 Read spatial data with other packages
There are a wide variety of packages that you can use to supplement {sf}.
library(arcgislayers)
feature_layer <- arc_open("https://services.arcgis.com/njFNhDsUCentVYJW/arcgis/rest/services/MDOT_SHA_County_Boundaries/FeatureServer/0")
arc_select(feature_layer)Simple feature collection with 24 features and 17 fields
Geometry type: POLYGON
Dimension: XY
Bounding box: xmin: 607669.3 ymin: 80997.33 xmax: 1870976 ymax: 757701.4
Projected CRS: NAD83(HARN) / Maryland (ftUS)
First 10 features:
OBJECTID COUNTY_NAME COUNTY_INTEGER COUNTY_ABBREVIATION COUNTY_DIGITCAPS
1 1 Prince George's 16 PG 16
2 2 Cecil 7 CE 7
3 3 Howard 13 HO 13
4 4 Talbot 20 TA 20
5 5 Garrett 11 GA 11
6 6 Baltimore City 24 BC 24
7 7 Carroll 6 CL 6
8 8 Washington 21 WA 21
9 9 Worcester 23 WO 23
10 10 Frederick 10 FR 10
COUNTY_2DIGITCAPS DISTRICT_NAME DISTRICT_NUMBER COUNTY_FIP COUNTYNUM TSD_ID
1 16 District 3 3 33 16 3
2 07 District 2 2 15 7 21
3 13 District 7 7 27 13 10
4 20 District 2 2 41 20 18
5 11 District 6 6 23 11 4
6 24 No District 0 510 24 23
7 06 District 7 7 13 6 1
8 21 District 6 6 43 21 1060
9 23 District 1 1 47 23 2
10 10 District 7 7 21 10 24
COUNTY DISTRICT Shape__Area Shape__Length COUNTY_ALLCAPS Region
1 Prince George's 3 13902243040 720691.0 PRINCE GEORGE'S <NA>
2 Cecil 2 11643675836 494670.7 CECIL <NA>
3 Howard 7 7056288549 471922.1 HOWARD <NA>
4 Talbot 2 13193323636 657963.0 TALBOT <NA>
5 Garrett 6 18339518744 680782.9 GARRETT <NA>
6 Baltimore City 0 2562810672 207145.0 BALTIMORE CITY <NA>
7 Carroll 7 12623420879 651107.6 CARROLL <NA>
8 Washington 6 13031238277 948164.4 WASHINGTON <NA>
9 Worcester 1 16297527948 694972.5 WORCESTER <NA>
10 Frederick 7 18613257293 683407.8 FREDERICK <NA>
geometry
1 POLYGON ((1313442 476110.4,...
2 POLYGON ((1660741 628728.5,...
3 POLYGON ((1391417 567919.8,...
4 POLYGON ((1611348 457521.9,...
5 POLYGON ((769889 754580.7, ...
6 POLYGON ((1445534 565803.6,...
7 POLYGON ((1371781 748182.4,...
8 POLYGON ((936016.4 751530.6...
9 POLYGON ((1870321 284172.9,...
10 POLYGON ((1213130 747989.9,...19 Examples
::: notes
KML
Source: https://developers.google.com/kml/documentation/kmlreference :::
