ERDDAP
Easier access to scientific data |
Brought to you by NOAA NMFS SWFSC ERD |
The URL specifies what you want: the dataset, a description of the graph or the subset of the data, and the file type for the response.
griddap request URLs must be in the form
https://coastwatch.pfeg.noaa.gov/erddap/griddap/datasetID.fileType{?query}
For example,
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.htmlTable?analysed_sst[(2002-06-01T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]
Thus, the query is often a data variable name (e.g., analysed_sst),
followed by [(start):stride:(stop)]
(or a shorter variation of that) for each of the variable's dimensions
(for example, [time][latitude][longitude]).
Details:
Which fileType should I use?
It's entirely up to you. In general, pick the fileType which meets your needs and is easiest to use. You will
use different fileTypes in different situations, e.g., viewing a graph in a browser (.png) vs. viewing the data
in a browser (.htmlTable) vs. downloading a data file (e.g., .nc or .csv) vs. working with some software tool
(e.g., .nc or .odv). If you are going to download lots of large files, you might also want to give some weight
to fileTypes that are more compact and so can be downloaded faster.
The fileType options for downloading gridded data are:
Data fileTypes | Description | Info | Example |
---|---|---|---|
.asc | View OPeNDAP-style ISO-8859-1 comma-separated text. | info | example |
.csv | Download a ISO-8859-1 comma-separated text table (line 1: names; line 2: units; ISO 8601 times). | info | example |
.csvp | Download a ISO-8859-1 .csv file with line 1: name (units). Times are ISO 8601 strings. | info | example |
.csv0 | Download a ISO-8859-1 .csv file without column names or units. Times are ISO 8601 strings. | info | example |
.das | View the dataset's metadata via an ISO-8859-1 OPeNDAP Dataset Attribute Structure (DAS). | info | example |
.dds | View the dataset's structure via an ISO-8859-1 OPeNDAP Dataset Descriptor Structure (DDS). | info | example |
.dods | OPeNDAP clients use this to download the data in the DODS binary format. | info | example |
.esriAscii | Download an ISO-8859-1 ESRI ASCII file (lat lon data only; lon must be all below or all above 180). | info | example |
.fgdc | View the dataset's UTF-8 FGDC .xml metadata. | info | example |
.graph | View a Make A Graph web page. | info | example |
.help | View a web page with a description of griddap. | info | example |
.html | View an OPeNDAP-style HTML Data Access Form. | info | example |
.htmlTable | View a UTF-8 .html web page with the data in a table. Times are ISO 8601 strings. | info | example |
.iso19115 | View the dataset's ISO 19115-2/19139 UTF-8 .xml metadata. | info | example |
.itx | Download an ISO-8859-1 Igor Text File. Each axis variable and each data variable becomes a wave. | info | example |
.json | View a table-like UTF-8 JSON file (missing value = 'null'; times are ISO 8601 strings). | info | example |
.jsonlCSV1 | View a UTF-8 JSON Lines CSV file with column names on line 1 (mv = 'null'; times are ISO 8601 strings). | info | example |
.jsonlCSV | View a UTF-8 JSON Lines CSV file without column names (mv = 'null'; times are ISO 8601 strings). | info | example |
.jsonlKVP | View a UTF-8 JSON Lines file with Key:Value pairs (missing value = 'null'; times are ISO 8601 strings). | info | example |
.mat | Download a MATLAB binary file. | info | example |
.nc | Download a NetCDF-3 binary file with COARDS/CF/ACDD metadata. | info | example |
.ncHeader | View the UTF-8 header (the metadata) for the NetCDF-3 .nc file. | info | example |
.ncml | View the dataset's structure and metadata as a UTF-8 NCML .xml file. | info | example |
.nccsv | Download a NetCDF-3-like 7-bit ASCII NCCSV .csv file with COARDS/CF/ACDD metadata. | info | example |
.nccsvMetadata | View the dataset's metadata as the top half of a 7-bit ASCII NCCSV .csv file. | info | example |
.ncoJson | Download a UTF-8 NCO lvl=2 JSON file with COARDS/CF/ACDD metadata. | info | example |
.odvTxt | Download time,lat,lon,otherVariables as an ODV Generic Spreadsheet File (.txt). | info | example |
.timeGaps | View a UTF-8 list of gaps in the time values which are larger than the median gap. | info | example |
.tsv | Download a ISO-8859-1 tab-separated text table (line 1: names; line 2: units; ISO 8601 times). | info | example |
.tsvp | Download a ISO-8859-1 .tsv file with line 1: name (units). Times are ISO 8601 strings. | info | example |
.tsv0 | Download a ISO-8859-1 .tsv file without column names or units. Times are ISO 8601 strings. | info | example |
.wav | Download a .wav audio file. All columns must be numeric and of the same type. | info | example |
.xhtml | View a UTF-8 XHTML (XML) file with the data in a table. Times are ISO 8601 strings. | info | example |
ArcGIS
.esriAsc
.esriAsc is an old and inherently limited file format. If you have ArcGIS 10 or higher, we strongly recommend
that you download gridded data from ERDDAP in a NetCDF .nc file, which can be opened directly by ArcGIS 10+
using the
Make
NetCDF Raster Layer tool in the Multidimension Tools toolbox.
If you have ArcGIS 9.x or lower:
ArcGIS is a family of Geographical Information Systems (GIS) products from ESRI: ArcView, ArcEditor, and ArcInfo.
To download data for use with ArcGIS 9.x or lower:
in ERDDAP, choose the .esriAscii file type to save a latitude longitude subset of data for just one point in time.
The file's extension will be .asc. This file format was designed by ESRI to transfer coverage data between computers.
Then, import the data file into your ArcGIS program:
Shapefiles - Sorry, ERDDAP currently does not distribute grid data as shapefiles.
Ferret
is a free program for visualizing and analyzing large and complex gridded
datasets. Ferret should work well with all datasets in griddap since griddap is
fully compatible with OPeNDAP. See the
Ferret documentation.
Note that the griddap dataset's OPeNDAP base URL that you use with Ferret's
set data, for example,
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41
won't ever have a file extension at the end.
IDL - IDL is a commercial scientific data visualization program. To get data from ERDDAP into IDL, first use ERDDAP to select a subset of data and download a .nc file. Then, use these instructions to import the data from the .nc file into IDL.
JSON .json files are widely used to transfer data to JavaScript scripts running on web pages. All .json responses from ERDDAP (metadata, gridded data, and tabular/in-situ data) use the same basic format: a database-like table. For data from grid datasets, ERDDAP flattens the data into a table with a column for each dimension and a column for each data variable. For example,
{ "table": { "columnNames": ["time", "latitude", "longitude", "analysed_sst"], "columnTypes": ["String", "float", "float", "double"], "columnUnits": ["UTC", "degrees_north", "degrees_east", "degree_C"], "rows": [ ["2014-02-03T09:00:00Z", 34.9969, -134.995, 16.037], ["2014-02-03T09:00:00Z", 34.9969, -134.984, 16.033], ["2014-02-03T09:00:00Z", 34.9969, -134.973, null], ... ["2014-02-03T09:00:00Z", 36.9965, -132.995, 15.285] ] } }All .json responses from ERDDAP have
JSONP (from .json) - Jsonp is an easy way for a JavaScript script on a web page to import and access data from ERDDAP. Requests for .geoJson, .json, and .ncoJson files may include an optional jsonp request by adding &.jsonp=functionName to the end of the query. Basically, this just tells ERDDAP to add functionName( to the beginning of the response and ")" to the end of the response. The first character of functionName must be an ISO 8859 letter or "_". Each optional subsequent character must be an ISO 8859 letter, "_", a digit, or ".". If originally there was no query, leave off the "&" in your query. After the data download to the web page has finished, the data is accessible to the JavaScript script via that JavaScript function. Here is an example using jsonp with ERDDAP and jQuery (thanks to Jenn Patterson Sevadjian, of CenCOOS/now PolarWatch).
MATLAB .mat users can use griddap's .mat file type to download data from within MATLAB. Here is a one line example:
load(urlwrite('https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.mat?analysed_sst[(2002-06-01T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]', 'test.mat'));(You may need to percent encode the query part of the URL.) The data will be in a MATLAB structure. The structure's name will be the datasetID (for example, jplMURSST41). The structure's internal variables will have the same names as in ERDDAP, (for example, use fieldnames(jplMURSST41)). If you download a 2D matrix of data (as in the example above), you can plot it with (for example):
imagesc(jplMURSST41.longitude, jplMURSST41.latitude, squeeze(jplMURSST41.analysed_sst), [0 32]) set(gca, 'YDir', 'normal')The numbers at the end of the first line specify the range for the color mapping. The 'set' command flips the map to make it upright.
There are also Matlab Xtractomatic scripts for ERDDAP, which are particularly useful for getting environmental data related to points along an animal's track (e.g., GTOPP data).
ERDDAP stores datetime values in .mat files as "seconds since 1970-01-01T00:00:00Z".
To display one of these values as a String in Matlab, you can use, e.g.,
datastr(cwwcNDBCMet.time(1)/86400 + 719529)
86400 converts ERDDAP's "seconds since" to Matlab's "days since". 719529 converts
ERDDAP's base time of "1970-01-01T00:00:00Z" to Matlab's "0000-01-00T00:00:00Z".
.mat files have a maximum length for identifier names of 32 characters. If a variable name is longer than that, ERDDAP changes the name of the variable when writing the .mat file: it generates a hash digest of the variable name and appends that after the first 25 characters. Thus, long variable names that only differ at the end will still have unique names in the .mat file. ERDDAP administrators: you can avoid this problem by specifying shorter variable destinationNames.
NetCDF .nc - Requests for .nc files return the requested subset of the dataset in a standard, NetCDF-3, 32-bit, .nc file.
If you are using NetCDF-Java, don't try to directly access an ERDDAP dataset or subset as a .nc file. (It won't work, mostly because that .nc file isn't a static, persistent file. It is a virtual file.) Instead, use one of these two options:
NetcdfFile nc = NetcdfDataset.openFile("https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41", null);(don't use NetcdfFile.open; it is for local files only) or
NetcdfDataset nc = NetcdfDataset.openDataset("https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41");(NetcdfFiles are a lower level approach than NetcdfDatasets. It is your choice.)
NetcdfFile nc = NetcdfFile.open("c:\downloads\theDownloadedFile.nc");or
NetcdfDataset nc = NetcdfDataset.openDataset("c:\downloads\theDownloadedFile.nc");(NetcdfFiles are a lower level approach than NetcdfDatasets. It is your choice.)
In both cases, you can then do what you want with the nc object, for example, request metadata or request a subset of a variable's data.
.ncHeader - Requests for .ncHeader files will return the header information (text) that would be generated if you used ncdump -h fileName on the corresponding .nc file.
Ocean Data View .odvTxt - ODV users can download data in a ODV Generic Spreadsheet Format .txt file by requesting griddap's .odvTxt fileType. The dataset MUST include longitude, latitude, and time dimensions. Any longitude values (0 to 360, or -180 to 180) are fine. After saving the resulting file (with the extension .txt) in your computer:
OPeNDAP Libraries - Since ERDDAP is an OPeNDAP-compatible data server, you can use any OPeNDAP client library, such as NetCDF-Java, NetCDF-C, NetCDF-Fortran, NetCDF-Perl, Java-DAP2, Ferret, or the Pydap Client, to get data from an ERDDAP griddap dataset. When creating the initial connection to an ERDDAP griddap dataset from any OPeNDAP library:
For example, with the NetCDF-Java library, you can use:
NetcdfFile nc = NetcdfDataset.openFile("https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41", null);(don't use NetcdfFile.open; it is for local files only) or
NetcdfDataset nc = NetcdfDataset.openDataset("https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41");(NetcdfFiles are a lower level approach than NetcdfDatasets. It is your choice.)
Pydap Client
users
can access griddap datasets via ERDDAP's standard OPeNDAP services.
See the
Pydap Client instructions for accessing gridded data.
Note that the name of a dataset in ERDDAP will always be a single word,
(e.g., jplMURSST41 in the OPeNDAP dataset URL
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41 )
and won't ever have a file extension (unlike, for example, .nc for the
sample dataset in the Pydap instructions).
Python is a widely-used computer language that is very popular among scientists. In addition to the Pydap Client, you can use Python to download various files from ERDDAP as you would download other files from the web:
import urllib urllib.urlretrieve("https://baseHttpsUrl/erddap/griddap/datasetID.fileType?query", "outputFileName")(You may need to percent encode the query part of the URL.)
import urllib2 response = urllib2.open("https://baseHttpsUrl/erddap/griddap/datasetID.fileType?query") theContent = response.read()There are other ways to do this in Python. Search the web for more information.
erddapy (ERDDAP + Python) is a Python library that "takes advantage of ERDDAP’s RESTful web services and creates the ERDDAP URL for any request like searching for datasets, acquiring metadata, downloading data, etc."
"Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text" using any of over 40 programming languages, including Python and R. Here are two sample Jupyter Notebooks that access ERDDAP using Python: ERDDAP Advanced Search Test and ERDDAP Timing. Thanks to Rich Signell.
R Statistical Package - R is an open source statistical package for many operating systems. In R, you can download a NetCDF version 3 .nc file from ERDDAP. For example:
download.file(url="https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.nc?analysed_sst[(2002-06-01T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]", destfile="/home/bsimons/test.nc")(You may need to percent encode the query part of the URL.) Then import data from that .nc file into R with the RNetCDF, ncdf, or ncdf4 packages available from CRAN. Or, if you want the data in tabular form, download and import the data in a .csv file. For example,
download.file(url="https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.csv?analysed_sst[(2002-06-01T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]", destfile="/home/bsimons/test.csv") test<-read.csv(file="/home/bsimons/test.csv")There are third-party R packages designed to make it easier to work with ERDDAP from within R: rerddap, rerddapXtracto, and plotdap. Thanks to rOpenSci and Roy Mendelssohn.
There are also R Xtractomatic scripts for ERDDAP,
which are particularly useful for getting
environmental data related to points along an animal's track (e.g.,
GTOPP data).
.timeGaps is a response which is unique to ERDDAP. It lets you
view a list of gaps in a dataset's time values which are larger than the median gap.
This is useful for ERDDAP administrators and end users when they want
to know if there are unexpected gaps in the time values for a dataset
that is expected to have regularly spaced time values.
This option ignores all request constraints.
.wav - ERDDAP can return data in .wav files, which are uncompressed audio files.
Making an Image File with a Graph or Map of Gridded Data
If a griddap request URL specifies a subset of data which is suitable for making
a graph or a map, and the fileType is an image fileType, griddap will return an image
with a graph or map.
griddap request URLs can include optional graphics commands which let you
customize the graph or map.
As with other griddap request URLs, you can create these URLs by hand or have a
computer program do it. Or, you can use the Make A Graph web pages, which simplify
creating these URLs (see the "graph" links in the table of
griddap datasets).
The fileType options for downloading images of graphs and maps of grid data are:
Image fileTypes | Description | Info | Example |
---|---|---|---|
.geotif | View a grayscale GeoTIFF .tif file (for lat lon data; lon must be all below or all above 180). | info | example |
.kml | View a Google Earth .kml file (for lat, lon data only). | info | example |
.smallPdf | View a small .pdf image file with a graph or map. | info | example |
View a standard, medium-sized .pdf image file with a graph or map. | info | example | |
.largePdf | View a large .pdf image file with a graph or map. | info | example |
.smallPng | View a small .png image file with a graph or map. | info | example |
.png | View a standard, medium-sized .png image file with a graph or map. | info | example |
.largePng | View a large .png image file with a graph or map. | info | example |
.transparentPng | View a .png image file (just the data, without axes, landmask, or legend). | info | example |
Image Size - ".small" and ".large" were ERDDAP's original system for making different-sized images. Now, for .png and .transparentPng images (not other image file types), you can also use the &.size=width|height parameter to request an image of any size.
.transparentPng - The .transparentPng file type will make a graph or map without the graph axes, landmask, or legend, and with a transparent (not opaque white) background. This option can be used for any type of graph or map. When &.draw= is set to anything other than surface, the default image size is 360x360 pixels. Use the &.size=width|height parameter to specify a different size. When &.draw= is set to surface, ERDDAP makes an image where each data point becomes one pixel, which may result in a huge image. If the request takes too long or fails (perhaps for an odd apparent reason, like a Proxy Error) either use a stride (see below) value greater than 1 (e.g., 5) for the x and y axis variables, or use the &.size=width|height parameter to restrict the image size, so that the image isn't huge.
Incompatibilities
Some results file types have restrictions. For example, Google Earth .kml is only
appropriate for results with longitude and latitude values. If a given request is
incompatible with the requested file type, griddap throws an error.
Command Line Downloads with curl
If you want to download a series of files from ERDDAP, you don't have to request each file's
ERDDAP URL in your browser, sitting and waiting for each file to download.
If you are comfortable writing computer programs (e.g., with C, Java, Python, Matlab, r)
or scripts (e.g., Python, bash, tcsh, PowerShell, or Windows batch files),
you can write a program or script with a loop (or a series of commands)
that imports all of the desired data files.
Or, if you are comfortable running command line programs
(from a Linux or Windows command line, or a Mac OS Terminal), you can use curl (or a similar program like
wget)
to save results files from ERDDAP into files on your hard drive,
without using a browser or writing a computer program or script.
ERDDAP + curl is amazingly powerful and allows you to use ERDDAP in many new ways.
On Linux and Mac OS X, curl is probably already installed.
On Mac OS X, to get to a command line, use "Finder : Go : Utilities : Terminal".
On Windows, you need to
download curl
(the "Windows 64 - binary, the curl project" variant worked for me on Windows 10)
and install it.
On Windows, to get to a command line, click on "Start" and type
"cmd" into the search text field.
Please be kind to other ERDDAP users: run just one script or curl command at a time.
Instructions for using curl are on the
curl man page and in this
curl tutorial.
But here is a quick tutorial related to using curl with ERDDAP:
curl --compressed -g "https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.png?analysed_sst%5B(2002-06-01T09:00:00Z):(2002-07-01T09:00:00Z)%5D%5B(29)%5D%5B(-135)%5D&.draw=linesAndMarkers&.vars=time%7Canalysed_sst%7C&.marker=1%7C3&.color=0xFF9900&.colorBar=%7CC%7CLinear%7C%7C%7C" -o jplMURSST41_example.png(That example includes --compressed because it is a generally useful option, but there is little benefit to --compressed when requesting .png files because they are already compressed.)
In curl, as in many other programs, the query part of the erddapUrl must be
percent encoded. To do this, you need to convert
special characters (other than the initial '&' and the main '=' of a constraint)
into the form %HH, where HH is the 2 digit hexadecimal value of the character.
Usually, you just need to convert a few of the punctuation characters: % into %25,
& into %26, " into %22, < into %3C, = into %3D, > into %3E, + into %2B, | into %7C,
space into %20, [ into %5B, ] into %5D,
and convert all characters above #127 into their UTF-8 bytes and then percent encode
each byte of the UTF-8 form into the %HH format (ask a programmer for help).
Note that percent encoding is generally required when you access ERDDAP via
software other than a browser. Browsers usually handle percent encoding for you.
In some situations, you need to percent encode all characters other than
A-Za-z0-9_-!.~'()* .
Programming languages have tools to do this (for example, see Java's
java.net.URLEncoder
and JavaScript's
encodeURIComponent()) and there are
websites that percent encode/decode for you.
curl --compressed "https://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.png?sst%5B%282010-09-[01-05]T12:00:00Z%29%5D%5B%5D%5B%5D%5B%5D&.draw=surface&.vars=longitude|latitude|sst&.colorBar=|||||" -o BAssta5day201009#1.png
curl -v --data 'user=myUserName&password=myPassword' -c cookies.txt -b cookies.txt -k https://baseurl:8443/erddap/login.html
curl --compressed -v -c cookies.txt -b cookies.txt -k https://baseurl:8443/erddap/griddap/datasetID.fileType?query -o outputFileName
Using [start:stride:stop] When requesting dimension (axis) variables or data variables, the query may specify a subset of a given dimension by identifying the [start{{:stride}:stop}] indices for that dimension.
Or, if you change the fileType in the URL from .png to .graph, you can see a Make A Graph
web page with that request loaded:
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.graph?analysed_sst[(2002-06-01T09:00:00Z):(2002-07-01T09:00:00Z)][(29)][(-135)]
That makes it easy for humans to modify an image request to make a similar graph or map.
Or, if you change the fileType in the URL from .png to a data fileType (e.g., .htmlTable),
you can view or download the data that was graphed:
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.htmlTable?analysed_sst[(2002-06-01T09:00:00Z):(2002-07-01T09:00:00Z)][(29)][(-135)]
A sample URL to view a .png of a map is
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.png?analysed_sst[(2002-06-01T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]
Or, if you change the fileType in the URL from .png to .graph, you can see a Make A Graph
web page with that request loaded:
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.graph?analysed_sst[(2002-06-01T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]
Or, if you change the fileType in the URL from .png to a data fileType (e.g., .htmlTable),
you can view or download the data that was mapped:
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.htmlTable?analysed_sst[(2002-06-01T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]
Some ERDDAP installations do have authentication enabled. Currently, ERDDAP only supports authentication via Google-managed email accounts, which includes email accounts at NOAA and many universities. If an ERDDAP has authentication enabled, anyone with a Google-managed email account can log in, but they will only have access to the private datasets that the ERDDAP administrator has explicitly authorized them to access. For instructions on logging into ERDDAP from a browser or via a script, see Access to Private Datasets in ERDDAP.
To generate a citation for a dataset:
If you think of the dataset as a scientific article, you can generate a
citation based on the author (see the "creator_name" or "institution" metadata),
the date that you downloaded the data, the title (see the "title" metadata),
and the publisher (see the "publisher_name" metadata).
If possible, please include the specific URL(s) used to download the data.
If the dataset's metadata includes a
Digital Object Identifier (DOI), please
include that in the citation you create.
If the error response was generated by ERDDAP (not by some other part of the Internet,
e.g., Tomcat, Apache, routers, or your browser),
it will come with an (OPeN)DAPv2.0-formatted, plain text, UTF-8-encoded error message
as the payload of the response, e.g.,
Error {
code=404;
message="Not Found: Your query produced no matching results.
(time>=2019-03-27T00:00:00Z is outside of the variable's actual_range: 1970-02-26T20:00:00Z to 2019-03-26T15:00:00Z)";
}
Notes:
Or, you can join the ERDDAP Google Group / Mailing List by visiting
https://groups.google.com/forum/#!forum/erddap
and clicking on "Apply for membership".
Once you are a member, you can post your question there or search to see if the question
has already been asked and answered.