Easier access to scientific data
Brought to you by NOAA NMFS SWFSC ERD
|
ERDDAP
Easier access to scientific data |
Brought to you by NOAA NMFS SWFSC ERD |
ERDDAP was designed around the idea that most datasets are huge, so most users
just need or want a subset of the dataset that they are interested in
(e.g., a smaller geographic area, a smaller time range, or not all of the data variables).
But we
understand that some users actually do want an entire dataset, or at least the subset
which is found in a subset of the source data files. If that's you, then the "files"
system may be for you. One advantage of the "files" system is that you can see each
file's size (in bytes) and Last Modified time (Zulu time zone),
so it is easy to see if a file has been changed.
Similarly, when you request a subset of data from one of ERDDAP's Data Access Forms,
you can specify the file type (e.g., .nc, .csv, .json, .mat) that you want to receive in
response. Naturally, the source data files available via "files" are just available in
one file type. If you aren't happy with the source file's file type, you might prefer
using the dataset's Data Access Forms instead.
in the HTTP GET request header sent
by the client. If a supported compression type ("gzip", "x-gzip", or "deflate") is found
in the accept-encoding list, ERDDAP includes "content‑encoding" in the HTTP response
header and compresses the data as it transmits it.
It is up to the client program to look for "content-encoding" and decompress the data accordingly.
Requesting compression is optional, but compressed responses are often 3-10 times faster,
so this is a big time savings if you are downloading lots of large files.
(Note that there is no benefit
to requesting compressed .png files since the files' contents are already compressed.)
.
(although you should either handle "deflate"'d responses or not request "deflate").
and install it. To get to a command line in Windows, click on "Start" and type
"cmd" into the search text field.
("Win32 - Generic, Win32, binary (without SSL)" worked for me in Windows 7.)
Instructions for using curl are on the
curl man page and in this
curl tutorial.
But here is a quick tutorial related to using curl with ERDDAP:
curl --compressed -g "https://coastwatch.pfeg.noaa.gov/erddap/files/cwwcNDBCMet/NDBC_41004_met.nc" -o ndbc/41004.ncIn curl, as in many other programs, the query part of the erddapUrl must be percent encoded
:
all characters in parameter values (the parts after '=' signs) other than A-Za-z0-9_-!.~'()*
must be encoded as %HH, where HH is the 2 digit hexadecimal value of the character,
for example,
a space becomes %20. Characters above #127 must be converted to UTF-8 bytes, then each UTF-8
byte must be percent encoded (ask a programmer for help). There are
websites that percent encode and decode for you
.
If you get the URL from your browser's address text field, this may be already done.
them in the erddapURL as %5B, %5D, %7B, %7D, respectively.
Fortunately, these are almost never in "files" filenames.
Then, in the erddapUrl, replace a zero-padded number (for example 01) with a
range of values (for example, [01-15] ), or replace a substring (for example 41004) with a list of values (for example, {41004,41009,41010} ).
The #1 within the output fileName causes the current value of the range or
list to be put into the output fileName. For example,
curl --compressed "https://coastwatch.pfeg.noaa.gov/erddap/files/cwwcNDBCMet/NDBC_{41004,41009,41010}_met.nc" -o ndbc/#1.nc
Similarly, for a few audio file types (notably .mp3, .ogg, and .wav), you will see an audio control which allows you to listen to the audio file.
These preview features will only work for certain file types, in certain browsers, in certain operating systems. They rely on browser features, so they are largely out of our control.
Alternatively, if you click on the link for an image, audio, or video file, a viewer
or player will open in a separate window. (If your browser asks you what you want
to do with the file, tell it to handle the media file itself (not via other software),
and tell it to
remember this choice so that it will be used automatically in the future.)
If you instead choose to download and work with the original source files offered by ERDDAP's "files" system, you have to figure out how you want to work with the files. Fortunately, there are lots of software tools for working with the various file types:
.
which give a basic description of the file type and list software that can
be used to work with the files (view, read, write, edit, etc.).
Or, you can use your favorite search engine to search for what you want.
See the
Wikipedia List of Audio File Formats.
If you want to do other things with these files, there are numerous programs to play and edit audio files including:
,
an open source program to record, edit, and max on Linux, Mac OSX, and Windows.
.
.
is a common type of binary data file.
There are a few software packages that can work with .hdf files, including:
is free, commonly used software to make graphs and maps from .hdf files.
is the official library from the HDF Group to read and write all .hdf files.
, and
R
can read .hdf files via an add-on library.
See the
Wikipedia List of Image File Formats.
If you want to do other things with image files, there are numerous image viewing and editing programs including:
,
an open source program for all operating systems.
,
a free image editor for Windows.
.
.
is a common type of binary data file.
There are two subcategories of .nc files: version 3 files (still widely used) and version 4 files (which are actually .hdf files with a few changes). Files of both versions have the extension .nc and can be read by programs that read .nc files.
There are a large number of software packages that can work with .nc files, including:
and
Ncview
(which can also be installed
via Conda).
,
the main software libraries for C, C++, Fortran, or Java to read and write .nc files.
,
Matlab, and
R
can read .nc files (perhaps via an add-on library), make graphs and maps, and
work with the data in .nc files.
.
are different than word processing files, which have special embedded
formatting commands. If you import a text file into a word processor and make changes
to the file, be sure to then save the file as an ASCII text file once again.
Or, avoid this problem by using a text editor program.
If you edit .tsv (tab separated value) files, be very careful to maintain the tabs which separate the values in different "columns" on each row. By default (even in many text editors), tabs often appear as spaces (or a few spaces). So be sure to use the editor's feature that makes tabs visible (as a special symbol) so that you can maintain the tabs between values.
.xml files are technically text files, but there are advantages to using separate XML editors to work with them.
There are dozens of text editor programs for every operating system, including:
.
See the
Wikipedia List of Video File Formats.
If you want to do other things with video files, there are numerous video playing and editing programs including:
.
.
are structured text files. You can view them in your browser or in a text editor,
but there is also specialized software for working with XML files.
See this
Wikipedia Comparison Of XML Editors.
.
This is used by some client software (for example, audio and video players in web browsers)
to request chunks of the file instead of the whole file.
Accessing a remote file via byte ranges is often slow and inefficient. Sometimes it's worth it for reading small samples of remote files, notably audio and video files. But the more times you need access to the file, the more efficient it is to just download the file and then work with the local file.
ERDDAP's /files/ system refuses all byte range requests to .nc and .hdf files, so don't even try to use Netcdf-java/c, ncview, Ferret, or other software tools to connect to .nc or .hdf files served by ERDDAP's /files/ system as if they were local files. These requests are blocked because this approach is horribly inefficient and often causes other problems. Instead: