Reference Observations¶
Overview¶
The Solar Forecast Arbiter imports reference observation data from multiple
measurement networks. All of the logic for creating the appropriate Solar
Forecast Arbiter sites and observations and updating observations with new data
from the network can be found in the
solarforecastarbiter.io.reference_observations
subpackage. Code for
retrieving data from the network’s APIs are spread between the
solarforecastarbiter.io.fetch
subpackage, and the
pvlib python iotools
module.
A list of these networks and their Solar Forecast Arbiter modules can be found in the Available Network Handlers section. A map of all of the sites available in the reference dataset can be found on the Solar Forecast Arbiter project website.
Structure¶
The solarforecastarbiter.io.reference_observations
subpackage
contains python modules and data files in JSON and CSV format.
Data Files¶
- sfa_reference_sites.csv
- The master list of reference sites. See the comment at the top of this file for descriptions of its fields. The file contains extra fields that are not found in the Solar Forecast Arbiter API schema for Sites. These fields are for use with the source network’s API and are stored in the extra_parameters field when the site is created for use in subsequent updates.
- <network>_reference_sites.json
- Network-specific files containing site and observation metadata in the Solar Forecast Arbiter API’s JSON format. These are used when the master CSV does not contain all of the columns needed to accurately define a site or observation.
Modules¶
solarforecastarbiter.io.reference_observations.reference_data
- This module coordinates the initialization and update process. It also contains the NETWORKHANDLER_MAP dictionary, which maps network names to the correct Network Handlers. The functions in the module are utilized by the CLI referencedata command.
solarforecastarbiter.io.reference_observations.common
- The common module contains utility functions for use throughout the reference_data subpackage. It has useful functions for converting external data into Solar Forecast Arbiter Datamodel objects and network-agnostic utilities for preparing and posting data to the Solar Forecast Arbiter API. Most Network Handlers rely heavily on these functions.
Network Handlers¶
Network Handlers are network specific modules that implement a handful of
functions with a common interface. See
solarforecastarbiter.io.reference_observations.surfrad
for an
example.
The required network handler functions are:
- initialize_site_observations(api, site)
Create an observation at the site for each variable available from the network.
- initialize_site_forecasts(api, site)
Create a forecast for each observation at the site.
- update_observation_data(api, sites, observations, start, end)
Retrieve data from the network then format and post it to each observation at the site.
- api:
solarforecastarbiter.io.api.APISession
- sites: list of
solarforecastarbiter.datamodel.Site
- observations: list of
solarforecastarbiter.datamodel.Observation
- start: datetime
- end: datetime
- api:
- (optional) adjust_site_parameters(site)
In instances where the master site CSV does not contain enough metadata about the site, (e.g. when a PV plant requires modeling_parameters) this function may be used to update the site metadata before it is posted to the API.
- site: dict
Available Network Handlers¶
- SURFRAD: NOAA Surface Radiation Budget Network
- SOLRAD: NOAA SOLRAD Network
- CRN: NOAA U.S. Climate Reference Network
- NREL MIDC: National Renewable Energy Laboratory Measurement and Instrumentation Data Center
- UO SRML: University of Oregon Solar Radiation Monitoring Laboratory
- DOE RTC: DOE Regional Test Centers for Solar Technologies*
- DOE ARM: DOE Atmospheric Radiation Measurement*
- NREL PVDAQ: National Renewable Energy Laboratory PV Data Acquisition*
- EIA: U.S. Energy Information Adminstration Open Data*
- WRMC BSRN: World Radiation Monitoring Center - Baseline Surface Radiation Network*
* Requesting data from these networks requires a valid api key or other credentials for the associated API.