1.0.0b5 (April 27, 2020)¶
This is the fifth 1.0 beta release.
Highlights include:
- support for event forecasts and reports
- using plotly to generate report figures
- support for normalized metrics in the reports
- an option to account for uncertainty using a deadband in reports
- support for forecast skill in reports
See below for the full list of changes.
API Changes¶
- Added
solarforecastarbiter.reports.figures
which contains modules for creating Bokeh and Plotly report figures. - Added
solarforecastarbiter.reports.figures.plotly_figures
to create report figures using plotly (GH359)(GH388) solarforecastarbiter.reports.figures.bokeh_figures.output_svg()
now takes aselenium.webdriver.remote.webdriver.WebDriver
as an optional argument (GH345)- Add metrics for deterministic event forecasts
solarforecastarbiter.metrics.event.probability_of_detection()
,solarforecastarbiter.metrics.event.false_alarm_ratio()
,solarforecastarbiter.metrics.event.probability_of_false_detection()
,solarforecastarbiter.metrics.event.critical_success_index()
,solarforecastarbiter.metrics.event.event_bias()
, andsolarforecastarbiter.metrics.event.event_accuracy()
. (GH347) (GH348) - Add functions to get the time range of values available in the API:
solarforecastarbiter.io.api.get_observation_time_range()
,solarforecastarbiter.io.api.get_forecast_time_range()
,solarforecastarbiter.io.api.get_probabilistic_forecast_constant_value_time_range()
. (GH369) - Start and end are now optional parameters for the solararbiter referencedata update CLI command (GH369)
solarforecastarbiter.vavlidation.tasks.validate_timestamp()
has been replaced withsolarforecastarbiter.vavlidation.tasks.validate_defaults()
which also adds a nighttime flag (GH382)- Removed
normalizer
keyword argument fromsolarforecastarbiter.metrics.calculate_metrics()
andsolarforecastarbiter.metrics.calculate_deterministic_metrics()
. Normalization is now determined by the attributes of the datamodel objects. (GH379) - Many functions in
solarforecastarbiter.metrics.deterministic
gain a keyword argumenterror_fnc
for specifying a function that calculates the nominal difference forecast - observation. Among other things, this is useful for specifying a deadband. (GH378) - Added support for probabilistic forecasts with
solarforecastarbiter.metrics.calculator.calculate_metrics()
modifications andsolarforecastarbiter.metrics.calculator.calculate_probabilistic_metrics()
addition. (GH315) (GH266) (GH367) - Added api function to get all values of a
solarforecastarbiter.datamodel.ProbabilisticForecast
with addition ofsolarforecastarbiter.io.api.get_probabilistic_forecast_values()
. (GH367) - Removed
ref_pair
fromsolarforecastarbiter.metrics.calculator.calculate_metrics()
andref_fx_obs
fromsolarforecastarbiter.metrics.calculator.calculate_deterministic_metrics()
andsolarforecastarbiter.metrics.calculator.calculate_probabilistic_metrics()
. Instead, use the reference forecast attributes ofsolararbiter.ForecastObservation
andsolararbiter.ProcessedForecastObservation
to supply reference forecast metadata and values. (GH386, GH387) solarforecastarbiter.metrics.preprocessing.resample_and_align()
now requires input reference forecast time series (optionallyNone
), and also returns processed reference forecast time series (orNone
). (GH386, GH387)- Added capability to import reference data from DOE ARM sites in the
solarforecastarbiter.io.reference_observations.arm
module. Data can be initialized and updated via the Solar Forecast Arbiter CLIreferencedata
command.(GH390) - Added support for event observations and forecasts with
solarforecastarbiter.datamodel.Observation()
modifications andsolarforecastarbiter.datamodel.EventForecast()
addition. (GH357) (GH361) (GH399) - Added support for evaluating event forecasts with
solarforecastarbiter.metrics.calculator.calculate_metrics()
modification,solarforecastarbiter.metrics.calculator.calculate_event_metrics()
addition andsolarforecastarbiter.metrics.preprocessing.resample_and_align()
modifications. (GH357) (GH380) (GH361) - Added support for plotting timeseries of event observations and forecasts to
solararbiter.plotting.timeseries.make_basic_timeseries()
. (GH394)
Enhancements¶
- Report plots are now created using Plotly. Bokeh figure generation moved to
solarforecastarbiter.reports.figures.bokeh_figures
for users still interested in bokeh figures.(GH359) - Report timeseries plots no longer draw lines over missing data points (GH351).
- Double clicking a forecast or observation in the legend of report timeseries plots will toggle the display of all other timeseries. (GH253)
- Clean up any PhantomJS drivers created to render SVGs (GH344) (GH349)
- Reference data updates can now proceed from the last value in the API to avoid any artificial data gaps (GH369) (GH353)
- GHI validation now includes checks on if a period is cloud-free and will be flagged with CLEARSKY (GH210) (GH382)
- All data validation now includes the generation of the NIGHTTIME flag (GH293) (GH382)
- Metrics automatically determine normalization based on Observation type.
AC power observations are normalized by AC capacity; DC power by DC
capacity. Normalized metrics set to
nan
for all other variables. (GH370) (GH379) - Metrics
'mae', 'mbe', 'rmse', 'mape', 'nmae', 'nmbe', 'nrmse', 's'
may now be calculated using a deadband. The deadband is specified as a percentage of the observations. The error forecast - observation is set to 0 within the deadband. The deadband is controlled using thesolarforecastarbiter.datamodel.ForecastObservation
andsolarforecastarbiter.datamodel.ForecastAggregate
uncertainty
argument.None
implies no deadband and a float sets the deadband. Additionally, forsolarforecastarbiter.datamodel.ForecastObservation
, the string'observation_uncertainty'
may be supplied to set the deadband equal tosolarforecastarbiter.datamodel.Observation.uncertainty
. (GH358, GH378) - Reports can now report the forecast skill metric (s) when a reference forecast is provided. The metadata table of forecasts and observations now includes reference forecasts. (GH386, GH387)
- Reports now support evaluating event forecasts. (GH380) (GH361)
Bug fixes¶
- Fix using deprecated ‘M’ string for a month offset in
solarforecastarbiter.io.reference_observaions.srml.fetch()
(GH341) (GH342) - Fix CLI report generation when status not set in report metadata (GH345)
- Fix bug with APISession list functions when only a single value is available. (GH367)
- Fix type errors in observation validation by ensuring all time series values from
solarforecastaribter.io.api
functions are coerced to float and converting to float before validation (GH252) (GH384) - Enforce metric order consistency. (GH352) (GH396)
- Eliminate most warnings from test suite. (GH385) (GH395)
solarforecastarbiter.metrics.deterministic.forecast_skill()
now returns 0 if both forecast and reference forecast errors are 0. (GH395)- Clarify report data validation section text and table. (GH413) (GH422)
Contributors¶
- Will Holmgren (@wholmgren)
- Leland Boeman (@lboeman)
- Cliff Hansen (@cwhanse)
- Tony Lorenzo (@alorenzo175)
- Justin Sharp (@MrWindAndSolar)
- Aidan Tuohy
- Adam Wigington (@awig)
- David Larson (@dplarson)