California Independent System Operator Corporation

California Independent System Operator Corporation

California ISO

Flexible Ramping Product Uncertainty Calculation and Implementation Issues

April 18, 2018

Prepared by: Kyle Westendorf, Department of Market Monitoring

Department of Market Monitoring – California ISO April 18, 2018

CAISO/DMM/KW iii

TABLE OF CONTENTS

Executive Summary ...................................................................................................................... 1

1 Overview ............................................................................................................................ 3

1.1 Overview of flexible ramping product ..................................................................................................... 3 1.2 Flexible ramping product demand curves and uncertainty ..................................................................... 4 1.3 Review of uncertainty requirements as implemented ............................................................................ 5 1.4 Summary of implementation errors ........................................................................................................ 7

2 Implementation errors and issues ........................................................................................ 9

2.1 Advisory net load error calculation error ................................................................................................ 9 2.2 Wind and solar error .............................................................................................................................. 12 2.3 Reference interval error ........................................................................................................................ 12 2.4 15-minute market VER forecast inputs .................................................................................................. 13 2.5 Uncertainty thresholds used to cap uncertainty requirements ............................................................ 16 2.6 ISO load error in the uncertainty distributions ...................................................................................... 18 2.7 Other implementation issues ................................................................................................................ 19

3 Analysis of market impacts ................................................................................................ 21

3.1 Corrected uncertainty calculation ......................................................................................................... 21 3.2 Power balance constraint relaxation ..................................................................................................... 23 3.3 Impact on flexible ramping product prices and procurement ............................................................... 25 3.4 Sufficiency test ....................................................................................................................................... 29

4 Appendix .......................................................................................................................... 31

4.1 Implemented versus corrected balancing area uncertainty requirements ........................................... 31 4.2 Shadow price impact analysis scenarios ................................................................................................ 38 4.3 Supplemental analysis of implemented uncertainty and demand curves ............................................. 42 4.4 ISO Tariff ................................................................................................................................................ 45 4.5 Technical details in the Business Practice Manual ................................................................................ 45


CAISO/DMM/KW 1

Executive Summary

The ISO implemented a new market feature known as the flexible ramping product in November 2016. This product is designed to enhance reliability and market performance by procuring flexible ramping capacity in the real-time market to help manage volatility and uncertainty of real-time imbalance demand. The amount of flexible capacity the product procures is derived from a demand curve which reflects a calculation of the optimal willingness-to-pay for that flexible capacity. The demand curves allow the market optimization to consider the trade-off between the cost of procuring additional flexible ramping capacity and the expected reduction in power balance violation costs.

Since implementation of the flexible ramping product, the Department of Market Monitoring (DMM) has raised numerous concerns and questions about the implementation and performance of the flexible ramping product. In February 2018, DMM identified numerous errors in how the demand curves used to procure flexible capacity have been calculated. The demand curves for the flexible ramping product are intended to be calculated based on the expected ramp needed to meet scheduled loads, as well as the uncertainty surrounding ramping needs. DMM’s review indicates that the uncertainty component of the demand curve was not properly calculated using historical data on the error surrounding forecasted ramping needs.

These errors in calculation of the uncertainty component of the flexible ramping product demand curve appear to have had the following impacts:

• The demand curves from the flexible ramping product were systematically biased in the opposite direction of the net load ramp. This caused requirements to be significantly lower than intended in many hours with relatively high ramping needs, and significantly higher than intended in other hours which tend to have lower ramping needs.

• The overall impact of these errors on flexible ramping market results was significant. DMM estimates that prices and purchased quantities of upward ramping capacity were lower than intended in up to about half of all 15-minute intervals. During these intervals, the correct requirements averaged almost 400 MW greater than historical procurement on average (i.e. 949 MW compared to 564 MW procured).

• The systematic under-procurement of flexible ramping capacity during key hours may have increased the frequency of power balance violations. However, it is not possible to determine whether any particular power balance violation would have been resolved had the flexible ramping product been implemented correctly.

• The systematic errors leading to procurement of less flexible ramping capacity during many intervals could have contributed to other notable market trends in 2017, such as the increased and systematic use of the load bias during ramping hours by grid operators. However, any such indirect impacts cannot be assessed.

This report provides a more detailed description of these errors and the estimated impact of these error from the time the flexible ramping product was implemented in November 2016 through early 2018.

Beginning in late February 2018, the ISO began to implement a number of changes to correct these implementation errors. While the issues with the largest impact have been addressed, the ISO continues working to resolve several remaining issues.


CAISO/DMM/KW 3

1 Overview

The ISO implemented a new market feature known as the flexible ramping product on November 1, 2016. This product is designed to enhance reliability and market performance by procuring flexible ramping capacity in the real-time market. This flexible capacity can be utilized to help manage volatility and uncertainty of real-time imbalance demand.

Beginning in 2017, DMM has raised numerous concerns and questions about the implementation and performance of the flexible ramping product. These concerns included the systematic nature of the net load forecast error calculation that drives flexible ramping requirements and procurement.1

In February 2018, DMM identified numerous specific errors in how the flexible ramping product was implemented. The errors involve the calculation of the demand curves used to procure flexible capacity. These errors appear to have caused the market software to under-procure upward flexible capacity during key net load ramping intervals. In addition the errors would have impacted flexible ramping prices and payments depending on the hour.

Beginning in late February 2018, the ISO began to implement a number of changes to correct these implementation errors. While the issues with the largest impacts have been addressed, the ISO continues working to resolve several remaining issues. This report describes the implementation errors as well as some of the implications and impacts of these implementation issues on market results since the product was implemented in November 2016.

1.1 Overview of flexible ramping product

The flexible ramping product is designed to ensure that there is sufficient flexible ramping capacity available in real-time to address uncertainty that can arise from load or renewable generation. This uncertainty stems from the amount of net load that will exist in the real-time market. Net load refers to the difference between system loads minus output from wind and solar generation. This represents the portion of load that needs to be met by other sources of energy, including dispatchable gas-fired generation that is used to balance changes in intermittent sources of renewable energy and other factors affecting demand for real-time energy.

When the real-time market software runs the optimization for the current binding interval, the net load values in future advisory intervals are not known. The market software must use a forecast for the net load in these future advisory intervals. However, the net load in those future intervals may differ from this advisory forecast. Therefore, flexible ramping capacity is needed not only to make the advisory interval’s forecasted net load feasible (or maintain power balance) but also to make a larger range of potential net loads in the advisory interval feasible.

The flexible ramping product is designed to ensure a margin of sufficient ramping capacity beyond the forecasted ramping needs to protect against power balance violations. As more wind and solar capacity

1 For example, DMM’s 2016 Annual Report specifically noted that ”the hourly profile of the flexible ramping demand curves

suggests that there are systematic net load forecast errors for some hours of the day. A better understanding of the underlying causes for these errors would be valuable.” See Annual Report on Market Issues and Performance, Department of Market Monitoring, May 2017, p.120, http://www.caiso.com/Documents/2016AnnualReportonMarketIssuesandPerformance.pdf

http://www.caiso.com/Documents/2016AnnualReportonMarketIssuesandPerformance.pdf


CAISO/DMM/KW 4

is added to the system, this is increasing the need for such flexible capacity to manage any increase in net load volatility and reduce the frequency of power balance constraint relaxations.

The flexible ramping product procures both upward and downward flexible capacity, in both the 15-minute and 5-minute markets. Procurement in the 15-minute market is intended to ensure that enough ramping capacity is available to meet the needs of both the upcoming 15-minute market runs and the three 5-minute market runs with that 15-minute interval. Procurement in the 5-minute market is aimed at ensuring that enough ramping capacity is available to manage differences between consecutive 5-minute market intervals.

1.2 Flexible ramping product demand curves and uncertainty

A key component of the flexible ramping product market design is that the amount of flexible capacity that the product procures is derived from a demand curve which reflects a calculation of the optimal willingness-to-pay for that flexible capacity. The demand curves allow the market optimization to consider the trade-off between the cost of procuring additional flexible ramping capacity and the expected reduction in power balance violation costs. If implemented correctly, this approach can increase market efficiency by signaling an explicit trade-off between the costs and benefits of procuring more or less flexible ramping capacity for a given interval.

The demand curves for the flexible ramping product are intended to be calculated based on the expected ramp needed to meet scheduled loads, as well as the uncertainty surrounding ramping needs. This uncertainty was intended to be calculated using historical data on the error surrounding forecasted ramping needs.

The demand curves are calculated independently for each hour of the day, and differ by market (15-minute and 5-minute) and direction (upward ramping and downward ramping). There are separate demand curves calculated for each energy imbalance market area in addition to a system-level demand curve. These demand curves are intended to be calculated from historical net load forecast error data.2

The flexible ramping product is incorporated into the ISO’s market optimization as a “soft” constraint which can be relaxed depending on the cost of meeting the constraint. With this approach, the demand curves are first entered into the market software as segments of relaxation capacity that reflect the expected cost of a power balance constraint violation for the level of foregone capacity procurement. The maximum amount of capacity on the demand curve (or uncertainty) is then treated as a requirement and is met in every interval through a combination of flexible ramping capacity procurement or relaxation capacity.3

The system-level demand curve for the entire is CAISO/EIM footprint is always enforced in the market. However, the uncertainty requirement for the individual balancing areas is reduced in every interval by their transfer capability.4 The demand curves specific to the individual areas can therefore be binding 2 Weekdays use data for the same hour from the last 40 weekdays. For weekends, the last 20 weekend days are used. 3 While the uncertainty requirement is commonly referred to as a requirement, DMM notes that this value reflects the end

point of the demand curve, or the maximum amount of flexible capacity the market is willing to pay for, rather than a hard requirement.

4 In each interval, the upward uncertainty requirement is reduced by net import capability while the downward uncertainty requirement is reduced by net export capability. If the balancing authority area fails the flexible ramping sufficiency test in the corresponding direction, the uncertainty requirement will not include this reduction.


CAISO/DMM/KW 5

when insufficient transfer capability is present, which indicates that the area is unable to benefit from the flexible capacity from other areas. When the uncertainty requirement for all of the individual areas is zero, then only the system-level uncertainty requirement is active. DMM has recommended modifications in how flexible ramping product demand curves for different balancing areas are treated in the market optimization.5 However, this paper focuses on the impact of the implementation errors on the system-level uncertainty requirements and demand curves. 1.3 Review of uncertainty requirements as implemented

This section reviews results of how the ISO calculated the uncertainty component of the flexible ramping product demand curve since the product was implemented in late 2016 through early 2018. As illustrated in this section, these results appeared inconsistent with what DMM expected given the flexible ramping design and market conditions. These outcomes are the result of specific implementation errors identified in this report.

Figure 1.1 and Figure 1.2 show the calculated hourly distribution of system-level net load errors used for the 15-minute and 5-minute market uncertainty requirements and demand curves for February 20, 2018. The box plots reflect the distribution of net load errors in the previous 40 weekdays for each hour. The red lines show the 2.5th and 97.5th percentile that create the uncertainty requirements for each hour on the day. If the 97.5th percentile of observations is below zero or if the 2.5th percentile of observations is above zero, then the uncertainty requirement is set at zero megawatts instead. Further, the uncertainty requirements are capped at the other end by uncertainty thresholds.6

As shown in the example, the distributions reveal systematic net load errors during many hours with very negative net load error (and very low or zero upward uncertainty requirements) during the early morning and evening hours when net load is consistently ramping up (e.g. morning hours 5 to 7 and evening hours 14 to 19). Conversely, net load errors were highest in hours when net load is typically ramping down (e.g. hours 8 to 9, 21 to 2).

This pattern is reflective of outcomes since the implementation of the flexible ramping product for both the 15-minute and 5-minute markets. Figure 1.3 shows the average calculated net load error by quarter for the system in the 15-minute market during 2017.7 In particular, calculated net load errors were often very negative during peak net load-ramping periods. This contributed to lower-than-expected upward uncertainty requirements during these periods, hours in which power balance shortages have occurred more often.

5 For additional information on this issue, see DMM’s Q3 2017 Report on Market Issues and Performance, December 2017, pp.

49-52: http://www.caiso.com/Documents/2017ThirdQuarterReport-MarketIssuesandPerformance-December2017.pdf. 6 For the system in the 15-minute market, the threshold is set at -1,200 megawatts in the downward direction and 1,800 MW

in the upward direction. For the system in the 5-minute market, this is -300 MW and 500 MW in the downward and upward direction, respectively. For more information on the uncertainty thresholds, see Section 2.5.

7 Corresponding values for the 5-minute market show a very similar pattern but at about one-third of the scale.

http://www.caiso.com/Documents/2017ThirdQuarterReport-MarketIssuesandPerformance-December2017.pdf


CAISO/DMM/KW 6

Figure 1.1 Calculated hourly distribution of 15-minute market system net load error (February 20, 2018)

Figure 1.2 Calculated hourly distribution of 5-minute market system net load error (February 20, 2018)

7,000

8,000

9,000

10,000

11,000

12,000

13,000

14,000

-3,000

-2,000

-1,000

0

1,000

2,000

3,000

4,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Net

load

err

or (M

W)

Hour

9,000

9,200

9,400

9,600

9,800

10,000

10,200

10,400

10,600

10,800

11,000

-1,000

-800

-600

-400

-200

0

200

400

600

800

1,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Net

load

err

or (M

W)

Hour


CAISO/DMM/KW 7

Figure 1.3 Hourly average calculated 15-minute market system net load error by quarter

1.4 Summary of implementation errors

Results of DMM’s analysis indicate that systematic net load errors illustrated in Figure 1.1 through Figure 1.3 were primarily the result of a specific implementation error in the calculation of the uncertainty component of the demand curve, as described below.

• The net load errors in the hourly historical distribution were designed (or intended) to be calculated as the difference between (1) the binding net load forecast for the next interval and (2) the advisory net load forecast for the same corresponding time interval from the prior market run.

• However, when the flexible ramping product was implemented, the net load error calculation was instead based on the difference between the binding and advisory interval in the same market run between two sequential time intervals, or the negative of the expected change in net load between two sequential time intervals.

A more detailed description of this error is provided in Section 2 of this report. This error resulted in the net load error pattern observed in Figure 1.1 through Figure 1.3 which biased flexible ramping capacity demand and procurement in the direction opposite the net load ramp.

In addition, DMM has identified other issues with the uncertainty calculation that have had an impact on the accuracy of the net load error observations, but appear to have had a smaller impact on the overall pattern and outcome. These includes the following:

• The wind and solar values used to calculate the net load forecast were pulled from the interval prior to the load values. All components of the net load calculation should be pulled from the same interval.

-1,500

-1,000

-500

0

500

1,000

1,500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Aver

age

net l

oad

erro

r (M

W)

Q1 Q2 Q3 Q4


CAISO/DMM/KW 8

• Net load errors in the implemented distribution reflect the uncertainty in the current interval rather than the uncertainty in the next interval. The flexible ramping product is expected to procure flexible ramping capacity in a binding interval, 𝑡𝑡, based on the expected uncertainty in the next binding interval, 𝑡𝑡 + 1.

• The 15-minute market wind and solar values used to calculate the advisory 15-minute market net load were instead pulled from the last 5-minute level output corresponding to the 15-minute market period.

• Uncertainty requirements are capped by undocumented uncertainty thresholds that have been binding more frequently than expected and are based on outdated and erroneous flexibility needs.

These implementation issues and errors are discussed in further detail in Section 2. Section 3 outlines several implications of the implementation errors described in Section 2. This section includes a comparison of the uncertainty requirements resulting from the incorrect method used until recently with DMM’s estimate of the uncertainty requirements had the calculation been implemented correctly. Section 3 also examines the impact of these errors on the frequency of power balance constraint relaxations and the impact on flexible ramping shadow prices and procurement.


CAISO/DMM/KW 9

2 Implementation errors and issues

This section provides a detailed description of the flexible ramping product implementation errors and issues identified by DMM that are examined in this report.

2.1 Advisory net load error calculation error

Flexible ramping product procurement and prices are determined through demand curves, expected to be calculated from historical net load forecast error, or uncertainty. For the 5-minute market, this calculation is intended to be equal to the binding net load in the next market run minus the first advisory interval of the current market run, as shown in Figure 2.2 (B2 – A1).

However, the implementation of the flexible ramping product was based on the binding net load in the current market run minus the first advisory interval of the current market run (B1 – A1). This is analogous to the negative of the expected change in net load (i.e. net loadt – net loadt+1). By calculating uncertainty in this manner (between sequential time intervals), the calculation systematically biases flex ramp procurement in the direction opposite of the net load ramp (down when net load is ramping up and vice versa). This has resulted in periods of zero or very low megawatt uncertainty requirements during hours with significant net load ramps.

Figure 2.1 5-minute market histogram construction (BPM for Market Operations)8

For example, Figure 2.2 shows the average implemented upward uncertainty requirement in the 5-minute market with average hourly system net load (right-axis), where net load is load minus wind and solar. The figure also includes the hourly average inverse change in net load between sequential intervals. As show in the figure, the implemented uncertainty requirements (pulled from the 97.5th percentile of net load error observations) tracked closely with the opposite of the net load ramping direction.

8 See Market Operations Business Practice Manual version 55, pp. 239:

https://bpmcm.caiso.com/BPM%20Document%20Library/Market%20Operations/BPM_for_Market%20Operations_V55_redline.pdf.

https://bpmcm.caiso.com/BPM%20Document%20Library/Market%20Operations/BPM_for_Market%20Operations_V55_redline.pdf



CAISO/DMM/KW 10

Figure 2.2 Average system net load and upward uncertainty requirements (2017)

This same issue is also present with the 15-minute market uncertainty calculation. However in the 15-minute market calculation, the systematic bias because of the net load ramp direction can be offset by differences between 15-minute and 5-minute forecasts. The 5-minute market uncertainty calculation compares two 5-minute intervals while the 15-minute market uncertainty calculation compares a binding 5-minute interval with a corresponding advisory 15-minute interval. As a result, the uncertainty requirements in the 15-minute market, pulled from the 2.5th and 97.5th percentile of net load error observations, were zero megawatts less frequently than in the 5-minute market, but the overall distributions were typically still shifted in the direction opposite the net load ramp.

In late February 2018, the ISO corrected the net load error distributions so that these were based on an advisory and binding net load forecast in the same time-interval. These distributions were used in the market to calculate the uncertainty requirements and demand curves beginning February 22, 2018.

Figure 2.3 and Figure 2.4 show the difference between the hourly system-level uncertainty requirements on February 20, 2018 (pre-fix) and February 22, 2018 (post-fix) for the 15-minute and 5-minute markets, respectively. The upward uncertainty requirements were equal to the upper lines while the downward uncertainty requirements were equal to the lower lines. The uncertainty requirements used in the market are capped at zero megawatts at one end and at the uncertainty thresholds at the other.9 Since the implementation of the new distributions, upward and downward uncertainty requirements have been non-zero during all hours.

9 The uncertainty thresholds are discussed in further detail in Section 2.5.

0

10,000

20,000

30,000

40,000

50,000

-400

-200

0

200

400

600

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Net

load

(MW

)

MW

Implemented upper uncertainty requirementNegative change in net loadNet load

(net loadt - net loadt+1)


CAISO/DMM/KW 11

Figure 2.3 15-minute market system-level uncertainty requirements (February 20 versus February 22, 2018)

Figure 2.4 5-minute market system-level uncertainty requirements (February 20 versus February 22, 2018)

-1,500

-1,000

-500

0

500

1,000

1,500

2,000

2,500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Syst

em-le

vel u

ncer

tain

ty re

quire

men

t (M

W) February 20, 2018 February 22, 2018 Uncertainty thresholds

-400

-300

-200

-100

0

100

200

300

400

500

600

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Syst

em-le

vel u

ncer

tain

ty re

quire

men

t (M

W)

February 20, 2018 February 22, 2018 Uncertainty thresholds


CAISO/DMM/KW 12

Other issues have been identified with the uncertainty calculation that impact the accuracy of the net load error observations, but have had a relatively smaller impact on the on the overall pattern and outcome. The ISO have addressed most but not all of these issues, but has indicated their intention to do so. These issues are described in the following sections.

2.2 Wind and solar error

The uncertainty requirements are expected to be calculated from historical net load forecast error between a binding and corresponding advisory interval where the net load forecast is equal to the load forecast minus the forecasts of wind and solar variable energy resources (VERs). However, the VER data pulled to calculate net load for the uncertainty calculation was pulled from the previous interval as the load data. Net load error is intended to be calculated from load and VER components in the same interval. This ultimately impacts the accuracy of the uncertainty distributions that feed into the demand curves and market outcomes.

The ISO has corrected net load error distributions to resolve this misalignment. These distributions were used in the market to calculate the uncertainty requirements and demand curves beginning March 23, 2018.

2.3 Reference interval error

The flexible ramping product is intended to procure flexible ramping capability in a given interval based on the estimated uncertainty from observations in the following interval. For instance for the 5-minute market, uncertainty should be calculated as the binding net load in the next market run (next interval) minus the first advisory interval of the current market run (next interval). Therefore, uncertainty observed in interval 8 – for example – is expected to be assigned a reference interval of 7 when creating the hourly distributions of net load errors. These distributions would then reflect a sample of uncertainty for the following interval and procurement would be based on these values accordingly.

However, the net load error calculated with the implementation of the flexible ramping product was based on load differences between two time intervals in the current market run and VER differences between two time intervals in the previous market run rather than net load error in the next interval.

Table 2.1 illustrates the implementation issues described in Section 2.1 through 2.3 for the 5-minute market uncertainty calculation with an example. The columns show sequential 5-minute market intervals while the rows show three sequential market runs. The blue cells represent the binding intervals while the orange cells reflect the first advisory intervals.

As illustrated in Table 2.1, the net load error (uncertainty) assigned to interval 7 is expected to be equal to the binding net load in the next market run (B2) minus the first advisory net load of the current market run (A1). However, the load error was implemented as the binding load forecast in the current market run (B1) minus the first advisory load forecast in the current market run (A1). In addition, the VER forecast error was pulled from the previous interval as the load forecast error, or the binding VER forecast in the previous market run (B0) minus the first advisory VER forecast in the previous market run (A0).

In practice, the net load errors are grouped in hourly distributions such that the lagged reference interval only impacts observations at the start of the hours that should have otherwise been included in the distribution of the previous hour. The impact of this issue on the demand curves and associated market outcomes is therefore estimated to be small.


CAISO/DMM/KW 13

Table 2.1 Example. Flexible ramping product uncertainty calculation implementation – interval 7 (5-minute market)

2.4 15-minute market VER forecast inputs

The issues described in Section 2.1 through 2.3 impacted both the 5-minute market and 15-minute market net load error calculations. The issue described in this section is specific to the 15-minute market net load error calculation.

For the 15-minute market, net load error is calculated as the difference between the maximum or minimum binding net load forecast of three corresponding 5-minute market intervals and the first advisory 15-minute market net load forecast of the current market run, as shown in Figure 2.5 (b3 – A).10

10 The hourly 15-minute market uncertainty distribution is composed of the maximum of the three 5-minute market

observations when the net load error is positive and the minimum of the three 5-minute market observations when the net load error is negative.

Expected error calculation (load and VER):Hour ending:RTD interval: 6 7 8 9 10 …

Previous market run (binding in interval 6) B0 A0 …

Current market run (binding in interval 7) B1 A1 …

Next market run (binding in interval 8) B2 A2 …

Load forecast error implementation:Hour ending:RTD interval: 6 7 8 9 10 …

Previous market run B0 A0 …

Current market run B1 A1 …

Next market run B2 A2 …

VER forecast error implementation:Hour ending:RTD interval: 6 7 8 9 10 …

Previous market run B0 A0 …

Current market run B1 A1 …

Next market run B2 A2 …

17

17

17


CAISO/DMM/KW 14

Figure 2.5 15-minute market histogram construction (BPM for Market Operations)11

VER forecast data is reported in 5-minute level increments. Therefore, for each 15-minute market run, there are three binding 5-minute forecasts, three first-advisory 5-minute forecasts, and so on. The average of the three 5-minute intervals reflects the 15-minute market VER forecast that wind and solar resources are dispatched to in the 15-minute market. However, the input 15-minute market VER forecast value used for the net load error calculation with the implementation of the flexible ramping product reflected instead the last 5-minute interval in an advisory 15-minute period. This ultimately biased the uncertainty requirements based on the direction of wind and solar ramp.

The ISO recalculated the net load error distributions to resolve this issue. Beginning March 31, 2018, the 15-minute market VER forecast used to calculate the uncertainty distributions were calculated from the average of the three 5-minute intervals rather than a single 5-minute interval.

Table 2.2 illustrates the implementation issues described in Section 2.1 through 2.4 for the 15-minute market uncertainty calculation with an example. The columns again show sequential 15-minute and 5-minute market intervals while the rows show sequential market runs. The blue cells represent the binding intervals while the orange cells reflect the first advisory intervals.

The net load error (uncertainty) assigned to interval 2 is expected to be equal to the maximum or minimum binding net load of three 5-minute intervals corresponding to the next 15-minute market run (b5 through b7) minus the first advisory net load of the current 15-minute market run (A1).

However, load error was implemented as the binding load of one of three 5-minute intervals corresponding to the current 15-minute market run (b2 through b4) minus the first advisory load of the current 15-minute market run (A1). In addition, the VER forecast error was pulled from the previous interval as the load forecast error and the 15-minute VER forecast was pulled from the last 5-minute interval in an advisory 15-minute period.

11 See Market Operations Business Practice Manual version 55, pp. 239-240:





CAISO/DMM/KW 15

Table 2.2 Example. Flexible ramping product uncertainty calculation implementation – interval 2 (15-minute market)

Expected error calculation (load and VER):15-minute market interval …

5-minute market interval 1 2 3 4 5 6 7 8 9 …Previous 15-minute market run …

Current 15-minute market run …

Next 15-minute market run …

Corresponding 5-minute market runs b1 a1 …

b2 a2 …

b3 a3 …

b4 a4 …

b5 a5 …

b6 a6 …

b7 a7

Load forecast error implementation:15-minute market interval …

5-minute market interval 1 2 3 4 5 6 7 8 9 …Previous 15-minute market run …

Current 15-minute market run …

Next 15-minute market run …


b2 a2 …

b3 a3 …

b4 a4 …

b5 a5 …

b6 a6 …

b7 a7

VER forecast error implementation:15-minute market interval …

5-minute market interval 1 2 3 4 5 6 7 8 9 …Previous 15-minute market run B0 B0 B0 A0 A0 A0 …

Current 15-minute market run B1 B1 B1 A1 A1 A1 …

Next 15-minute market run B2 B2 B2 …


b2 a2 …

b3 a3 …

b4 a4 …

b5 a5 …

b6 a6 …

b7 a7

…

B2

1 2 3

B0 A0 …

B1 A1

…

1 2 3

A0

B1

1

B0

B2

A1

2 3


CAISO/DMM/KW 16

2.5 Uncertainty thresholds used to cap uncertainty requirements

With the implementation of the flexible ramping product, the ISO implemented threshold values to cap the uncertainty requirements. The threshold values are different for each balancing area (and the system), 5-minute market versus 15-minute market, lower versus upper. However, the same threshold value is used across each hour as well as weekday versus weekend.

These values were created from the 98th percentile of a longer duration than the 40-day period that is used for the standard weekday uncertainty distribution. When the calculated uncertainty requirement (From the 2.5 and 97.5 percentile of observations) is greater than the threshold value, the threshold value is used instead. These values were expected to prevent extreme outlier or erroneous net load errors from impacting the uncertainty requirement and associated market outcomes.

However, these values have not been updated since the implementation of the flexible ramping product in November of 2016. When these threshold values are applied, the uncertainty requirement is therefore disconnected from the more recent hour-specific uncertainty requirement that would be expected to be more representative of real-time flexibility needs for that interval. Further, these threshold values are based on the erroneous uncertainty calculation described in Section 2.1 through 2.4. The ISO has indicated that they plan to update the thresholds once all of the uncertainty calculation issues have been addressed.

With the implementation of the flexible ramping product, these threshold values were expected to apply to a small percentage of hours. However, these threshold values have been binding in greater frequency depending on the BAA, market, direction and month.12

Table 2.3 contains the uncertainty thresholds that were implemented with the flexible ramping product in 2016 and are still currently used in the market. Portland General Electric were given the same threshold values as PacifiCorp West when they joined the energy imbalance market in 2017.

12 Since the fix to address the first issues described in Section 2.1, the uncertainty thresholds have been binding more

frequently overall, particularly in the 5-minute market for PacifiCorp West, PacifiCorp East, Puget Sound Energy, and Portland General Electric. The upper and lower 5-minute market uncertainty thresholds were binding in PacifiCorp West in around 80 percent of intervals each.


CAISO/DMM/KW 17

Table 2.3 Flexible ramping product implemented uncertainty thresholds

As mentioned earlier, the system level demand curve (and uncertainty requirement) is always enforced in the market while the demand curves for the individual areas only apply when insufficient transfer capability is present (often as a result of failing the flexible ramping sufficiency test). As a result, the majority of flexible ramping capacity is procured to meet the system level uncertainty requirement. Figure 2.6 shows the percent of hours in which the upper and lower thresholds were applied for the system uncertainty requirement by month in the last year. As shown in the figure, the downward threshold values shown in Table 2.3 were applied frequently in the past year. In particular, the 15-minute market lower threshold of -1,200 was applied in over 40 percent of hours between August and October, 2017.

In addition, the uncertainty thresholds are not documented in the Tariff, Business Practice Manual, or Business Requirements Specification. Given the frequency in which these thresholds set the uncertainty requirements rather than the hourly distributions, DMM has recommended that the process is revaluated and included in the Business Practice Manual.

Lower Threshold

Upper Threshold

System15-minute market -1200 18005-minute market -300 500

Arizona Public Service15-minute market -350 4005-minute market -100 100

California ISO15-minute market -1000 10005-minute market -250 300

NV Energy15-minute market -250 2505-minute market -40 40

PacifiCorp East15-minute market -300 300

5-minute market -50 50PacifiCorp West

15-minute market -175 1505-minute market -25 25

Puget Sound Energy15-minute market -135 1355-minute market -25 25


CAISO/DMM/KW 18

Figure 2.6 Frequency of applied uncertainty thresholds – system

2.6 ISO load error in the uncertainty distributions

New net load error distributions based on a binding and advisory net load forecast in the same time interval have been used in the market since February 22, 2018. Since this fix, system level and ISO specific 5-minute market uncertainty requirements have not accounted for any load error within the ISO. This is because the input ISO load data used in the market and in the uncertainty calculation does not contain differences between the binding interval in the next market run and first advisory interval in the current market run.

Load forecast data in the ISO is updated continuously for a binding interval and future advisory intervals, but is not automatically consumed by the market. Instead the real-time load forecast is manually pushed to the market periodically by ISO operators. When this occurs, the binding and first advisory load forecasts are not impacted while the second advisory interval and onwards are updated. As a result, the advisory ISO load forecast in the current market run is almost always equal to the binding load forecast in the next market run.

However, this issue has no direct impact on the frequency of power balance constraint relaxations as the uncertainty calculation accurately reflects ISO load uncertainty included in the market optimization. Effectively, ISO load forecast error in the next interval is always zero so additional flexible ramping capacity is not procured by the flexible ramping product to cover ISO load uncertainty.

The manual approach by ISO operators is taken to prevent potentially erroneous forecast data from automatically flowing into the market optimization and impacting market results. The ISO has indicated that they are looking into a more automated long-term solution with regards to pushing the load forecast to the market.

0%

10%

20%

30%

40%

50%

60%

Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb

2017 2018

Perc

ent o

f hou

rs a

pplie

d

Lower threshold applied (15-minute market)Upper threshold applied (15-minute market)Lower threshold applied (5-minute market)Upper threshold applied (5-minute market)


CAISO/DMM/KW 19

2.7 Other implementation issues

Both DMM and the ISO have identified and discussed other issues with the implementation of the flexible ramping products, as described below.

Since the implementation of the flexible ramping product, the demand curves for individual balancing areas are included in the constraint for system-level procurement. Initially, segments of relaxation capacity specific to the individual balancing area demand curves could be used to meet system-level uncertainty even when the uncertainty requirements for the individual balancing areas was reduced to zero. This approach resulted in system-level procurement of flexible ramping capacity and associated flexible ramping shadow prices that were lower than what would be consistent using the system-level demand curves alone.

On July 13, 2017, an adjustment was made to limit the use of flexible ramping product demand curves from individual balancing areas when sufficient transfer capability connected the area with system conditions. However, since this adjustment was made, resources providing flexible ramping capacity to meet system-level flexibility needs have often received lower payments based on the area-specific demand curve rather than the system-level demand curve though sufficient transfer capacity was present.13 A fix for the issue went into production effective April 4, 2018.

The ISO has also identified an issue related to the deliverability of flexible ramping product procurement. The concern being the potential for system-level flexible ramping capacity procurement external to the ISO to be stranded behind energy imbalance market transfer constraints when prices in the ISO and surrounding areas are extremely high and in need of flexible ramping capacity. The ISO discussed a proposed enhancement to resolve the issue at the Market Surveillance Committee meeting on February 2, 2018.14

13 For additional information on this pricing issue, see DMM’s Q3 2017 Report on Market Issues and Performance, December

2017, pp. 49-52: http://www.caiso.com/Documents/2017ThirdQuarterReport-MarketIssuesandPerformance-December2017.pdf.

14 Market Surveillance Committee Flexible Ramping Product Performance Discussion, February 2, 2018, slides 5-7: http://www.caiso.com/Documents/Presentation-FlexibleRampingProductPerformanceDiscussionFeb22018.pdf.



http://www.caiso.com/Documents/Presentation-FlexibleRampingProductPerformanceDiscussionFeb22018.pdf


CAISO/DMM/KW 21

3 Analysis of market impacts

This section provides an assessment of the impacts and implications of the implementation errors described in Section 2. This section includes a comparison of the uncertainty requirements resulting from the incorrect method used until recently with DMM’s estimate of the uncertainty requirements had the calculation been implemented correctly. This section also examines the impact of these errors on the frequency of power balance constraint relaxations as well as the impact on flexible ramping shadow prices and procurement.

3.1 Corrected uncertainty calculation

For this report, DMM re-calculated the uncertainty requirements using the correct methodology and data. DMM believe that these corrected uncertainty requirements are highly consistent with what the uncertainty requirements would have been had the flexible ramping product been implemented as designed. Thus, the difference between these corrected requirements and the requirements used by the ISO reflect the impact of the errors described in Section 2.

Figure 3.1 and Figure 3.2 show corrected average hourly uncertainty requirements had the uncertainty distributions been calculated as designed for the 5-minute market and 15-minute market, respectively. The corrected upward and downward uncertainty requirements in the figures are pulled from the 2.5th and 97.5th percentile of recalculated hourly distributions of net load error during 2017 after correcting the issues described in the previous section. The blue lines show corrected upward and downward system-level uncertainty requirements between March and December, 2017. For comparison, the green lines show average hourly system-level uncertainty requirements used in the market by the ISO during the same period.

In the 5-minute market, the impact of incorrectly calculating uncertainty based on the difference between sequential time intervals is significant, as shown in Figure 3.1. The incorrect calculation of uncertainty biased the requirements in the direction opposite of the net load ramp. During hours when the corrected uncertainty requirements are greater than the implemented uncertainty requirements, flexible ramping capacity procurement were expected to be higher.

As shown in Figure 3.1, upward uncertainty requirements in the 5-minute market were expected to be around 270 MW higher on average between hours ending 15 and 18. Downward uncertainty requirements were expected to be larger by around 120 MW on average during morning hours ending 8 through 12 when solar generation is ramping up. In other hours, the incorrect uncertainty calculation resulted in higher than expected uncertainty requirements -- which would tend to cause inefficiently higher ramping capacity procurement and prices.

As illustrated in Figure 3.2, the errors had a similar impact on the uncertainty requirements in the 15-minute market. As shown in Figure 3.2, if the flexible ramping product been implemented correctly the 15-minute market upward uncertainty requirements would have been around 460 MW higher on average between hours ending 15 and 19.


CAISO/DMM/KW 22

Figure 3.1 Average hourly 5-minute market system-level uncertainty requirements (March, 2017 – December, 2017)

Figure 3.2 Average hourly 15-minute market system-level uncertainty requirements (March, 2017 – December, 2017)

-600

-400

-200

0

200

400

600

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Syst

em-le

vel u

ncer

tain

ty re

quire

men

t (M

W)

Implemented uncertainty Corrected uncertainty Thresholds

-2,000

-1,500

-1,000

-500

0

500

1,000

1,500

2,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Syst

em-le

vel u

ncer

tain

ty re

quire

men

t (M

W)

Implemented uncertainty Corrected uncertainty Thresholds


CAISO/DMM/KW 23

3.2 Power balance constraint relaxation

One of the key objectives of the flexible ramping product is to address the challenges of maintaining power balance in real-time between supply and demand. In addition to procuring ramp capability for the forecasted net load ramp, the flexible ramping product allows the market to account and procure for uncertainty surrounding this forecasted value that could otherwise result in an infeasibility.

When insufficient incremental or decremental energy is available for real-time dispatch to solve the market solution, the power balance constraint is relaxed and energy prices are set inefficiently at an administrative penalty parameter rather than an economic bid.

The flexible ramping product procures additional ramping capability to cover forecast error that may materialize when it is economic to do so. The demand curves define the trade-off between the cost of procuring additional flexible ramping capacity and the expected reduction in power balance constraint violation costs with the additional ramping capacity.

As designed, the upward and downward demand curves are based on a distribution of net load errors in a 95 percent confidence interval. Therefore, when the full amount of the upward and downward uncertainty requirements are procured in flexible ramping capacity for a given interval, the majority of potential net loads in the advisory interval are expected to become feasible based on the historical data. However, because of the issues in the uncertainty calculation with the implementation of the flexible ramping product, the demand curves were disconnected from actual uncertainty needs and this expectation no longer held.

Figure 3.3 shows the hourly frequency of power balance constraint relaxations as a result of insufficient incremental energy during 2017 in the 5-minute market. The figures also include the average implemented and corrected upper uncertainty requirements from the previous section, bounded by zero and the uncertainty thresholds. Under-supply infeasibilities in the 5-minute market were relatively frequent between hours ending 15 and 19 when load net of wind and solar is typically ramping up. As shown in the figure, these hours typically had lower than expected upward uncertainty requirements as a result of the implementation error in the uncertainty calculation.

Figure 3.4 shows the same information for power balance constraint relaxation due to insufficient decremental energy, or oversupply, in the 5-minute market. Procurement of downward flexible capacity with the flexible ramping product is intended to better position resources to respond to net load forecasts that are too high and reduce instances of excess generation. Instances of excess generation between hours ending 8 and 12 occurred during periods when downward uncertainty requirements were expected to be larger.

Figure 3.5 shows the same information for under-supply infeasibilities in the 15-minute market. Under-supply infeasibilities in this market during 2017 were concentrated between hours ending 18 and 20. In hours ending 18 and 19, upward uncertainty requirements were expected to be higher. At the other end, over-supply infeasibilities did not occur in the 15-minute market during 2017.

The feasibility of these solutions for both under-supply and overs-supply cannot be determined had the demand curves been implemented correctly based on actual uncertainty needs. For instance, the market may have forgone the capacity procurement at a positive shadow price. Alternatively, the additional procured flexible ramping capacity may have been stranded behind energy imbalance market transfer constraints as a result of the issue discussed in Section 2.7. However, it is likely that that


CAISO/DMM/KW 24

systematic under-procurement of flexible ramping capacity during key upward and downward net load ramping hours increased the frequency of power balance violations.

Figure 3.3 Frequency of 5-minute market under-supply power balance constraint relaxation

Figure 3.4 Frequency of 5-minute market over-supply power balance constraint relaxation

0

100

200

300

400

500

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Syst

em-le

vel u

pwar

d un

cert

aint

y re

quire

men

t (M

W)

Pow

er b

alan

ce c

onst

rain

t rel

axat

ion

(per

cent

of 5

-min

ute

inte

rval

s)

Under-supply infeasibility (shortage)Implemented upper uncertainty requirementCorrected upper uncertainty requirement

-300

-200

-100

0

0.0%

0.5%

1.0%

1.5%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Syst

em-le

vel l

ower

unc

erta

inty

requ

irem

ent

(MW

)

Pow

er b

alan

ce c

onst

rain

t rel

axat

ion

(per

cent

of 5

-min

ute

inte

rval

s)

Over-supply infeasibility (excess)Implemented lower uncertainty requirementCorrected lower uncertainty requirement


CAISO/DMM/KW 25

Figure 3.5 Frequency of 15-minute market under-supply power balance constraint relaxation

3.3 Impact on flexible ramping product prices and procurement

Figure 3.6 shows the percent of intervals between March and December, 2017, when the system-level flexible ramping demand curve bound, and had a positive shadow price. In the 15-minute market, the frequency of positive prices coincided with higher levels of demand for upward and downward ramping capacity, per the implemented uncertainty calculation. In the 5-minute market, system-level flexible ramping prices were non-zero much less frequently than in the 15-minute market, during less than 1 percent of intervals.

Figure 3.7 and Figure 3.8 provide an hourly overview of the impact of the incorrect uncertainty calculation on 15-minute market system-level flexible ramping shadow prices between March and December, 2017.15 These estimates of market impacts were determined by reviewing historical uncertainty requirements, procurement, and shadow price outcomes relative to the expected uncertainty requirement.16

15 Corresponding charts for the 5-minute market were not included as the impact on 5-minute market shadow prices is

estimated to be very low. However, the impact on 5-minute market procurement can be significant as shown in Table 4.4 and Table 4.5.

16 This analysis assumes the same uncertainty thresholds were active. If both the implemented and corrected uncertainty requirements were capped by an uncertainty threshold, no impact is therefore expected to occur.

0

400

800

1,200

1,600

2,000

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Syst

em-le

vel u

pwar

d un

cert

aint

y re

quire

men

t (M

W)

Pow

er b

alan

ce c

onst

rain

t rel

axat

ion

(per

cent

of 5

-min

ute

inte

rval

s)Under-supply infeasibility (shortage)Implemented upper uncertainty requirementCorrected upper uncertainty requirement


CAISO/DMM/KW 26

Figure 3.6 Hourly frequency of historical non-zero flexible ramping shadow price (March – December, 2017)

For instance, if the system-level upward ramping demand curve was historically binding at a non-zero shadow price, then a likely impact can be determined depending on whether the expected uncertainty requirement should have been higher or lower. In this scenario, if the expected uncertainty requirement was greater than the implemented uncertainty requirement, then the demand for upward flexible ramping capacity shifts to the right and therefore the price and/or procurement was expected to be higher, or likely decreased prices and/or procurement as a result of the implementation error.

Alternatively, if the expected uncertainty requirement was greater than both the implemented uncertainty requirement and procurement, but the historical shadow price was zero, then the shadow price may have been non-zero with the higher expected requirement, or possible decreased prices due to the incorrect calculation.

0%

20%

40%

60%

80%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Freq

uenc

y of

pos

itive

sys

tem

pric

e

Upward ramping 15-minute marketDownward ramping 15-minute marketUpward ramping 5-minute marketDownward ramping 5-minute market


CAISO/DMM/KW 27

Figure 3.7 Impact of incorrect uncertainty calculation on system-level upward ramping shadow prices (15-minute market, March – December, 2017)

Figure 3.8 Impact of incorrect uncertainty calculation on system-level downward ramping shadow prices (15-minute market, March – December, 2017)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Perc

ent o

f 15-

min

ute

inte

rval

s

Likely decreased shadow prices Likely increased shadow prices

Possible decreased shadow prices Possible increased shadow prices

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Perc

ent o

f 15-

min

ute

inte

rval

s

Likely decreased shadow prices Likely increased shadow prices

Possible decreased shadow prices Possible increased shadow prices


CAISO/DMM/KW 28

Table 3.1 and Table 3.2 provide a summary of upward and downward ramping impacts in the 15-minute market by the direction of the impact on shadow prices and/or procurement. The tables includes average requirement, procurement, and shadow price amounts when the impact conditions exist. For a more detailed break-down of the scenarios as well as 5-minute market impacts, see Appendix Section 4.2.

Results of this analysis shows for upward ramping in the 15-minute market that:

• It is highly likely that the errors decreased upward ramping shadow prices and/ or procurement during about 5 percent of 15-minute intervals.

• During another 45 percent of 15-minute intervals, it is possible that the errors decreased upward ramping shadow prices and likely decreased upward ramping procurement.

• During the 50 percent of intervals when upward ramping procurement was lower due to the errors, the correct requirements averaged almost 400 MW greater than historical procurement on average (i.e. 949 MW compared to 564 MW procured).

• During 15 percent of 15-minute intervals, analysis shows that the errors highly likely increased upward ramping prices and/ or procurement. In these instances, expected requirements were more than 600 MW less than historical procurement on average.

• During 31 percent of 15-minute intervals, DMM estimates a likely possible positive impact on upward ramping procurement, but not on corresponding shadow prices. In these instances, expected requirements were almost 400 MW less than historical procurement on average.

Appendix Section 4.2 provides a full list of scenarios and more detailed results of this analysis.

Table 3.1 15-minute market upward ramping impact frequency and averages (March – December, 2017)

Table 3.2 15-minute market downward ramping impact frequency and averages (March – December, 2017)

Corrected (DMM)Percent of intervals

Average expected requirement

Average implemented requirement

Average procurement

Average upward ramping shadow price

Likely/possible decreased prices/procurement 50% 949 541 564 $1.36Likely/possible increased prices/procurement 15% 614 1,274 1,254 $10.89Likley increased procurement only 31% 585 959 953 $0No estimated impact 4% 608 613 832 $2.23

HistoricalImpact of implementation error on shadow

prices and/or procurement

Corrected (DMM)Percent of intervals


Average implemented requirement

Average procurement


Likely/possible decreased prices/procurement 37% 907 449 600 $0.22Likely/possible increased prices/procurement 2% 726 1,034 684 $6.59Likley increased procurement only 39% 696 1,051 1,042 $0No estimated impact 22% 782 728 1,158 $0.09

HistoricalImpact of implementation error on shadow

prices and/or procurement


CAISO/DMM/KW 29

3.4 Sufficiency test

The 15-minute market uncertainty requirement is also used in the flexible ramping sufficiency test. In order to pass the hourly flexible ramping sufficiency test in a given direction (upward or downward), the balancing area needs to show sufficient ramping capability from the start of the hour to each of the four 15-minute intervals in the hour. Failing the sufficiency test and limiting transfers as a result can impact the frequency of power balance constraint relaxation across balancing areas.

The requirement for the flexible ramping sufficiency test is calculated as the forecasted change in load plus the uncertainty minus two discounts, diversity benefit and flexible ramping credits. The uncertainty value used in the flexible ramping sufficiency test is the same as the hourly 15-minute market uncertainty requirement for each balancing area. As a result, the implementation of the net load error distributions as the difference between two consecutive time-intervals also had an impact on the total requirements for the sufficiency test.

Further, the diversity benefit that is discounted from the total requirement is driven by the uncertainty requirements. The diversity benefit reflects that system-level flexible ramping needs are typically smaller than the sum of the individual balancing area flexible ramping needs because of reduced uncertainty across a larger footprint. As a result, balancing areas receive a prorated diversity benefit discount based on this proportion. The diversity benefit is equal to the system uncertainty requirement divided by the sum of all of the individual balancing area uncertainty requirements.

Combined, upward and downward flexible ramping sufficiency test requirements were impacted by the incorrect uncertainty requirements. However as shown in Section 4.1, the impact of the implementation issue on the 15-minute market uncertainty requirements was smaller than in the 5-minute market for the individual balancing areas. Charts showing the impact of the calculation error on the uncertainty requirements for each of the individual balancing areas are included in the Appendix in Section 4.1. Generally, the net load errors at the 2.5th and 97.5th percentile of observations in the implemented and corrected 15-minute market distributions were more comparable than in the 5-minute market.

To the extent that the uncertainty requirements were significantly different for some hours and balancing areas, the impact of the uncertainty calculation error on the failure rate of the balancing area’s sufficiency tests cannot be determined. During hours when the sufficiency test requirements were expected to be higher, the energy imbalance market entity would likely provide more ramping capacity when feasible to meet the anticipated requirement.


CAISO/DMM/KW 31

4 Appendix

4.1 Implemented versus corrected balancing area uncertainty requirements

This sections shows average hourly implemented versus corrected uncertainty requirements for each of the individual areas between March 2017 and December 2017. The corrected uncertainty requirements reflect expected outcomes had the flexible ramping product been implemented as designed. These results were recalculated from net load error observations during all of 2017 with the issues in Section 2 resolved. The upward and downward uncertainty requirements are pulled from the 2.5th and 97.5th percentile of hourly net load error distributions. Weekday distributions use data for the same hour from the last 40 weekdays. For weekends, the last 20 weekend days are used instead. For comparison, the green lines show average hourly uncertainty requirements that were used in the market.

The upward uncertainty requirements are depicted by the upper lines while the downward uncertainty requirements are depicted by the lower lines. The hourly uncertainty requirements are capped at zero megawatts at one end and at the uncertainty thresholds at the other.


CAISO/DMM/KW 32

Figure 4.1 5-minute market Arizona Public Service uncertainty requirements (March, 2017 – December, 2017)

Figure 4.2 15-minute market Arizona Public Service uncertainty requirements (March, 2017 – December, 2017)

-150

-100

-50

0

50

100

150

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)

Implemented uncertainty Counterfactual uncertainty Thresholds

-400

-300

-200

-100

0

100

200

300

400

500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)



CAISO/DMM/KW 33

Figure 4.3 5-minute market California ISO uncertainty requirements (March, 2017 – December, 2017)

Figure 4.4 15-minute market California ISO uncertainty requirements (March, 2017 – December, 2017)

-500

-400

-300

-200

-100

0

100

200

300

400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)


-2,000

-1,500

-1,000

-500

0

500

1,000

1,500

2,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)



CAISO/DMM/KW 34

Figure 4.5 5-minute market NV Energy uncertainty requirements (March, 2017 – December, 2017)

Figure 4.6 15-minute market NV Energy uncertainty requirements (March, 2017 – December, 2017)

-50

-40

-30

-20

-10

0

10

20

30

40

50

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)


-300

-200

-100

0

100

200

300

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)



CAISO/DMM/KW 35

Figure 4.7 5-minute market PacifiCorp East uncertainty requirements (March, 2017 – December, 2017)

Figure 4.8 15-minute market PacifiCorp East uncertainty requirements (March, 2017 – December, 2017)

-80

-60

-40

-20

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)


-400

-300

-200

-100

0

100

200

300

400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)



CAISO/DMM/KW 36

Figure 4.9 5-minute market PacifiCorp West uncertainty requirements (March, 2017 – December, 2017)

Figure 4.10 15-minute market PacifiCorp West uncertainty requirements (March, 2017 – December, 2017)

-50

-40

-30

-20

-10

0

10

20

30

40

50

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)


-200

-150

-100

-50

0

50

100

150

200

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)



CAISO/DMM/KW 37

Figure 4.11 5-minute market Puget Sound Energy uncertainty requirements (March, 2017 – December, 2017)

Figure 4.12 15-minute market Puget Sound Energy uncertainty requirements (March, 2017 – December, 2017)

-60

-40

-20

0

20

40

60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)


-200

-150

-100

-50

0

50

100

150

200

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Unc

erta

inty

requ

irem

ent (

MW

)



CAISO/DMM/KW 38

4.2 Shadow price impact analysis scenarios

The shadow price impact results used in Figure 3.7 and Figure 3.8 were determined by reviewing historical uncertainty requirements, procurement, and shadow prices relative to the expected uncertainty requirement. DMM identified nine potential scenarios to review based on the relative outcome of these variables. The scenarios for upward ramping are included in Table 4.1. The scenarios are the same for downward ramping except with $152 instead of $247 as the downward ramping shadow price cap. The second to last column describes the impact of the implementation error on flexible ramping shadow prices. Flexible ramping capacity procurement is also impacted by the specified direction except for scenario 5, which does not have an estimated impact of shadow prices but has a likely positive impact on procurement.

Table 4.2 through Table 4.5 provide a summary of upward and downward ramping impacts in the 15-minute market and 5-minute market by the scenario number from Table 4.1. The tables includes average requirement, procurement, and shadow price amounts when the specified conditions exist.


CAISO/DMM/KW 39

Table 4.1 Flexible ramping product shadow price and procurement impact analysis scenarios

Scenario # Criteria DescriptionImpact of implementation

error on shadow pricesImpact of implementation

error on procurement1 (1) Expected requirement > implemented requirement

(2) $0 < Shadow price < $247With higher expected requirement, flex ramp demand shifts right. Shadow price was binding before at a positive shadow price — price/procurement expected to be higher.

Likely decreased prices Likely decreased procurement

2 (1) Expected requirement > implemented requirement (2) Implemented requirement ≥ procurement (3) Shadow price = $0

With higher expected requirement, flex ramp demand shifts right. Shadow price may become binding at a positive shadow price. Procurement expected to be higher.

Possible decreased prices Likely decreased procurement

3 (1) Expected requirement > procurement (2) Procurement > implemented requirement

Similar to scenario 2. Shadow price may become binding at a positive shadow price. Procurement expected to be higher.

Possible decreased prices Likely decreased procurement

4 (1) Implemented requirement > expected requirement (2) $0 < Shadow price < $247

With lower expected requirement, flex ramp demand shifts left. Shadow price was binding before at a positive shadow price — price/procurement expected to be lower.

Likely increased prices Likely increased procurement

5 (1) Implemented requirement > expected requirement (2) Shadow price = $0

With lower expected requirement, flex ramp demand shifts left, but shadow price already at zero. Procurement expected to be lower.

No estimated impact Likely increased procurement

6 (1) Implemented requirement > expected requirement (2) Shadow price = $247

With lower expected requirement, flex ramp demand shifts left. Shadow price may have been less than maximum of $247. Procurement may have been lower.

Possible increased prices Possible increased procurement

7 (1) Expected requirement > implemented requirement (2) Shadow price = $247

With higher expected requirement, flex ramp demand shifts right, but shadow price already at maximum.

No estimated impact No estimated impact

8 (1) Procurement > expected requirement (2) Procurement > implemented requirement

Flexible ramping capacity was readily available such that procurement would have been higher than the requirement regardless.


9 (1) Expected requirement = implemented requirement Whether by the thresholds or coincidence, the requirements were the same.



CAISO/DMM/KW 40

Table 4.2 15-minute market upward ramping frequency and averages by scenario (March – December, 2017)

Table 4.3 15-minute market downward ramping frequency and averages by scenario (March – December, 2017)

Corrected (DMM)

Scenario # Percent of intervals Shadow prices Procurement


Average implemented requirement Average procurement


1 5% Likely decreased prices Likely decreased procurement 1,151 870 838 $13.122 36% Possible decreased prices Likely decreased procurement 957 577 573 $0.003 9% Possible decreased prices Likely decreased procurement 806 224 379 $04 15% Likely increased prices Likely increased procurement 614 1,275 1,255 $10.675 31% No estimated impact Likely increased procurement 585 959 953 $06 0.01% Possible increased prices Possible increased procurement 780 923 249 $2477 0.01% No estimated impact No estimated impact 727 631 38 $2478 4% No estimated impact No estimated impact 472 478 730 $09 0.4% No estimated impact No estimated impact 1,763 1,763 1,728 $13.14

HistoricalImpact of implementation error on …

Corrected (DMM)




Average downward ramping shadow price

1 1% Likely decreased prices Likely decreased procurement 1,131 991 664 $6.742 14% Possible decreased prices Likely decreased procurement 957 715 700 $0.003 21% Possible decreased prices Likely decreased procurement 860 238 528 $04 2% Likely increased prices Likely increased procurement 726 1,034 684 $6.595 39% No estimated impact Likely increased procurement 696 1,051 1,042 $06 0% Possible increased prices Possible increased procurement N/A N/A N/A N/A7 0% No estimated impact No estimated impact N/A N/A N/A N/A8 20% No estimated impact No estimated impact 744 684 1,162 $09 2% No estimated impact No estimated impact 1,196 1,196 1,113 $1.08

HistoricalImpact of implementation error on …


CAISO/DMM/KW 41

Table 4.4 5-minute market upward ramping frequency and averages by scenario (March – December, 2017)

Table 4.5 5-minute market downward ramping frequency and averages by scenario (March – December, 2017)

Corrected (DMM)





1 0.1% Likely decreased prices Likely decreased procurement 213 98 62 $40.012 12% Possible decreased prices Likely decreased procurement 214 115 114 $03 30% Possible decreased prices Likely decreased procurement 219 26 105 $04 0.4% Likely increased prices Likely increased procurement 139 355 339 $26.315 37% No estimated impact Likely increased procurement 144 314 313 $06 0.01% Possible increased prices Possible increased procurement 149 360 118 $2477 0% No estimated impact No estimated impact N/A N/A N/A N/A8 21% No estimated impact No estimated impact 140 102 248 $09 0.2% No estimated impact No estimated impact 143 143 143 $0

Impact of implementation error on … Historical

Corrected (DMM)





1 0% Likely decreased prices Likely decreased procurement N/A N/A N/A N/A2 9% Possible decreased prices Likely decreased procurement 244 129 128 $0.003 20% Possible decreased prices Likely decreased procurement 184 35 111 $04 0.02% Likely increased prices Likely increased procurement 152 288 223 $5.505 31% No estimated impact Likely increased procurement 152 268 267 $06 0% Possible increased prices Possible increased procurement N/A N/A N/A N/A7 0% No estimated impact No estimated impact N/A N/A N/A N/A8 40% No estimated impact No estimated impact 131 122 359 $09 0.03% No estimated impact No estimated impact 110 110 110 $0

Impact of implementation error on … Historical


CAISO/DMM/KW 42

4.3 Supplemental analysis of implemented uncertainty and demand curves

Frequency of zero requirements

The calculation of net load error with the implementation of the flexible ramping product has resulted in periods of low or zero megawatt uncertainty requirements that would not have existed otherwise. In particular, very low or zero megawatt requirements occurred frequently in the upward direction during hours when net load is ramping up and in the downward direction when net load is ramping down.

During most of these periods, the market still procured a positive quantity of flexible ramping capacity. The flexible ramping product is co-optimized with both energy and ancillary services. This capacity was therefore readily available such that there was no opportunity cost associated with providing the flexibility in lieu of energy or ancillary services. This also indicates that the payments for such capacity would be zero as there was no calculated demand for it.

The amount of capacity procured in these cases is likely to have been less than would have been procured had demand curves been calculated as designed and therefore below expected uncertainty needs. Further, the demand curves would have been more likely to bind at a non-zero shadow price had the uncertainty requirements been higher.

Figure 4.13 shows the percent of hours during 2017 in which the system-level uncertainty requirement – or the maximum demand for flexible ramping capacity – was zero megawatts in the 5-minute market. Uncertainty requirements in the 15-minute market were zero megawatts much less frequently, but were often lower than expected during the same hours that impacted the 5-minute market. In particular, 5-minute market upward uncertainty requirements were frequently zero megawatts during morning hours, 5 and 6, as well as peak-load ramping hours, 15 through 19. 5-minute market downward uncertainty requirements were always zero megawatts in 2017 during hours 22 through hour 2.


CAISO/DMM/KW 43

Figure 4.13 Frequency of zero megawatt system uncertainty requirements (5-minute market)

Implemented flexible ramping product demand curves

Flexible ramping product procurement and prices are driven from the demand curves, where the uncertainty requirements are the upper end of the demand curve, or the maximum demand for flexible ramping capacity. The demand curves allow the market optimization to consider the trade-off between the cost of procuring additional flexible ramping capacity and the expected reduction in power balance constraint relaxation costs with the additional ramping capacity.17 Generation capacity that satisfies the demand for flexible ramping capacity then receives payments based on the flexible ramping shadow price derived from this trade-off.

Figure 4.14 shows average system-level flexible ramping demand in the 15-minute market during 2017 at three price points.18 The positive bars show demand for upward flexible ramping capacity, and the negative bars show demand for downward ramping capacity. The demand at $0 reflects the uncertainty requirement, or the maximum demand for upward and downward capacity when it is free. For example, in hour ending 8, the ISO demanded around 1,500 MW of upward capacity if the price was $0/MWh, but around 1,000 MW if the price was $100. The willingness-to-pay for upward capacity is typically higher

17 The ISO uses the $1,000/MWh penalty price for power balance shortages to calculate the upward demand curve and the -

$155/MWh penalty price for power balance excesses to calculate the downward demand curve. The probability of a power balance constraint relaxation is calculated using the historical net load forecast error data. The demand curves are capped such that the price cannot exceed $247/MWh in the upward direction and -$152/MWh in the downward direction.

18 The underlying demand curves can have up to nine segments, and the prices and quantities for those steps will differ across hours, markets, and areas.

0

50

100

150

200

250

300

350

400

450

500

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Aver

age

proc

urem

ent (

MW

)

Perc

ent o

f hou

rs

Zero requirement (downward) Zero requirement (upward)Average procurment (downward) Average procurement (upward)


CAISO/DMM/KW 44

than for downward capacity because of the higher cost of a power balance shortage relative to a power balance excess.

The figure shows that the willingness-to pay for system-level upward flexible capacity is low during the net load ramping hours, regardless of whether or not the uncertainty requirement is non-zero. When the number of positive net load error observations in the upward distribution is small, the expected cost of a power balance constraint relaxation becomes also low such that the uncertainty needs can be foregone at a relatively low cost. The reverse is also true with a small number of negative observations for downward demand.

Figure 4.14 Hourly average system-level flexible ramping demand curves during 2017 (15-minute market)

-2,000

-1,500

-1,000

-500

0

500

1,000

1,500

2,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24←

Dow

nwar

d ca

paci

ty

Upw

ard

capa

city

→

Dem

and

(MW

)

Demand at $100 Demand at $50 Demand at $0


CAISO/DMM/KW 45

4.4 ISO Tariff

The flexible ramping product and uncertainty requirement are described in Section 44 of the ISO tariff as follows:19

44.1 In General.

The CAISO may enforce flexible ramping constraints in the Real-time Market to meet Forecasted Movement and Uncertainty Requirements, using tools as further described in the Business Practice Manual that estimate the Demand Forecast and Supply forecast error, as set forth in this Section 44.

44.2.4.1 Requirement.

The CAISO will determine the Uncertainty Requirement for each Real-Time Market run, by each BAA and for the EIM Area overall.

44.2.4.2 Procurement Curve.

(a) Generally. Based on statistical analysis of the Uncertainty Requirement, the CAISO will calculate constraint relaxation parameters to ensure the total cost of the Uncertainty Awards will not exceed the cost of expected power balance violations in absence of the Uncertainty Award, by each Balancing Authority Area and for the EIM Area overall, as set forth in the Business Practice Manual.

4.5 Technical details in the Business Practice Manual

The flexible ramping product and uncertainty requirement are described in the Market Operations Business Practice Manual as follows:20

7.1.3 Flexible Ramping Product

Flexible Ramping Product (FRP) is a market-based product, it was developed to address the operational challenges of maintaining power balance in the real-time dispatch. FRP is composed of Flexible Ramping Up (FRU) and Flexible Ramping Down (FRD) capacities. This product is only procured in the Real-Time Unit Commitment (RTUC), which includes the Fifteen Minute Market, and Real-Time Dispatch (RTD) market. The FRP is resource ramping capacity that is reserved from scheduling or dispatch in the current market to address uncertainty that may materialize in real time. The purpose of FRP is to insure against insufficient ramp capability in real time that may result in extreme prices.

19 See Section 44 – Flexible Ramping Product as of Nov 1, 2016:

http://www.caiso.com/Documents/Section44_FlexibleRampingProduct_asof_Nov1_2016.pdf. 20 See Market Operations Business Practice Manual version 55, pp. 234-246:


http://www.caiso.com/Documents/Section44_FlexibleRampingProduct_asof_Nov1_2016.pdf




CAISO/DMM/KW 46

7.1.3.1.3 Demand Curve to Meet Uncertainty

The ISO will procure additional flexible ramping capability using a surplus demand curve to meet the upper (upward) and lower (downward) flexible ramping uncertainty requirements. The amount of surplus flexible ramping procured in both the FMM and RTD markets is dependent on (1) the incremental amount of upward and downward flexible ramp needed to account for net demand forecast uncertainty of the next market run’s binding interval (i.e. uncertainty requirement), and (2) the market price of supplying the additional flexible ramping capability to the markets.

Incremental upward and downward uncertainty requirements are provided via a 30-day histogram that tracks the net forecast error for each hour of the day. These requirements are posted the day prior. Note that the market will not require flexible ramping capacity for a given direction (up or down) when the net demand forecast movement exceeds the uncertainty requirement in the opposite direction.

For example, Figure 1(a) shows the net demand forecast movement is 200 MW in the upward direction. The upper (or upward) uncertainty requirement is 50 MW and shall be considered by the upward surplus demand curve. The lower (or downward) uncertainty requirement for that interval is 220 MW. In this case, only the amount of lower uncertainty in excess of the forecast movement will be considered for the downward surplus demand curve (which is 20 MW). If the downward uncertainty were below 200 MW for that interval, downward flexible ramping capability would not be required.

The flexible ramping surplus demand curve will establish the price of not fulfilling the flexible ramping requirement for a given hour over a trade date by applying the flexible ramping cap price to the probability density function from the uncertainty histogram. This allows the market energy price to determine whether all or some of the upward and downward uncertainty requirements are met. In other words, if the energy price is lower than the expected cost of not meeting the flexible ramping uncertainty requirement, more FRP will be procured to cover the ramping requirement uncertainty. Conversely, if the market energy price is above the expected cost, then no additional FRP will be procured to cover the ramping uncertainty requirement.

Figure 2 below shows how the final construction of upward and downward flexible ramping surplus demand curves. Both an upward and downward curve is provided separately for each of the FMM and RTD markets.


CAISO/DMM/KW 47

Where,

FRDS = Flexible ramp down surplus capability FRDP = Administrative flexible ramp down price floor (FRDP = –$247/MWh) FRDR = Total flexible ramp down requirement (non-positive) FRUS = Flexible ramp up surplus capability. FRUP = Administrative flexible ramp up price ceiling (FRUP = $247/MWh) FRUR = Total flexible ramp up requirement XD1 = Lowest net forecast error for negative histogram bin j XU1 = Highest net forecast error for positive histogram bin i

The ISO will construct histograms as an approximation of the probability distribution of net demand forecast errors to be used to procure for uncertainty. It will construct separate histograms for FRU and FRD for each hour, separately for RTD and FMM. The histogram for RTD will be constructed by comparing the net demand for the first advisory RTD interval to the net load in the same time interval for the next financially binding RTD run. Figure 3 shows two consecutive RTD 5-minute market runs, RTD1 and RTD2. The ISO will construct the histograms by subtracting the net demand from the first market run used for the first advisory interval (A1) from the net demand the second market run used for the binding interval (B₂).


CAISO/DMM/KW 48

For FMM, the ISO will construct separate histograms for FRU and FRD.

• For FRU, the histograms will be constructed based on the difference of the net demand the market used in the FMM for the first advisory RTUC interval and the maximum net demand the market used for the three corresponding RTD intervals.

• For FRD, the histograms will be constructed based on the difference of the net demand the market used in the FMM for the first advisory RTUC interval and the minimum net demand the market used for the three corresponding RTD intervals. Figure 4 below shows two RTUC intervals: the FMM (i.e. the RTUC binding interval) and the first advisory interval (labeled “A”). It illustrates how the FRU histogram will be constructed by comparing the net demand the FMM used for first advisory RTUC interval to the maximum net demand the market used for the corresponding three RTD binding intervals (b1, b₂, b₃).

The FRU histogram will use the observation b3 – A. This represents the maximum ramping need. The variable b₃, represents the maximum net load in the three RTD intervals. The FRD histogram will use observation b1 – A as this is the minimum ramping need. Ultimately in this example, the FRD


CAISO/DMM/KW 49

observation is positive and therefore will not be used directly in the demand curve creation. It will however be used to calculate the 95th percentile load forecast error and therefore needs to be captured in the histogram. The ISO shall use a rolling 30 day average, with a separate histogram for weekends and holidays, to evaluate the historical advisory RTUC imbalance energy requirement error pattern for each RTUC hour. The ISO will also evaluate if hours with similar ramping patterns could be combined to increase the sample size used in the historical analysis. The ISO expects that the estimate of uncertainty will improve over time. Therefore, the actual method of calculating the demand curve will be included in the business practice manual versus including these details in the tariff.

California Independent System Operator Corporation

Documents

Transcript of California Independent System Operator Corporation