Evaluation of Models-3 CMAQ I. Results from the 2003 Release II. Plans for the 2004 Release Model...

Evaluation of Models-3 CMAQEvaluation of Models-3 CMAQ

I. Results from the 2003 ReleaseI. Results from the 2003 ReleaseII. Plans for the 2004 Release II. Plans for the 2004 Release

Model Evaluation Team MembersModel Evaluation Team Members

Prakash Bhave,Prakash Bhave, Robin Dennis,Robin Dennis, Brian EderBrian Eder, Peter Finkelstein, Rob Gilliam, , Peter Finkelstein, Rob Gilliam, Alice Gilliland, Steve Howard, John Irwin, Daiwen Kang, George Pouliot, Alice Gilliland, Steve Howard, John Irwin, Daiwen Kang, George Pouliot,

Alfreida Rankin, Shawn Roselle, Jenise Swall, Shaocai YuAlfreida Rankin, Shawn Roselle, Jenise Swall, Shaocai Yu

Atmospheric Modeling DivisionAtmospheric Modeling DivisionNational Exposure Research LaboratoryNational Exposure Research LaboratoryU.S. Environmental Protection AgencyU.S. Environmental Protection Agency

Atmospheric Sciences Modeling DivisionAtmospheric Sciences Modeling DivisionAir Resources LaboratoryAir Resources Laboratory

National Oceanic and Atmospheric AdministrationNational Oceanic and Atmospheric Administration

Research Triangle Park, NC 27711Research Triangle Park, NC 27711

Part I. Results of the 2003 Release EvaluationPart I. Results of the 2003 Release Evaluation

The 2003 Release of CMAQ featured numerous changes:

- updated science - corrected implementations - efficiency enhancements - bug fixes

Largest impacts involved the aerosol modeling, especially:

- aerosol nitrate modeling updated with more recent science

Changes resulted in substantially lower concentrations of aerosol nitrate bringing them more in line with observations.

In order to characterize the impact of these changes and its overall performance, CMAQ was evaluated over two simulation periods:

- Summer 1999 (four weeks)- Winter 2002 (six weeks)

- Against observational data from four networks:

- IMPROVE - CASTNet- STN (only winter 2002)- SEARCH

- Against modeled data from the 2002 Release.


CMAQ Simulation AttributesCMAQ Simulation Attributes

Emissions:Emissions: Gas-phase emissions are based on the 1999 EPA NEI. Gas-phase emissions are based on the 1999 EPA NEI.

Chemistry:Chemistry: SAPRC99 gas-phase chemistry mechanism.SAPRC99 gas-phase chemistry mechanism.

Meteorology:Meteorology: Derived from MM5 and processed through MCIP (2.2). Derived from MM5 and processed through MCIP (2.2).

Resolution:Resolution: Horizontal:Horizontal: Contiguous U.S. Domain.Contiguous U.S. Domain.

32 km by 32 km grids(Summer)32 km by 32 km grids(Summer)

36 km by 36 km grids (Winter)36 km by 36 km grids (Winter)

Vertical:Vertical: Set on a sigma coordinate.Set on a sigma coordinate.

21 layers (Summer)21 layers (Summer)24 layers (Winter)24 layers (Winter)



Species: SO4, NO3, NH4, PM2.5, OC, EC

Statistics: Summary statistics and plots

Mean Bias (MB)

Normalized Mean Bias (NMB)

Root Mean Square Error (RMSE)

Normalized Mean Error (NME)

Results: Full evaluation available on AMD’s website:Full evaluation available on AMD’s website:

www.epa.gov/asmdnerl/models3www.epa.gov/asmdnerl/models3

Sulfate Sulfate ((μμg mg m-3-3)) June 15 – July 16, 1999June 15 – July 16, 1999

CASTNet IMPROVE

n 264 n 490

R 0.90 R 0.90

MB 1.74 MB 0.75

NMB(%) 38.0 NMB(%) 38.4

RMSE 2.89 RMSE 2.37

NME(%) 46.4 NME (%) 61.9

0

5

10

15

20

25

30

0 5 10 15 20 25 30

SO4-East (CASTNet)SO4-West (CASTNet)SO4-east (IMPROVE)SO4-west (IMPROVE)SO4-east (Search)

Mo

del

Observation

SO4

2- ( g m-3)

NME

NMB


0

5

10

0 5 10

SO4-East (CASTNet)SO4-West (CASTNet)SO4-east (IMPROVE)SO4-west (IMPROVE)SO4-east (STN)SO4-west (STN)

Mo

del

Observation

SO4

2- ( g m-3)

CASTNet IMPROVE STN

n 407 n 728 n 1149

R 0.83 R 0.85 R 0.66

MB 0.15 MB 0.29 MB -0.26

NMB(%) 9.2 NMB(%) 31.5 NMB(%) -12.0

RMSE 0.60 RMSE 0.72 RMSE 1.16

NME(%) 25.7 NME(%) 53.1 NME(%) 35.9

NME

NMB


Sulfate Sulfate ((μμg mg m-3-3)) Jan. 4 – Feb. 19, 2002Jan. 4 – Feb. 19, 2002

0

5

10

15

20

0 5 10 15 20

NO3-2002 Release (CASTNet)NO3-2002 Release (IMPROVE)NO3-2002 Release (STN)

CM

AQ

(2

002

Re

lea

se)

CMAQ (2003 Release)

NO3

- ( g m-3)

2002 Release vs. 2003 Release

January 4 to January 20, 2002


0

2

4

6

8

10

0 2 4 6 8 10

NO3-2002 Release (CASTNet)NO3-2002 Release (IMPROVE)

CM

AQ

(2

002

Re

lea

se)

CMAQ (2003 Release)

NO3

- ( g m-3)

June 15 to June 29, 1999

2003 Release produces substantially lower concentrations 2003 Release produces substantially lower concentrations of nitrate - bringing them more in line with observationsof nitrate - bringing them more in line with observations

2002

Rel

ease

2002

Rel

ease

2003 Release2003 Release


Nitrate Nitrate ((μμg mg m-3-3)) June 15 – July 16, 1999June 15 – July 16, 1999

0

1

2

3

4

5

6

0 1 2 3 4 5 6

NO3-East (CASTNet)NO3-West (CASTNet)NO3-east (IMPROVE)NO3-west (IMPROVE)NO3-east (Search)

Mo

del

Observation

NO3

- ( g m-3)

NME

NMB

CAST IMP

n 264 N 415

R 0.27 R 0.36

MB -0.15 MB -0.11

NMB(%) -30.8 NMB(%) -39.1

RMSE 0.57 RMSE 0.51

NME(%) 75.7 NME(%) 95.0


Nitrate Nitrate ((μμg mg m-3-3)) Jan. 4 – Feb. 19, 2002Jan. 4 – Feb. 19, 2002

NME

NMB

NME

NMB

0

5

10

15

20

25

30

0 5 10 15 20 25 30

NO3-East (CASTNet)NO3-West (CASTNet)NO3-east (IMPROVE)NO3-west (IMPROVE)NO3-east (STN)NO3-west (STN)

Mo

del

Observation

NO3

- ( g m-3)

CAST IMP STN

n 407 N 688 n 1044

R 0.76 R 0.54 R 0.39

MB 0.64 MB 0.20 MB -0.27

NMB(%) 46.8 NMB(%) 28.9 NMB(%) -8.0

RMSE 1.36 RMSE 1.26 RMSE 4.37

NME(%) 70.4 NME(%) 96.5 NME(%) 66.9

June 15 – July 16, 1999June 15 – July 16, 1999

Species SO4 NO3 PM2.5 NH4 OC EC

Network CAS IMP CAS IMP IMP CAS IMP IMP

n 264 490 264 415 457 264 396 396

r 0.90 0.90 0.27 0.36 0.71 0.86 0.32 0.69

MB 1.74 0.75 -0.15 -0.11 -0.73 0.31 -0.02 0.00

NMB (%) 38.0 38.4 -30.8 -39.1 -9.8 22.7 -1.5 1.00

RMSE 2.89 2.37 0.57 0.51 4.70 0.70 1.35 0.24

NME (%) 46.4 61.9 75.7 95.0 40.3 36.5 67.2 52.6

Jan. 4 - Feb.19, 2002Jan. 4 - Feb.19, 2002

Species SO4 NO3 PM2.5 NH4 OC EC

Network CAS STN IMP CAS STN IMP STN IMP CAS STN IMP STN IMP STN

n 407 1149 728 407 1044 688 927 714 407 1149 731 1106 731 1148

r 0.83 0.66 0.85 0.76 0.39 0.54 0.37 0.68 0.85 0.41 0.56 0.47 0.55 0.40

MB 0.15 -0.26 0.29 0.64 -0.27 0.20 0.51 1.49 0.33 0.07 0.14 -1.64 0.07 0.42

NMB(%) 9.2 -12.0 31.5 46.8 -8.0 28.9 4.1 40.3 40.0 5.2 17.2 -50.0 31.0 59.1

RMSE 0.60 1.16 0.72 1.36 4.37 1.26 10.4 3.81 0.51 1.46 0.93 3.32 0.48 1.17

NME (%) 25.7 35.9 53.1 70.4 66.9 96.5 50.0 68.9 47.2 57.7 70.0 60.4 81.0 95.0


SummarySummary

The 2003 Release of CMAQ featured many changes, the largest of which impacted aerosol nitrate

Other species continue to be simulated reasonably well:

- majority of correlations are greater than 0.60- majority of model simulations are within 2:1

- majority NMB within ±35%- majority NME less than 70%

(depending on the specie and observational network.)


Through collaboration with AMD, OAQPS will soon perform a full one year simulation (2001) using the 2004 Release of CMAQ.

Part II. Plans for the 2004 Release EvaluationPart II. Plans for the 2004 Release Evaluation

CMAQ Simulation AttributesCMAQ Simulation Attributes

Emissions: 2001 Extrapolated from 1999 EPA NEI (Version 3).

Chemistry: CBIV gas-phase chemistry mechanism.

Meteorology: MM5 processed through MCIP (2.2).

Resolution: Horizontal: Contiguous U.S. Domain.

36 km by 36 km grids

Vertical: Set on a sigma coordinate.

14 layers

A comprehensive evaluation of this simulation will characterize the performance of CMAQ as well as its individual components:

Meteorology modelEmissions modelChemical Transport Model

This evaluation will include the statistics presented earlier, along with:

new metrics - Shaocai Yu’s presentationdiagnostics - Robin Dennis’ presentation

And new, innovative techniques examining model performance over:

space and time – John Irwin’s presentation


Surface-Based: 2-m Temperature2-m Specific humidity10 m wind (speed, direction)Hourly precipitation

Upper-Air: wind profilestemperature profilesspecific humidity profiles

Space / Time: Local, regional, and domain based summariesDiurnal, synoptic, monthly and seasonal

summaries


Meteorology Model EvaluationMeteorology Model Evaluation



Example:Example:

2 m Temperature (K)2 m Temperature (K) 10:00 GMT10:00 GMT

1 July – 31 August1 July – 31 August


Example:Example:

2 m Temperature (K)2 m Temperature (K) Mean ErrorMean Error

Mean BiasMean Bias Winter Winter



Example:Example:

Precipitation Bias (cm)Precipitation Bias (cm) MonthlyMonthly

January; JulyJanuary; July


Spatial TechniquesSpatial Techniques

Example:Example:

KrigingKriging

CASTNetCASTNet

SOSO4 4 concentrationsconcentrations


Spatial TechniquesSpatial Techniques

Example:Example:

KrigingKriging

SO4 concentrationsSO4 concentrations

CMAQ - CASTNet CMAQ - CASTNet residualsresiduals


SummarySummary

Through collaboration with AMD, OAQPS will soon perform a full one year simulation (2001) using the 2004 Release of CMAQ.

An unprecedented, comprehensive evaluation of this simulation will characterize the performance of CMAQ as well as its individual components:

Meteorology modelEmissions modelChemical Transport Model

This evaluation will include new and innovative techniques examining the spatial and temporal characteristics of model performance.

Evaluation of Models-3 CMAQ I. Results from the 2003 Release II. Plans for the 2004 Release Model...

Documents

Transcript of Evaluation of Models-3 CMAQ I. Results from the 2003 Release II. Plans for the 2004 Release Model...