(2006). A conventional observation dataset and bias-corrected satellite temperature data are 對assimilated. Weather Rev., 135, 3541–3564, 2007. a, An interactive open-access journal of the European Geosciences Union, Efficient ensemble data assimilation for coupled models with the Parallel Data Assimilation Framework: example of AWI-CM (AWI-CM-PDAF 1.0). Janssen, Peter AEM 2008 Progress in ocean wave forecasting. ensemble square root Kalman filters, Mon. Further, the ensemble is initialized and the analysis step of the data assimilation can be executed at any time without restarting the model. By combining resolution and ensemble size, we have achieved the world's largest weather DA experiment using a global cloud-resolving model and an ensemble Kalman filter method. Earth System Model and the Data Assimilation Research Testbed, (2012b) for an ocean-only model. Weather Rev., 143, 1347–1467, 2015. Counillon, F., Drper, C., Frolov, S., Fujii, Y., Kumar, A., Laloyaux, P., For the first aspect, one has to adapt the run script. Adv. Geosci., 55, 110–118, 2013. a, b, c, d, e, f, g, Nerger, L., Hiller, W., and Schröter, J.: PDAF - The Parallel filter: an alternative to ensemble square root filters, Tellus, 60A, Dissertations and … Afterwards, the RMSE remains nearly constant, which is a typical behavior. Lett., 43, 752–759, 2016. , Snyder, C., Bengtsson, T., Bickel, P., and Anderson, J.: Obstacles to Each model uses a domain decomposition; i.e., each process computes a small region of the global domain in the atmosphere or the ocean. The weight vector wk and matrix W̃ are now given by. In ensemble data assimilation, b T is approximated by using the sample covariance estimated from an ensemble of model forecasts. Figure 7 shows the root-mean-square error (RMSE) of the SST in the analysis step with respect to the assimilated observations over time. OpenMP: OpenMP Application Program Interface Version 3.0, available at: Pardini, F., Corradini, S., Costa, A., Ongari, T. E., Merucci, L., Neri, A., Remote Sensing, 11, 234, Burgers, G., van Leeuwen, P. J., and Evensen, G.: On the Analysis Scheme in the In contrast, the analysis step shows a systematic time increase. system for seamless sea ice prediction based on the AWI climate model, J. (AWI-CM-PDAF version 1.0), Zenodo. Applications based on the AWI-CM-PDAF 1.0 code are given in Mu et al. Kalman Filter Technique, Mon. This text and reference focuses on various popular data assimilation methods, such as weak and strong constraint variational methods and ensemble filters and smoothers. Meteor. Assuming that these observations take about 15 s, such as the finalization of the coupled model, the execution time would increase by a factor of 4 for offline-coupled DA compared to online-coupled DA. The atmosphere uses a horizontal spectral resolution (T63, about 180 km) with 47 layers. There are four different types of routines, which are displayed in Fig. 4: Figure 4PDAF filter analysis step and related call-back routines provided by the user. Weather (2017). World Scientific, 63–83, 2005. , Nerger, L., Danilov, S., Hiller, W., and Schröter, J.: Using sea level data Q. J. Roy. Kalman filter for data assimilation in oceanography, J. Further, the default version of this routine splits the communicator MPI_COMM_WORLD. data assimilation involving the use of ensemble forecasts. This variability is partly caused by the time for DA coupling (see discussion below) but also by the fact that the semi-implicit time stepping of FESOM leads to varying execution times. model using Monte Carlo methods to forecast error statistics, J. Geophys. weather simulation is also critical; thus, we need more advanced data assimilation (DA) technology. This structure can also be used in the case of an offline coupling using separate programs for the model and the analysis step. Q. J. Roy. The online coupling for DA was already discussed in Nerger and Hiller (2013) for an earlier version of the ocean model used in the AWI-CM. Nerger et al. PDAF provides a generic framework containing fully implemented and parallelized ensemble filter and smoother algorithms like the LETKF (Hunt et al., 2007), the ESTKF (Nerger et al., 2012b), or the nonlinear NETF method (Tödter and Ahrens, 2015) and related smoothers (e.g., Nerger et al., 2014; Kirchgessner et al., 2017). model based on triangular prismatic elements with application in studying the For example, one could compute the analysis step not only by using the processes of model task 1 but also by using processes of several or all model tasks. This property is also important for coupled DA, where the state vector will be distributed over different compartments, such as the atmosphere and the ocean. PDAF (Nerger and Hiller, 2013, http://pdaf.awi.de, last access: 14 September 2020) is a free open-source software that was developed to simplify the implementation and application of ensemble DA methods. This interface routine is used to define parameters for the call to the PDAF library routines, so these do not need to be specified in the model code. Res., 113, C05015, Yu, L., Fennel, K., Bertino, L., Gharamti, M. E., and Thompson, K. R.: Insights Earth Sy., 5, 146–172, 2013. a, Tang, Q., Mu, L., Sidorenko, D., Goessling, H., Semmler, T., and Nerger, L.: Attendees will leave with a basic understanding of ensemble filter data assimilation that will allow them to better interpret ensemble forecasts. An interesting aspect is that when one runs separate assimilation systems for the two compartments with the same filter methodology, one can compute a strongly coupled analysis by only exchanging the parts of y, HXf, and R in between both compartments and then initializing the vectors containing observational information from all compartments in the assimilation system of each compartment. Ensemble Kalman Filter, Mon. individual phytoplankton fields, J. Geophys. Soc., The first half of the course will be interactive tutorials that combine lectures with complementary online exercises. The numbers mark the rank index of a process in a communicator. Even though we only perform weakly coupled DA here, we expect that the compute performance would be similar in the case of strongly coupled DA, as is explained in Sect. 6. Accordingly, each ocean model will be placed distant from the atmospheric model to which it is coupled. Thus studying weakly coupled DA, which is still used in most applications, has a value on its own. Good performance with small ensemble filters applied to models with many state variables may require ‘localizing’ the impact of an observation to state variables that are ‘close’ to the observation. Ensemble filter data assimilation is now widely used for numerical weather prediction, but it is also being applied for ocean, sea ice, land surface, space weather and many other geophysical systems. Thus, only a single-line call to each interface routine is added to the model code, which keeps the changes to the model code to a minimum. To assess the parallel performance of the assimilation system described above, AWI-CM is run here in the same global configuration as described by Sidorenko et al. effect of topography representation, J. Geophys. The ensemble data assimilation is based on an ensemble of short-range forecasts of atmospheric states, which are employed together with measurement data as starting point for atmospheric state determination. ECMWF Workshop, edited by: Zwieflhofer, W. and Mozdzynski, G., Functionality to interface between the model, which operates on physical fields, and the assimilation code, which only works on abstract state vectors, has to be provided in a case-specific manner by the users based on code templates. uncertainty estimates, J. Geophys. parallelization”). on multivariate updates of physical and biogeochemical ocean variables using Weather Rev., 126, 796–811, 1998. , Hunt, B. R., Kostelich, E. J., and Szunyogh, I.: Efficient data assimilation Ensemble filter data assimilation algorithms use a set (ensemble) of model state estimates to enable the assimilation process. S., Koernblueh, L., Lohmann, U., Pincus, R., Reichler, T., and Roeckner, E.: coupled data assimilation, Clim. Weather Rev., 126, 1719–1724, 1998. , Chang, Y.-S., Zhang, S., Rosati, A., Delworth, T. L., and Stern, W. F.: An Here, an updated coupling strategy is discussed that requires less changes to the model code. Further variation in the dependence on the ensemble size is visible for the pre- and post-step operations (red line). The number of processes should be set so that the requirements on the ensemble size for a successful assimilation can be fulfilled. The model-agnostic structure of the assimilation software ensures a separation of concerns in which the development of data assimilation methods can be separated from the model application. AWI-CM – output files and plot scripts, Zenodo. In DA experiments on ,a regional scale, a "Big Data Assimilation" system has been ,developed to capture rapidly changing convective weather by ,combining a 100,,m mesh limited,,area simulation with advan,ced ,weather radar [2,7,]. The routine includes calls to the routine “PDAF_print_info”, which print out information about execution times of different parts of the assimilation program as measured by PDAF as well as information about the memory allocated by PDAF. When the time for the DA coupling is subtracted from the forecast time, the variability is much reduced as the black dashed line shows. 2 outline • goals of data assimilation • links between da & ensemble forecasting • da issues in era of satellites • forecasting in a chaotic environment • estimating & sampling initial errors • estimating & sampling model related errors They will have a sufficient understanding of the DART system to evaluate whether using existing DART capabilities could be useful for their specific applications. Here, a configuration is used that computes the filter analysis step on the first coupled model task using the same domain decomposition as the coupled model. These methods are far more costly to compute than a single coupled model because of the required integration of the ensemble. For a more efficient execution, one has to ensure that the ocean–atmosphere pairs are placed close to each other. atmospheric observations into the ocean using strongly coupled ensemble data In this case, the compartment models frequently exchange information at the interface of the model domains to influence the integration of the other model compartment. The left-hand side of Fig. 2 (Fig. 2a) shows the typical flow of a coupled compartment model. Both QT and LM worked on optimizing the compute performance of the implementation of PDAF with AWI-CM. individual phytoplankton fields, J. Geophys. assimilation, Geophys. System with a Regional High-Resolution Atmosphere-Ocean Coupled Model Based This completes the model initialization; then the time stepping is computed. Res.-Oceans, 124, 470–490, Only strongly coupled DA is expected to provide fully dynamically consistent state estimates. These variations are due to the fact that the large compute application is widely spread over processors of the computer. Here, at the very beginning of the program, the parallelization is initialized (“init. for Coupled Data Assimilation Using a Lorenz Atmosphere and a Simple As this reading is model specific, it is performed by a user-provided routine that is called by PDAF as a call-back routine (see Sect. 3.4). Then, “PDAF_assimilate_lestkf” will check for the analysis time according to the individual number of time steps in the forecast phase. Given that y and R are initialized from information that is usually stored in files, one can also let the DA code coupled into each compartment model read these data and only exchange the necessary parts of HXf. https://doi.org/10.5281/zenodo.3823816, 2019b. a, OpenMP: OpenMP Application Program Interface Version 3.0, available at: (2016) proposed an interface-solver approach for variational DA methods, which leads to a particular solution for the variables close to the interface. In numerical weather prediction applications, data assimilation is most widely known as a method for combining observations of meteorological variables such as temperature and atmospheric pressure with prior forecasts in order to initialize numerical forecast models. for nonlinear data assimilation, Mon. This is achieved with a command line such asmpirun -np  N_O fesom.x : -np N_A  \ echam.x: -np  N_O fesom.x   -np N_A \ echam.x … Climate, 26, 10218–10231, 2013. a, Harlim, J. and Hunt, B. R.: Four-dimensional local ensemble transform Kalmn stochastic data assimilation methods for high-dimensional non-Gaussian The maximum ensemble size was here limited by the batch job size of the used computer. In future work, we plan to add the assimilation of atmospheric observations and to complete the implementation of strongly coupled data assimilation, which requires the exchange of observations in between the ocean and atmosphere. Thus, through Eq. (1), the algorithm can directly update both compartments, xA and xO, using observations of just one compartment. While particle filters do not assume Gaussianity of error distributions, they are difficult to use with high-dimensional models, because particular adaptions are required to avoid the case when the ensemble collapses to a single member due to the so-called “curse of dimensionality” (see Snyder et al., 2008). SMOS sea ice thickness data simultaneously, Q. J. Roy. External access to the ECHAM6 model is provided through their licensing procedure (https://mpimet.mpg.de/en/science/models/availability-licenses, last access: 14 September 2020). Weather Rev., Figure 3Example configuration of MPI communicators: (a) AWI-CM, (b) AWI-CM with PDAF extension for ensemble data assimilation. In the coupled model, this routine is executed before the parallelization of the coupler is initialized. The current most widely used ensemble filter methods are ensemble-based Kalman filters (Evensen, 1994; Houtekamer and Mitchell, 1998; Burgers et al., 1998). in a simple coupled climate model: The role of ocean-atmopshere interaction, In particular, for Ne=40 the execution time is almost identical to that of Ne=2. For example, Laloyaux et al. A common approach for data assimilation is ensemble-based methods which utilize an ensemble of state realizations to estimate the state and its uncertainty. Browne, P. A., de Rosnay, P., Zuo, H., Bennett, A., and Dawson, A.: Weakly 323–340, 1998. , Pradhan, H. K., Voelker, C., Losa, S. N., Bracher, A., and Nerger, L.: B. The method of hierarchical modelling allows us to calculate these probabilities. Principles and evaluation, Ocean Model., 6, 125–150, 2004. , Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic The weight is applied by modifying matrix R−1 in Eqs. (4) and (5). The ensemble allows the calculation of the uncertainty of its atmospheric variables at the time of the analysis. The ensemble mean was set to a model state for 1 January 2016 from a historical (climate) run of AWI-CM. While the frameworks use very similar filter methods, they differ strongly in the strategy of how the coupling between model and DA software is achieved. For further details on the model, we refer to Sidorenko et al. In this way, only single-line subroutine calls are added, which might be enclosed in preprocessor checks to allow users to activate or deactivate the data assimilation extension at compile time. However, in recent years coupled models of different Earth system compartments have become more common. This study pointed to the further complication of the choice of variable (linear or logarithmic concentrations for the biogeochemical compartment) for strongly coupled assimilation. Meteor. In addition to initial condition uncertainty, representing model errors (e.g. These alternative parallelization strategies are, however, more complex to implement and hence not the default in PDAF. (2016) in a twin experiment using an EnKF with dynamically estimated covariances between the atmosphere and ocean in a low-resolution coupled model. Meteorol. Weather Rev., 143, 1347–1467, 2015. Weather Rev., 145, 565–581, 2017. a, Kurtz, W., He, G., Kollet, S. J., Maxwell, R. M., Vereecken, H., and Hendricks Franssen, H.-J. where T is a projection matrix with j=Ne rows and i=Ne-1 columns defined by. 1.6 Objectives through perfect model test The purpose of this paper is to explore whether and how EDA could be used for fault stress state estimation and eventually for earthquake forecasting. These routines are executed by all processes that participate in the model integrations, and each routine acts on its process subdomain. In addition, the assimilation program would also need to read these restart files and write new restart files after the analysis step. Short course/workshop registration is not included in the 99th Annual Meeting registration, and short course/workshop registration does not include registration for the 99th AMS Annual Meeting. R., Wang, Y., and Wu, X.: Coupled data assimilation for integrated Earth The interface routines define parameters for PDAF and call PDAF library routines (green). Compared to the default setup in PDAF for a single-compartment model, we have adapted the routine to account for the existence of two model compartments. All parallel communication is performed within so-called communicators, which are groups of processes. Systems – Implementation Strategies and Scalability, Comput. The ECHAM6 source code is maintained by the Max Planck Institute for Meteorology and is freely available to the public (http://www.mpimet.mpg.de/en/science/models/mpi-esm/echam/; Max Planck Institute for Meteorology, 2019a). The assimilation in DA-SST strongly reduces the RMSE during the first 2 months. Only if this is true will the analysis step be executed; otherwise, the time stepping is continued. Analogous to many large-scale geoscientific simulation models, PDAF is implemented in Fortran and is parallelized using the Message Passing Interface standard (MPI; Gropp et al., 1994) as well as OpenMP (OpenMP, 2008). for Coupled Data Assimilation Using a Lorenz Atmosphere and a Simple Weather Rev., 136, 4629–4640, 2008. , Stevens, B., Giorgetta, M., Esch, M., Mauritsen, T., Crueger, T., Rast, S., (2018) used atmospheric and ocean models with 1∘ resolution. With this approach, the time increase of the forecast was reduced to about 40 % for the increase from Ne=2 to Ne=46. When used with coupled models of Earth system compartments, e.g., the atmosphere and the ocean, consistent joint states can be estimated. T., Zampieri, L., Losch, M., and Goessling, H. F.: Toward a data assimilation Weather Rev., 126, 1719–1724, 1998. a, Chang, Y.-S., Zhang, S., Rosati, A., Delworth, T. L., and Stern, W. F.: An If there are observations of both compartments, they are jointly used to compute the weights. Section 5 examines the assimilation behavior of an example application with AWI-CM. on multivariate updates of physical and biogeochemical ocean variables using • These sets of objective analyses are exactly the The blue color marks coupling routines whose parallelization needed to be adapted for the data assimilation. Adv. The EnKF originated as a version of the Kalman filter for large problems (essentially, the covariance matrix is replaced by the sample covariance ), and it is now an important data assimilation component of … The length of these experiments is chosen to be long enough so that the execution time is representative to assess the scalability. The tutorials will cover many of the ensemble filter algorithms that are used today for geophysical applications. (2012a). A common approach for data assimilation is ensemble-based methods which utilize an ensemble of state realizations to estimate the state and its uncertainty. The method worked well in low-order systems, but it was not immediately clear how it could be applied to the vastly larger models that are commonplace for atmospheric and oceanic pre-diction and simulation. Rev., 143, 4678–4694, 2015. , Liu, Z., Wu, S., Zhang, S., Liu, Y., and Rong, X.: Ensemble data assimilation which contains as many FESOM-ECHAM pairs as there are ensemble members. Dynam., 40, 775–803, 2013. a, Danilov, S., Kivman, G., and Schröter, J.: A finite-element ocean model: The different boxes can (but are not required to) be subroutine calls. To discuss strongly coupled filtering, let us assume a two-compartment system (perhaps the atmosphere and the ocean). However, in contrast to EMPIRE, the model usually is augmented by the DA functionality; i.e., model and DA are compiled into a joint program. The distribution of the processes is exemplified in Fig. 3a for the case of six processes in MPI_COMM_WORLD. This collection of in situ data contains about 1000 to 2000 profiles per day at depths between the surface and 5000 m depth. This section describes the assimilation framework and the setup of the DA program. Ensemble-based data assimilation techniques are being explored as possible alter- natives to current operational analysis techniques such as three- or four-dimensional variationalassimilation.Ensemble-basedassimilationtechniquesutiliseanensemble of parallel data assimilation and forecast cycles. This can be the case when, for example, an input/output (OI) server is used that reserves processes exclusively for the file operations. problems, Tellus A, 70, 1445364, https://doi.org/10.1080/16000870.2018.1445364, 2018. a, Wang, Q., Danilov, S., and Schröter, J.: Finite element ocean circulation Principles and evaluation, Ocean Model., 6, 125–150, 2004. a, Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic Softw., 68, 122–128, 2015. a, b, Browne, P. A., de Rosnay, P., Zuo, H., Bennett, A., and Dawson, A.: Weakly The improvements of the model fields by the DA of SST are mainly located in the upper 200 m of the ocean. Earth System Model and the Data Assimilation Research Testbed, On average, over the full year of the experiment, the RMSE of salinity is reduced by 5.6 %. Systems – Implementation Strategies and Scalability, Comput. X.: Improving sea ice thickness estimates by assimilating CryoSat-2 and Soc., 125, 723–757, 1999. . However, the assimilation effect will be rather small for these 10 analysis steps. Figure 5Execution times per model day for varying ensemble sizes for different parts of the assimilation program. A similar parallelization was also described by Browne and Wilson (2015). extension of the nonlinear ensemble transform filter, Tellus A, 69, 1327766, https://doi.org/10.1080/16000870.2017.1327766, Marine Syst., 16, To validate the assimilation with independent observations, temperature and salinity profiles from the EN4 data set (EN4.2.1) of the UK Met Office (Good et al., 2013) are used. data assimilation involving the use of ensemble forecasts. Keywords—numerical weather prediction, data assimilation, Fugaku I. JUSTIFICATION We performed a data-centric redesign of the ensemble DA system to solve the bottleneck of memory-intensive applications and file input/output (I/O), and realized a DA experiment by using a 3.5-km mesh, 1024-ensemble (which was 500 times Soc., 136, 1991–1999, 2010. , van Leeuwen, P. J., Künsch, H. R., Nerger, L., Potthast, R., and Reich, S.: The operations performed in each routine are rather elementary to keep the complexity of the routines low. Further, we expect a similar scalability in the case of strongly coupled DA. Temperature is more strongly improved than salinity. Brasseur, P., Kirchgessner, P., and Beckers, J.-M.: State-of-the-art (2019) discussed the strongly coupled DA for a coastal ocean–biogeochemical model assimilating real observations of sea surface temperature. Efficient ensemble data assimilation for coupled models with the Parallel Data Assimilation... Alfred-Wegener-Institut, Helmholtz-Zentrum für Polar- und Meeresforschung (AWI), Bremerhaven, Germany, Anderson, J., Hoar, T., Raeder, K., Liu, H., Collins, N., Torn, R., and While the RMSE of the salinity first increases during the first month, it is reduced from day 60, but until day 140 it is sometimes larger than at the initial time. Abstract. Particle filters for high-dimensional geoscience applications: a review, Q. Mahfouf, J.-F., MArtin, M., Pena, M., de Rosnay, P., Subramanian, A., Tardif, Meteor. Vetra-Carvalho, S., van Leeuwen, P. J., Nerger, L., Barth, A., Altaf, M. U., (2015). Soc., https://doi.org/10.1002/qj.3885, in press, 2020. a, b, c, Tödter, J. and Ahrens, B.: A second-order exact ensemble square root filter To build the system, a wrapper was developed to perform the online coupling of model and DA software.