Consolidating and aligning the audit of disaster related aid with ISSAI 100

The GUID should provide supplementary information, so some parts should be summarized: o Annex 1- Glossary: 5 pages; states extra detailed definitions. Definitions should relate to what is mentioned in the GUID; some of the definitions are mentioned only in the annexes (small-scale disaster, large-scale disaster, frequent and infrequent disaster, slow-onset disaster, sudden-onset disaster, Geographic Information Systems (GIS), prospective disaster risk management activities, corrective disaster risk management activities, prospective disaster risk management activities). o The GUID copies paragraphs from ISSAI 100 without any additions: Planning the Audit (paragraph 16) Develop an audit plan and design the audit (paragraphs 36, 37 and 38). Known disaster management cycle consist of only 4 categories of activities (Mitigation, preparedness, Response, and Recovery). Paragraphs (59) and (60): publication and distribution of reports depend on the law governing each audit office. Report and Follow-up section: nothing is mentioned regarding follow-up.

Bulgarian National Audit Office considers that GUID 5300 may include aspects related not only to local, but also to global health and life threats (e.g. COVID -19). There are INTOSAI resources already available on this topic, for example at: www.intosaicovid19.org и www.idi.no/en/covid-19  

PAS general comments and recommendations; Objective: The GUID is very much oriented towards crises such as tsunamis, earthquakes and nuclear accidents but, apparently, excluding health crises. It should be explicitly stated, if health crises are deliberately excluded from this GUID. On the other hand, if the GUID should also cover health crises, performance audits may look at international initiatives. These types of scenarios are not reflected in the draft guide. Moreover, performance audits could even be conducted in cooperation with international organisations such as WHO and UNDP in some cases. Scope: The draft GUID addresses the first principles involved in auditing Disaster Management. The section on Scope of the GUID mentions that for determining specificities of a majority of activities in various phases of Disaster Management, the principles of obtaining understanding and conducting risk assessment should suffice. The subject matter of Disaster Management is however vast, with multiple applicable frameworks and toolkits, for both the Executive/Management and for the Performance Evaluators including SAIs. In former ISSAIs 550, 5520 and 5530, there was a plethora of information available to an auditor, which could readily be customised to the requirements of any given audit. While the draft clearly meets its stated objective of bringing out the first principles involved in auditing disaster management, and Annexes II and III provide an auditor with examples of audit objectives and elements involved in planning an audit engagement, it may also be useful to develop a practitioners’ handbook in this area. Such a handbook could present, in one place, risk matrices, detailed checklists and audit tools applicable to various phases of the Disaster Management Cycle, as well as a listing of various initiatives, tools and frameworks available for furthering knowledge and skill sets in this area. Such handbooks are more amenable to revisions as well, if developed outside the IFPP. Former ISSAI 5500 series: Are we correct to assume that GUID 5330 will fully replace the former ISSAI 5500-series? If so, we believe that relevant and important information for auditing disaster management would be lost. Specifically, we refer to ISSAIs 5530 about risks of fraud and corruption and 5540 about the use of geospatial information. For example, ISSAI 5540 dealt with the specific characteristics of geospatial information and the importance of this information for all stages and activities in disaster management. This kind of information (and data) is completely different from the information and data frequently used by auditors. Relevant information from former ISSAI 5530 and ISSAI 5540 could be maintained in form of guidance material outside the IFPP, or as annexes to this GUID. Risk: There is no distinction between “risk” and “uncertainty” in this GUID. This might lead to the misconception that any “risk” (in the common acceptance of the term, as phenomena that can be modelled using probability laws) can be documented and inferred from past observation. For performance audit, it is also important to broaden the perspective from risk assessment to identifying risks and problems (in line with the PA standards). Audit criteria: It would be useful to have some examples of audit criteria typical for a Disaster Management Audit. Examples of such criteria could be best international practices, management strategies and targets, average numbers and indicators concerning loss or injuries, experience from rescue operations, rehabilitation and reconstruction, etc. Insurance: The GUID does not explicitly mention insurance. Even if States are their own insurers in general, they may cover residual risks with insurance or market products such as cat-bonds. Complex systems: There are no references to the complexity framework and risk management in complex systems. We live in an over-connected world, where small events in one part of a network could lead to a systemic disaster. By not covering this aspect, we may have a blind spot in the GUID. Performance Audit: One of the questions asked by the project group was “Does the proposed GUID provide sufficient basis for SAIs to perform performance and compliance audits of disaster management?” It is not clear, at what level of detail these GUIDs should be presented (generally), and to what extent they should connect to the practicalities of conducting an audit. As a standalone document, it is not in itself sufficient for SAIs to guide auditors in conducting a performance audit. However, that should not be the objective of a subject matter specific guidance document. The PAS' position is that a subject matter specific GUID should not replace or replicate the standards or corresponding GUIDs, but to be useful as practical guidance on auditing the specific subject matter, they need to refer the relevant standards and GUIDs, as appropriate. To sum up: PAS agrees with the current high-level presentation in this GUID. In this case, the presentation of the three audit types is sufficient, but there could be more references to the relevant ISSAIs and GUIDs throughout as this will guide the reader towards more information about how to conduct a performance, compliance, or financial audit. Specific comments: Para 3: During a disaster such as Covid-19, spanning over months or even years, the SAI also has an important role to play during the course of the disaster. Figure 1: Could we also include the process during a disaster period, such as Covid-19? This would not be a once off event but will span over time. Para 16: Also, refer to ISSAI 300 regarding planning a performance audit. Para 21: For clarity, we suggest aligning the definition of PA with the definitions in ISSAIs 300 and 3000 (including references). Para 23: Other relevant and important sources: researchers and academics working in this field. Para 30: “…specify the risks…” can we rather state: identify risks or problems (and not specify)? In order to accommodate performance audit methodology and to be in line with PA standards, please refer “identify risks or problems”. Para 41: “…based on the audit risks…” We suggest adding: and problems identified. (In line with PA objectives). Para 41: 1st bullet: there is currently no guidance on how to conduct “combined audits”, and the topic is somewhat controversial. We therefore suggest deleting that part of the sentence (“…or a combination of both”). It is not necessary for this GUID, nor is it helpful, since we cannot refer to any concrete guidance on how to do it. Para 42: 2nd bullet point “…ensure an appropriate coverage…” This criterion is not clear. Does it mean that the full scope of the disaster should be audited regardless of the resources available? Alternatively, does it mean that the resources available should determine the scope? Please clarify. Para 47: May want to mention also the role of international organisations. Para 49: In times of disaster, the role of the SAI is also to avoid adding to the burden of the audited bodies, already tested and under pressure. Any organisation between SAIs and the wider audit community must take this into account and aim for leaving the smallest possible “footprint". Paras 50 and 55: Section 50 covers organizing on the spot visits to gather information and to understand and record evidence. Section 55 highlights the challenges and mentions alternative methods. Even though on spot visits is likely to be the best way to gather evidence, there are some challenges. For example, the audit procedure might cause some inconvenience for an ongoing disaster response, especially during an emergency operation. Other parties involved may not understand the role of the SAI, and rather see SAI´s involvement as an obstruction to the operation. These challenges could potentially reduce the level of cooperation and/or increase the level of resistance from parties involved. The auditor should carefully prepare for and manage the risk of this problem. The GUID could provide a bit more detail and examples of challenges and alternative methods when encountering challenges/risks/potential problems related to emergency operations and SAI´s role in that respect. Para 55: Attention to beneficiaries and victims should be part of the audit approach (and is well suited for a performance audit). Para 60: “…may consider distributing…” In doing a performance audit, ISSAI 300 states that auditors should distribute widely. Refer ISSAI 300 requirements on reporting, this is to direct the performance auditor to PA standards and GUIDs for more information about how to do a PA. Annex I: General comment: the annex does not mention insurance. Annex I, early warning system: The content reflects a classical definition of “risk”, where probabilities can be inferred from past events. However, you may also want to consider including that “risk” in a broader sense encompasses “uncertainty”: we do not necessarily have a precise idea of the signals we must collect, or their meaning. You may consider including in the “early warning systems” scenario building and hypothetical models of unobserved disasters. Annex II (PA objectives), 1st bullet: “Determine if Government´s activities… such as…” It can also include legislation. Consider adding. Annex II, 4th bullet: not all SAIs have the mandate to audit policy. Most SAIs can only audit the implementation of policies. Consider modifying. Annex II, 6th bullet: “…appropriate expenditure”: ...PAS suggests adding “…and has been used to reach the intended objectives”. Consider revising. Annex II, 7th bullet: consider adding the “economic procurement of resources”. Annex II, 8th bullet: consider rephrasing. PAS suggests, “Determine if the goods and services (as per government's assistance or relief initiatives) reached the intended users in time, in the right quantity/quality at the lowest possible cost”. Annex II, 9th bullet: consider rephrasing. PAS suggests, “Determine whether recovery and control of operations were planned effectively and executed efficiently”. Annex II, 10th bullet: consider rephrasing. PAS suggests, “Determine if resources or disaster-related aid was procured economically”. Annex II, 11th bullet: consider rephrasing. PAS suggests, “Determine if human, financial and other resources were used efficiently”. Annex II, 12th bullet: consider deleting, or modifying. A performance audit generally leads to a report discussing exceptions from the set criteria. It will be difficult for an auditor to assess how effective the actions were. Keep examples simple for the sake of clarity. Annex III: In Annex III, under the heading – “Risk/problem analysis related to medium- and long-term post disaster activities”, we suggest adding: “Are there any instances of disaster related aid or funds being diverted for building infrastructure or developing projects not related to post disaster reconstruction?” The background for the suggestion is that there can be instances where earmarked funds/aid has been diverted at the cost of more necessary disaster resilient infrastructure/projects. Thus, this carries risks related to not only affecting disaster preparedness, but also significant financial risks related to depiction, disclosures and possibly fraud.

All this seems to be embeded in a classical definition of « risk » where probabilities can be infered from past events. However, one could mention that “risk” in a broader sense encompasses “uncertainty” : we do not have a precise idea of the signals we must collect or their exact meaning. As such, I would include in “early warning systems” scenario building and hypothetical models of unobserved disasters, just in case.

Not a single word on insurance

> Attention to beneficiaries and victims, even in a performance audit, should be part of the audit approach.

> In times of disaster, the role of the SAI is also to avoid adding to the burden of the audited bodies, already tested and under pressure. Any organisation between SAIs and the wider audit community must take this into account and aim for "the smallest footprint".

> Replace 'commence' by 'start over' > The increased risk of fraud in these times of disorganisation of circuits, lighter controls and massive injection of cash flows, goods and services is quite rightly pointed out.

> As well as researchers and academics working in this field

> Replace 'commence' by 'start over'

> Predisaster activities should also include an organisation of tasks between authorities (central, decentralised, territorial)

> Again I wonder if we should not encompass insurance activities and beyond risk-sharing

> State bodies in certain cases may also cover the residual risk with insurance products and in some extreme cases in sharing it with markets through cat-bonds

> This is certainly true but other reasons cannot be ignored : high density interconnection of systems making the world increasingly complex, automation and so on.

Questions addressed by the working group:   Does the proposed GUID provides a useful and relevant guidance material? The GUID appears to covers many aspects of Disaster Management in a clear, well organized and convincing form It is practical, and offers many illustrations and, even if they are not developed, case studies or examples   Are there other important disaster management matters that you consider useful and may be included in the proposed GUID? If yes, identify and explain.   The guide is very much oriented towards crises such as tsunamis, earthquakes and nuclear accidents, i.e. the major crises of the last twenty years excluding health crises, to which are added crises of slower occurrence such as droughts. It is even stated in the first paragraph that the major risk factors often seem to be linked to climate change. Though, if health crises are deliberately excluded, this should be explicitly stated in the guide. Another concern is that there is not any distinction made on “risk” and “uncertainty”. This might lead to the illusion that any “risk” (in the common acceptance of the term) can be documented and inferred from past observation, which is actually not the case. If one restricts the approach to “risk” (i.e. phenomena that can be modeled using probability laws), I guess many if not all aspects are covered in the GUID. To quote Sword-Daniels and al. (2018),[1] “For example, a study of NASA’s decision-making prior to the Columbia space shuttle disaster in 2003 found that, leading up to the disaster, there was overconfidence in quantitative data and marginalisation of non-quantifiable information, which created insensitivity to the uncertainty involved and loss of institutional memory”. Solutions may be found in scenario generation and analysis, early signals detection and interpretation, intelligence and exchanges in an interdisciplinary context with scientists from Academia. I wonder why insurance is not explicitly mentioned in this GUID. Even if States are their own insurers in general, they may cover residual risks with insurance or market products such as cat-bonds. I would recommend the paper by Michel-Kerjan, Zelenko2, Cárdenas, and Turgel (2011): Catastrophe Financing for Governments: Learning from the 2009-2012 MultiCat Program in Mexico, OECD Working Papers on Finance. Finally yet importantly, even if I have not spent too much time in thinking about a possible answer to this aspect of the GUID, I observe that one does not refer to the complexity framework and to risk management in complex systems (I refer to complexity in the physicists’ view). We live in an incredibly over connected world, a giant system, where small events in one part of the network may lead to a systemic disaster. I wonder if not covering this aspect (disasters because of interconnected complex systems with retroaction loops) specifically could lead to a blind spot in the GUID. Not sure. See for example Uusikylä P., Tommila P., Uusikylä I. (2020) “Disaster Management as a Complex System: Building Resilience with New Systemic Tools of Analysis.” In: Lehtimäki H., Uusikylä P., Smedlund A. (eds) Society as an Interaction Space. Translational Systems Sciences, vol 22. Springer, Singapore. https://doi.org/10.1007/978-981-15-0069-5_8   Does the proposed GUID provide sufficient basis from SAIs to perform performance and compliance audits of disaster management? - Certainly but I would suggest to take more time to deepen the analysis so to cover more theoretically grounded aspects of disaster management. - If the guide also covers health crises, performance audit should also cover the international actions put in place by each country concerned, which is not really reflected in the draft guide (moreover, performance audits should also be conducted within international organisations such as WHO and UNDP).     [1] « Embodied uncertainty: living with complexity and natural hazards”, Journal of Risk Research, 21(3), pp.290-307

The GUID 5330 provides guidance for the disaster management audit with to evaluate whether the activities before, during and after the disaster have the appropriate controls and guarantees. This GUID will be a great contribution to the Supreme Audit Institutions, providing guidance to carry out disaster risk management audits and ensure the adequate and transparent use of public resources and of those that come from contributions from donor countries during disasters, as well as the resources that are allocated to rehabilitation and reconstruction activities.