Setting Priorities for Large Research Facility Projects – The National Academies Recommendations and the NSB/NSF Report
Mark Coles
Deputy Director, Large Facility Projects
Office of Budget, Finance and Award Management
Introduction
National Academies study “Setting Priorities for Large Research Facility Projects” (Brinkman report) makes specific recommendations for prioritizing, planning, and overseeing the construction and operation of large facilities supported by NSF
The report cites the need to make greater resources available for pre-award planning of a project’s workscope
Implement rigorous post-award oversight through periodic external review
Summary of the NAS Study Recommendations
Development of a 10-20 year facility roadmap
Three levels of criteria for ranking:
Scientific and technical merit
Agency Strategic Criteria
National Criteria
New starts ranked in annual budget request using clear rationale based on roadmap
Enhanced project pre-approval planning and budgeting
Greater independent oversight and review needed
Review effectiveness of Deputy Director for Large Facilities in two years
OSTP should coordinates roadmaps across agencies
NSF Leadership and NSB should pay careful attention to implementation of proposed reforms
NSF/NSB Actions
NSF undertook a detailed analysis of the NAS report recommendations
Formal report prepared after broadly canvassing NSF for input
Committee formed under direction of Joe Bordogna:
John Hunt (chair) – Senior Advisor, Director’s Office
Wayne van Citters – AST Division Director, MPS
Rich Behnke – Section Head,ATM/GEO
Priscilla Nelson – Senior Staff Associate, Director’s Office
Mark Coles – Large Facilities Deputy Director, BFA
Patricia Crumley (secretary) – Program Analyst, Budget Division, BFA
NSF Assistant Directors and Director reviewed
Further NSB input and revision resulted in final version – on NSB web site (October 2004 Summary Report):
Embraces NAS recommendations for greater clarity and transparency to selection and prioritization process
NSF will take steps to enhance the robustness of the pre-construction project development process to improve cost projections for facilities recommended for construction
NSF Roadmap
NSF is now developing a Facility Plan (the “roadmap”)
Report of projects in construction and various stages of development
Includes detailed discussion of the development plans and criteria for projects in advanced stage of pre-construction development
Discussion of the scientific objectives and opportunities that provide the context and compel the need for their development
Will provide discussion of overarching considerations used for cross-disciplinary prioritization (first and second NAS ranking criteria)
Facility Plan will be a public document, updated annually by NSF Director
Facility Plan should become a strategic tool for communicating with research communities and government policy makers
First public draft of plan expected in April or May, 2005 following March NSB meeting
Expect to post for public comment
Process of Large Facility Project Development
Revises existing process defined in NSF’s Facilities Management and Oversight Guide: http://www.nsf.gov/pubs/2003/nsf03049/nsf03049.pdf
Defines a new stage in the pre-construction development process:
“Readiness Stage”:
Has well defined entrance and exit gates
Entrance requires formally defined and publicly distributed development plan
Readiness Stage Development plan included in NSF’s Facility Plan
Project Life-cycle stages
Horizon Stage
Blue sky thinking
Concept Stage
First NSF support for workshops, studies, small scale research
Development Stage
More focused development
Initial facility proposal
Formulation – with NSF – of project development plan for readiness stage
Readiness Stage
Activities that bring project to a “construction ready” status with cost projections that have a high degree of confidence
Begins 1-2 years prior to expected for NSB approval for inclusion in budget request
New start pool of NSB approved projects proposed for funding
Construction
Operations
Renewal, re-competition, termination, etc.
Project Development Plans
Written by project developers in consultation with NSF
Lays out the development trajectory for a project in the Readiness Stage
Technical, managerial activities needed to bring a project to construction readiness
Planning activities that result in mature construction budget estimates and sound projections of expected operations costs
Development budget requirements
NSF adds oversight decision points during Readiness Stage
Decision points (criteria for advance and “off-ramps”):
Developed by NSF
Approved by NSB
Made public – included in Facilities Plan
Readiness stage gatekeepers
Entry:
Community expresses strong endorsement
Advisory committee endorses first and second level criteria
Division and Directorate review and endorsement of project, including M&O budget
CFO and LFP Deputy assertion that budget and project plans are “construction ready”
MREFC panel assertion that project meets science objectives and maintains priority
NSF Director concurrence with MREFC panel
NSB approval
Exiting Readiness Stage
Projects exit Readiness Stage because:
Board approved
Fail to advance against project development plan criteria
Eclipse by other projects
Reprioritization
Other factors, as deemed appropriate by NSF Director
Successful exiting projects are in “Candidate for New Start” pool
NSB may re-prioritize pool when candidates are added
Reprioritization rationale will be made public
NSF Director proposes subset of new start pool to OMB for construction funding
OMB approves project inclusion in NSF budget request
Congress appropriates funds
Next steps
Revise NSF’s Facilities Management and Oversight Guide to incorporate these new policies
Challenges:
Describe specific implementation steps that incorporate these principles
Define steps to develop “sufficiently mature” budget projections prior to construction approval
What can be done to minimize risk of potential backlog of approved new starts?
Can this new framework can be used to aid fields (like Astronomy) that have needs for very long lead items as part of facility development?
Advisory Committee input welcome
Defining “construction ready”
Creation of a robust final project “baseline”:
Enabling technical developments are complete
There exist thorough cost and contingency estimates with high degree of certainty that project can be completed within scope
Project management is in place
Project Management Control System (PMCS) in place.
Expensive: 5-15% of total construction budget, but must be done eventually
PMCS alone is typically 1-2% of total project cost
Perhaps overly ambitious definition of “construction ready”
Baseline Development
Need to account for practical considerations, such as difficulty of fully staffing key positions prior to construction approval.
Need to preserve goal of defining scope and budget with “acceptable levels of uncertainty” prior to request for construction funds.
Need to define “acceptable”. For example:
High Energy Physics projects typically allocate 25-45% contingency (45% LHC detectors, 25%: IceCube, LIGO)
Projects with large software component can be higher
Projects with large “build to print” component much less.
Can the framework of NSB/NSF’s new process encompass the natural steps typical of project evolution?
Planning large projects naturally evolves through several stages that are typically well understood:
Conceptual design
Preliminary design
Final design
Construction
Commissioning
Operation
Match NSF’s internal policies to these general notions in M&O Guide
The Guide should also define how solicitations for these proposals may be made, if required
Common baseline evolution
Conceptual design
Functional requirements, definition of systems and functional areas.
Large cost uncertainty.
About 1/3 of total design effort.
Preliminary design
Site-specific design. Principal components, types of equipment, members, sizes defined.
Cost uncertainty much reduced, roughly 25% larger than final design cost uncertainty.
About 1/3 of total design effort.
Final design
Interconnections of components defined. Component vendors defined. Mounting and installation defined.
About 1/3 of total design effort.
Coordination with initiatives in other agencies
Want to make sure that Facilities Plan and the NSF’s Management and Oversight Guide broadly mesh NSF’s strategic planning with activities in other agencies – DOE, NASA
Incorporate project-specific factors, criteria, timing of inter-agency coordination within development plans – will be published as components of Facilities Plan
Natural place to articulate NSF’s intentions for specific international partnerships
Next steps in revision of Facilities Management and Oversight Guide
Identify specific implementation steps that support principles promulgated in NSB/NSF report
NSB review
Expect a period for public reaction and comment
Useful to have advisory committee input on these issues and other problems that exist in current Facilities Guide
Backup materials
NAS Study Findings
“There is a lack of funding for disciplines to conduct idea-generating and project-ranking activities and, once ideas have some level of approval, a lack of funding for conceptual development, planning, engineering, and design—information needed when judging whether a project is ready for funding in light of its ranking and for preparing a project for funding if it is selected”.
More planning resources will:
Reduce uncertainties in construction and operations budgets
Reduce uncertainty in construction schedule
Reduce likelihood of de-scoping
Thorough pre-construction risk assessment will create a more robust Project Execution Plan
Create an appropriate framework for cooperation for activities involving international and inter-agency partnerships as an integral part of in-depth planning.
Make project oversight during construction more straightforward, add more definition to the proposed workscope and decision points.
Allow more in-depth consideration of transition to operations and greater certainty in predicted O&M costs.
Avoid a funding hiatus. Overlap some development activities beyond NSB approval and inclusion in annual NSF budget submission.
NSF/NSB Report:
“The National Academies’ Report properly calls attention to the necessity for considerable pre-approval funding for planning and development when it questions whether there is sufficient NSF support for this "bottom up" process. NSF endorses the Report’s recommendations to provide researchers access to funding sufficient to develop compelling research agendas, to refine and prioritize their facility requirements, and to complete research and development on facility designs and needed technologies. The level and form of funding for planning and development will be reviewed, and an evaluation will be made of how project funds are best invested to attain robust plans and schedules with better cost projections, so that only well-defined and thoroughly-costed projects are brought forward for consideration by the Board.”
More NSF/NSB Report:
“The Director and the Board recognize the need to strengthen oversight of the implementation of large facility projects, which will require increased investments of NSF staff time and travel funds. The Report emphasizes the importance of initial planning and definition of technical scope, budget, and schedule, followed by periodic post-award status reviews held on-site by external experts, with implementation of a transparent process for management of changes to a project’s implementation plan.”
