Deliverable template

3strategy to reach out to new communities: concept of Virtuous cycle

The EGEE vision is to deploy a grid infrastructure for multiple research communities. In order to achieve this vision, the EGEE proposal is structured in three main areas of activity: services, middleware and networking, which are described in the sections SA, JRA and NA of EGEE proposal and Technical annex (description of work) [1].

It is essential to the success of EGEE that the three areas of activity reach a high level of synergy. The “virtuous cycle” represented on figure 1 describes the strategy and desired contributions of the different project activities to reach integration of new user communities on the infrastructure:

• 
  middleware activities implement a new set of grid and network services that are deployed on the infrastructure by service activities
  
• 
  networking activities use the new grid services for test cases and pilot applications and train established users communities
  
• 
  dissemination uses the results obtained by the established user community to attract new scientific communities
  
• 
  new scientific communities get integrated into the project networking activities
  
• 
  new scientific communities involved in networking activities bring new requirements which are transmitted to the project middleware activities. Middleware activities implement new grid services.
  

Figure 1: graphical representation of the virtuous cycle extracted from [1].

This feedback loop illustrates the whole project strategy for successfully reaching new scientific communities. In the rest of the document, the word generic will be used to describe the applications deployed on EGEE infrastructure by these new communities to differentiate them from the pilot applications in High Energy Physics and Life Sciences which were planned from project day 1.

3.1Implementation

3.1.1Main activities involved in the virtuous cycle

Here we describe the implementation of the virtuous cycle within the project. In the following, we will use the EGEE activity acronyms as listed below:

• 
  Networking Activities
  
  1. 
     NA1: Management of the Integrated Infrastructure Initiative project
     
  2. 
     NA2: Dissemination and Outreach
     
  3. 
     NA3: Training and Induction
     
  4. 
     NA4: Application Identification and Support
     
  5. 
     NA5: Policy and International Cooperation
     
• 
  Service Activities
  
  1. 
     SA1: Grid Operations, Support and Management
     
  2. 
     SA2: Network resource provision
     
• 
  Joint Research Activities
  
  1. 
     JRA1: Middleware Engineering and Integration Research
     
  2. 
     JRA2: Quality Assurance
     
  3. 
     JRA3: Security
     
  4. 
     JRA4: Network Services Development
     

Key actors of the virtuous cycle are SA1/SA2, NA4, NA3, NA2, JRA1 and JRA3:

• 
  SA1 and SA2 contribute by delivering a high quality service with relevant facilities and supporting the user communities in running their applications.
  
• 
  NA4 ensure that software developers are engaged in building relevant applications and aid in their deployment.
  
• 
  NA3 conduct various levels of training that equip the users to work successfully.
  
• 
  NA2 ensures that information regarding the development and services of EGEE are actively communicated to the widest possible audience.
  
• 
  JRA1 and JRA3 develop the middleware services needed for successful application deployment.
  

NA1 is also involved due to the many connections it has established with organisations throughout Europe and beyond and via the dissemination activities (keynote speeches etc.) given by the project management. NA1 also contributes by identifying the strategic directions for the project as a whole.

It is crucial that these activities effectively coordinate and complement each other. These activities are involved on three main axes.

3.1.2SA1 contribution to the virtuous cycle

The grid operations activity, SA1, is central to the EGEE project. It is responsible for building and operating the grid infrastructure, supporting its deployment and operation and supporting the user communities in running their applications. As such, the activity has strong interactions with most of the other activities, setting requirements, providing feedback and collaborating on areas of common interest and need.

In the following paragraphs we describe these various interactions and collaborations.

The interactions of SA1 with the middleware activities (JRA1, JRA3 and JRA4) are some of the most important in the project. The feedback from the operations experience is crucial in making progress towards a set of robust, secure, functional, and usable middleware. The primary interaction with middleware is through the JRA1 activity - the middleware developers. Even before the beginning of the project there were regular and frequent meetings between the SA1 and JRA1 groups to make sure that essential groundwork was laid. This included coordination and discussion of build systems, test frameworks, and the interaction on a practical level between middleware development and testing within JRA1 and the subsequent certification with SA1. The discussions covered the process of how middleware releases should be packaged and deployed. In addition, an important discussion was to bring to the attention of the developers and integration teams, the existing knowledge and experience based on the current generation of middleware, and how the existing problems and issues needed to be addressed for a long term deployment of second generation middleware. These meetings continue on a regular basis, as well as frequent informal discussions on practicalities between the teams. The Project Technical Forum (PTF) is also used as a forum in which the essential issues and requirements of the operations activities are expressed and prioritized to the middleware developers. For JRA4 - network development - the situation is simpler. Coordination is done between the activities either through the JRA4 interaction with JRA1 (for example in the PTF) or through specific focus meetings. There has as yet been little activity in this area. JRA3 has the responsibility to define and implement the security architecture. This is done by working primarily with JRA1 through the Middleware Security Working Group (MWSG). SA1 provides members of the MWSG and is the forum where the operational issues affected by that activity and proposals can be discussed. Security is a more complex area, however, and there are a number of activities in common:

• 
  Operational security. This is the responsibility of SA1, and is implemented through an operational security team, led by SA1 at CERN, and coordinating with site security officers at the participating sites. This team deals with operational issues and procedures such as incident response, auditing, and problem follow-up. JRA3 has a role as providers of tools and infrastructure to help support this team, and a responsibility to ensure that appropriate management, logging, and auditing are in place within the middleware components delivered from JRA1 and JRA3.
  
• 
  Security policy. This is a broader area and is addressed by the Joint Security Policy Group. This group grew out of the LCG security group, which mainly dealt with policy issues. From the start that group had members from outside LCG, and it was natural to expand the scope to include as many of the security people from as many grid projects as possible that need to collaborate. Thus, not only does it encompass LCG and EGEE but also formally includes Grid3/Open Science Grid in the US. The role of this body is to work on and agree on various policy issues, from user registration through acceptable use agreements, etc.
  
• 
  Certification Authorities. JRA3 has contributed to the set up the EUGridPMA, which agrees new Certification Authorities. It provides the accepted list to SA1, which deploys the list to the sites in the infrastructure. In addition, SA1 partners act as the registration authorities to many of the CA's. Many SA1 partners also operate and manage the CA services.
  

 

SA1 provides the communications channel to JRA1 and JRA3 for all operations and service activities and SA1 works together with SA2 through joint meetings to discuss operational requirements on networking provision. SA2 provide the liaison to the network providers - GEANT and the National Research and Educational Networks. However, at the operational level, it is the SA1 people who manage the large resource centres who are in contact with the network operators on a daily basis through trouble reporting and addressing operational issues. SA1 and SA2 work together to define service level requirements and agreements that the applications need from the networks. This effort is at a very early stage.

SA1 and NA4 work very closely together to understand the needs of the applications and how they should use the grid infrastructure. The primary mechanism for this interaction is a joint SA1/NA4 group that has been set up to address specific issues:

• 
  What are the application requirements on the infrastructure?
  
• 
  What grid services do the applications need?
  
• 
  What resources do the applications bring, and what resources do they require?
  

The Regional Operation Centre managers are responsible for taking these requirements back to their constituencies and negotiating for the resources and services to be provided. The Core Infrastructure Centres (CICs) are responsible for providing the basic grid services, VO services etc. needed by the applications. On the other hand the VO themselves are responsible for managing the membership of their VO. Within LCG it was recognized very early on that there was a need for a team to work directly with the applications to help integrate the grid middleware with the experiment software. This "Experiment Integration Support" (EIS) team is a group of people expert in both grid middleware and application domains who are able to bridge the gap between the two. Given the present state of knowledge, this group is invaluable. The need for this kind of team is even more important for new applications that are less familiar with grid technology than the HEP experiments. Resources for this are not adequately available within the project, and this should be recognized as an essential requirement for the future. The SA1 EIS team has also been heavily involved in providing training for new applications. This is an important function, but a full provision of needed training goes beyond the scope of the resources that this team has. Again this is something that must be addressed.

In relation to NA3, SA1 has many and varied training needs. In addition, SA1 people contribute as trainers to training courses set up and provided by partners, coordinated through NA3. Some specific training needs for SA1 include:

• 
  Basic system administration for general services
  
• 
  OpenPBS, LSF, Condor system administration
  
• 
  Installing an EGEE site: LCFGng and Manual Installation
  
• 
  Running and monitoring basic Grid services
  
• 
  Integration of specific batch systems in an EGEE CE
  
• 
  Security and VO Management
  
• 
  Monitoring Tools
  

Concerning interaction with JRA2, SA1 provides a member of the Quality Assurance group that coordinates the quality control activities. As part of this role SA1 will provide requirements on metrics and issues that need to be quality controlled within other activities (such as in JRA1, JRA3), as well as providing metrics that the project can use to measure the progress and achievements of SA1 itself.

 

Finally, SA1 works very closely with other grid projects outside of EGEE, as well as supporting many participants in the EGEE grid service that are not EGEE project partners. In this area we see:

• 
  Collaborating institutions in: US, Canada, Taiwan, China, India, Pakistan, and industrial partners Hewlett Packard;
  
• 
  Collaborations with Grid3/Open Science Grid in the USA. Here the collaboration is at many levels: we have joint security policy and operations groups; we are collaborating on grid operations and monitoring; we are investigating ways in which to build true interoperability between the grids. In addition, there are several joint projects on storage management and other common fabric-grid interfaces, and joint ventures on standards within GGF.
  

 

3.1.3NA4 contribution to the virtuous cycle

NA4 acts as conductor for all actions related to user communities, in strong relationship with the other activities. Figure 2 provides another perspective on the virtuous cycle showing the contribution of the different project activities.

Figure 2: main activities involved in the virtuous circle

As defined in the Technical Annex [1], the Application Identification and Support activity supports the induction of new users, new communities and new virtual organisations into EGEE community. Tasks are:

• 
  New communities identification: to identify, through a well-defined procedure a portfolio of early user applications from a broad range of application sectors from academia, industry and commerce. This task is described in details in chapter 4.
  
• 
  New communities integration: to allocate a significant and well trained workforce to each new selected community to support development and production use of all of its applications on the EGEE infrastructure, to develop and disseminate appropriate information proactively addressing the needs of new user communities, to monitor the integration process and optimize cross-application fertilization to define common application interfaces and tools to assimilate and evaluate records of the work and provide information to the requirements and planning activities. This task is described in details in chapter 5.
  

After identification of new communities and determination of applications eligible for EGEE by the EGAAP committee, NA4 helps users in carrying and “gridifying” their applications, and in their execution inside virtual organisations. If necessary, requirements are produced to guide the middleware evolution to better fulfil the user's needs. NA4 also participates strongly in quality processes to ensure a robust production grid.

3.1.4From requirements to a high quality production platform

A quality process has been designed to produce robust middleware and to influence the operation of the production grid infrastructure in order to enhance services.  It will help demonstrate that the infrastructure is scalable and capable to host a wide range of applications. Particularly, it takes into account requirements of new users, by preparing functional test cases for each use case provided by these users. Figure 3 illustrates the roles and interactions of the different project activities to reach the common goal of a high quality production platform.

Pilot applications have been also defined to test the acceptability of new versions of the middleware, before certification by the production teams:

Figure 3: from requirements to high quality production platform

Metrics have been defined to continually monitor the quality of the grid environment provided by EGEE.

NB: a special role has been given to two pilot application areas – Particle Physics and Biomedical sciences. These two communities are already grid-aware and ready to deploy challenging real applications at the very beginning of EGEE. They are providing invaluable feedback to the whole infrastructure.

3.1.5Accompanying users

Users are widely supported by activities as illustrated on figure 4:

- training courses by experts, managed by NA3

- user support on porting applications to the EGEE infrastructure by NA4

- help desk & operation support via SA1:

Figure 4: accompanying users in the virtuous circle

3.1.6Virtual organisations of user communities and resources

An important role of NA4 is to determine what user communities need in terms of resources and to define, with the help of SA1 and SA2, a set of Virtual Organisations (VOs). These VOs are collections of individuals sharing direct access to computers, software and data. Each virtual organization has its own view of the infrastructure services and resources as shown on figure 5. The figure illustrates also that the VO administration is going to require a permanent dialog between NA4 VO manager and the site administrators where VO services and resources are hosted.

Figure 5: virtual organisation management in the virtuous circle