GridCoord DoW

Hungary

Summary of activities and strengths

The Grid research activities in Hungary have been developed in two main directions:

• 
  Infrastructure related projects
  
• 
  Application oriented projects
  

The infrastructure related projects have been aimed at developing Grid infrastructures that can be used for e-science by the Hungarian research community. These projects included middleware research as well as programming environment and Grid portal research. As a result of these Grid projects four different kinds of Grid systems have been successfully developed and tested in Hungary:

• 
  Hungarian ClusterGrid
  
• 
  HunGrid
  
• 
  JGrid
  
• 
  SZTAKI Desktop Grid
  

The other type of projects have been trying to exploring various Grid applications. These projects were run in the field of

• 
  Chemistry
  
• 
  Molecule pre-production for medicine design
  
• 
  Data-mining
  
• 
  Agriculture
  
• 
  Nuclear power station modelling
  
• 
  Brain research
  
• 
  Astrophysics
  
• 
  Particle physics research
  
• 
  Engineering
  
• 
  Mathematics
  

In many projects the grid middleware and infrastructure research were conducted together with application development.

The scientific strengths are mainly in the following topics:

• 
  Production grid design and deployment, including Condor, Globus, Jini and Boinc-based infrastructures;
  
• 
  Resource management and monitoring;
  
• 
  High-performance programming environment;
  
• 
  Portals and Applications.
  

Key projects

Hungarian ClusterGrid: national project on Grid infrastructure (NIIF/ HUNGARNET)

This project was initiated by the Ministry of Education (OM) in 2002 and currently funded by IHM. The project is now called as the ClusterGrid project (http://www.clustergrid.niif.hu/ ). It aims to integrate the Intel processor based PC labs of the Hungarian higher educational institutions into a single, large, countrywide interconnected set of clusters. The PCs are provided by participating Hungarian institutes, such as universities, polytechnics or public libraries. The central infrastructure and the coordination are provided by NIIF/HUNGARNET. All the contributors use their PCs for their own purposes (such as educational or office-like purposes) during the official working hours, and offer their infrastructure for high-throughput computation whenever they do not use them, i.e. during the nights and the unoccupied week-ends. The combined use of "day-shift" (i.e. individual mode) and "night-shift" (i.e. grid mode) enables to utilize CPU cycles (which would have been lost anyway) to provide firm computational infrastructure to the national research community. The applied technologies are based on Condor and SGE for task distribution and VPN for security. Currently more than 1400 PCs of 29 clusters are connected from 20 higher educational institutions.

In order to establish such a production system several grid research and development works have been done:

• 
  Research on VPN technology and its application in the ClusterGrid system
  
• 
  Solving the problem of automatic switch between the two working modes: day-shift and night-shift
  
• 
  Developing a broker system and its connection to the underlying Condor system
  

Hungarian Supercomputing Grid: national project on Grid infrastructure (NIIFI)

In 2000-2001 several supercomputers and PC clusters were installed in Hungary as major supercomputing resources. All were placed at different institutes and were used by a growing user community from academy. However, it turned out that there exist applications where the capacity of individual supercomputers and clusters were not sufficient to solve the problems in reasonable time. The solution for this problem was to connect these high-performance computing resources and to use them as a supercomputing Grid. One of the main goals of the project was to establish a Hungarian Supercomputing Grid (HSG) based on the Hungarian and international results of cluster and Grid computing. The project was strongly related with two already running Hungarian Grid projects (VISSKI and DemoGrid) and several national projects from other countries (Globus, Condor, INFN Grid, UK e-science).

HSG created a layered Grid structure. The top layer was the application layer where a Monte-Carlo method based nuclear physics application was investigated. The user could access the HSG by the Grid portal developed in the project. The application could be developed by the P-GRADE parallel program development environment, which was integrated with Condor in the framework of the project. As a result the user could generate directly Condor jobs (containing parallel PVM or MPI program) from the P-GRADE environment. P-GRADE was also extended with workflow creation and execution mechanisms for Condor and Globus Grids. Condor was used as the Grid level resource manager in the HSG. The basic middleware services came from Globus. The fabric contained the Hungarian supercomputers and clusters connected by the Hungarian academic network. On the supercomputers and clusters local job schedulers like LSF, PBS, Condor, Sun Grid Engine were used. A further achievement of the project was the development of a Grid accounting system.

JGrid: national project on Grid infrastructure (Univ. of Veszprem)

JGrid is a Java/Jini based service-oriented grid middleware and service framework. It is a platform (hardware and OS) independent, dynamic, self-healing grid system that provides wide-area service discovery, security and authentication for service access and communication. Its core services include a Compute service to execute sequential and parallel interactive Java tasks, a Batch service to run C/Fortran/etc jobs on integrated batch runtime systems, and Storage service to provide access to user data. Additional services exist for intelligent service brokering, creating compute clusters, managing workflow execution, and real-time media streaming. The system can execute programs generated automatically by the P-GRADE environment. Future versions will include support for location-based discovery, access from mobile devices, and various business services.

In its first 2 years, the project focused on the infrastructure and core services needed for computing grids. During the next two years it will focus on business, finance, healthcare and entertainment areas. A small testbed is in operation including 3 sites in Hungary. Planning is underway to expand this system to 13-15 sites.

Hungarian Grid: national project on Grid infrastructure (SZTAKI, RMKI)

This project was supported by the Ministry of Information Technology (IHM). The goal of the project was to create an integrated Hungarian Grid infrastructure based on the results of the various other Hungarian Grid projects and international Grid achievements, and to provide a unified portal framework by which all these systems can be accessed in an interoperable way.

Within the framework of this project we have created two more Grid infrastructures for the Hungarian Grid users. The first one is HunGrid that is the Hungarian Virtual organization of the EGEE Grid extended with some Hungarian Grid results:

• 
  P-GRADE portal
  
• 
  Mercury Grid monitor
  

Meanwhile ClusterGrid is a two-shift Grid, HunGrid has been providing a 7/24 service since March 2005. The other advantage of HunGrid compared to ClusterGrid that it can handle very large data files. The extension of EGEE with P-GRADE portal and Mercury Grid monitor enables the users to run complex workflows in HunGrid and to graphically observe on-line the execution of the workflows and their component MPI jobs – feature that is not available for the EGEE users. That was the reason why SEE-Grid (South East European Grid) also has chosen P-GRADE portal as their official portal.

The second new Grid infrastructure developed in the project is the SZTAKI Cluster Grid that serves as a model for the Hungarian companies and institutes how to build their local in-house Grid system. We expect a breakthrough by the introduction of this Grid concept since in this way any community can quickly build its own Grid system with minimal efforts and investment. The previously available Grid infrastructures (HunGrid, ClusterGrid) required special authorization for the users and provided only limited access for members of the scientific community and no access for people in industry and business. All have been changed now by the SZTAKI Cluster Grid concept. Anybody can create his/her own Grid, even a family.

To provide the unified portal framework we have developed the P-GRADE Grid portal that is a workflow-based multi-grid portal. It provides high-level graphical user interface to create and manage workflows. It is a multi-grid portal, i.e. it can handle certificates for several grids and using these certificates can execute jobs of a workflow in several Grids simultaneously. In this way, different Grids become interoperable via the portal. The portal is connected to HunGrid, Hungarian ClusterGrid, SEE-Grid, GridLab, UK NGS and we have demonstrated that all these Grids can work together to solve a single workflow generated by a user who has got certificate to all these Grid systems. P-GRADE portal version 2.1 is used as the official portal of HunGrid and SEE-Grid, and agreed by the UK NGS leadership to be one of the supported portals for the UK NGS.

SuperCluster Grid: User-oriented unification of the Hungarian ClusterGrid and SuperGrid systems (SZTAKI)

This project is supported by the Ministry of Education. As a result of the ClusterGrid project a significant high-performance computing Grid infrastructure has been created in Hungary. The ClusterGrid currently connects about 1100 PCs of 13 higher educational institutions. The system is constantly grows and the final goal is to connect more than 2000 PCs by the end of 2005. Although the ClusterGrid already works as a regular infrastructure service it raises several problems to be solved. The basic goal of this project is to solve these problems by providing new functions for the ClusterGrid like:

• 
  checkpoint handling for PVM and MPI programs,
  
• 
  Grid brokering,
  
• 
  workflow management,
  
• 
  high-level user interfaces,
  
• 
  Grid monitoring,
  
• 
  handling large data files and databases in the Grid,
  
• 
  accounting system support,
  
• 
  application of VPN and IPv6 technologies.
  

In order to provide these new features we adapt the results of the Hungarian SuperGrid project and integrate with the solutions applied in the ClusterGrid project.
The second goal of the project is to develop six different scientific and engineering applications for the ClusterGrid as a demonstration how to exploit the ClusterGrid infrastructure in order to solve complex scientific problems. The applications cover problems in chemistry, architecture, astrophysics, optics, particle physics and biology.

Research projects with a significant industrial impact

Until 2004 all the Grid projects running in Hungary were devoted for the academic community. In 2004 the interest of industry significantly increased for Grid technology and they initiated or actively participated in organizing Grid projects. Three such projects should be mentioned.

MEGA project (BME)

The most important objective of the project is to migrate Grid computing technology and solution from the academic community to the industrial environment and to create and solve real industrial applications in such an industrial utility-based Grid environment.

ADMEToxGrid project (Comgenex Ltd.)

The main purpose of the ADMEToxGrid project is to develop an enterprise grid system that is suitable for predicting the ADME/Tox parameters of millions of compounds in a short time and in a secure manner, while also exploiting the free capacity of the office computers. In addition, the other goal of the project is the development of a grid frame infrastructure that can be applied easily to create a grid system that is suitable for running programs such as parameter predictions. Following the demands expected in the drug industry, data security and fault tolerance play an important role in the design and implementation of the system. The grid provides a seamless way of handling the different program versions - when installing a new program version in the grid, the new parameter values can be updated in the database in a controlled manner. The grid will be tested and validated based on a million compound set from ComGenex’s molecule database. The final grid system will be in operation during ComGenex’s compound manufacturing processes.

Data Mining Grid (SZTAKI)

The goal of the project is to extend existing data mining algorithms with new features that enable the on-line trying of several algorithm options and selecting the best fitting one. Algorithms with large computing needs are also considered and hence the Grid execution mechanism is applied. The project aims at producing a prototype that later can be commercialized.

Desktop Grid project (Econet Holding)

The goal of the project is to extend the applicability of the very successful BOINC-like desktop grid (DG) concept in two directions. On one hand Econet will create a Global desktop Grid system where companies can solve their long-running applications by global desktop grid where individual PC owners can offer their PCs and they will be paid for their PCs by the Grid Infrastructure Service Provider (who will be Econet).
Since companies are anxious about security Econet will develop a highly secure DG system that even banks could use. On the other hand SZTAKI will extend the BOINC concept with the possibility of connecting DG systems in hierarchy which means that not only desktop machines but even complete DGs can offer resources for a higher level DG. This hierarchical construction of local DGs well fits to the organizational hierarchy of many companies and institutes.