Strategies for Managing Electronic Records

WHAT IS A RECORD?

Why do we need to ask this question? After all the profession has been comfortable with the definition of the record for many decades. As some archivists might say, I know a record when I see it. Well, in fact, seeing or viewing the record is part of the problem, and it is why some archivists are suggesting the profession needs a more precise definition of a record.

As discussed earlier, the creation and retention of complete and inviolate records documenting business events are not the primary objectives of most transaction processing systems. In an environment where records often exist as logical and not physical entities, and where data documenting a business event is incomplete, volatile, and reflects primarily current or near-current data values, archivists are attempting to construct a conceptual model of a record that includes enough detail to permit one to describe and identify a record even though it cannot be viewed or accurately and completely represented as a physical object. The ultimate objective is to define a record with enough precision to inform systems designers when records are created and what kind of data needs to be captured. In addition, archivists recognize that they need to differentiate the concept of a record from the numerous, other forms of documentation, and to distinguish the mission of the archivist/records manager from that of other information and data professionals. Archivists increasingly are aware that they must be able to articulate to administrators, information technologists and other potential partners how records differ from other digital objects, and why it is important to capture and manage records.
Definition of a Record

So, what are records? How are records different from other types of recorded documentation, such as data, information, documents and knowledge?

Organizations collect, create, and use a wide variety of recorded documentation. There is data or the “raw facts about the organization and its business transactions.” ¹³ There is information, defined as "data that has been refined and organized by processing and purposeful intelligence."¹⁴ There are documents or "a grouping of formatted information objects that can be accessed and used by a person." ¹⁵ More recently there is knowledge, which is defined as something more than information because it includes the expertise, logic and reasoning developed by accomplished experts in a specific field to solve problems and make decisions. ¹⁶
Archivists argue that a record is a specific and unique type of information quite different in its creation and purpose than any of these other types of recorded documentation. Archivists have identified two distinguishing characteristics of records. First of all, records reflect business processes or individual activities; a record is not just a collection of data, but is the consequence or product of an event. Of course, this is not new concept; older definitions identify records with a process or an activity. What is new is the emphasis on defining more precisely and conceptually when the record is created by the business event or personal activity. The other part of the definition of a record stresses that records provide evidence of these transactions or activities. In other words, recorded documentation cannot qualify as a record unless certain evidence about the content and structure of the document and the context of its creation are present and available. Now again, this is not exactly a new concept. However, these newer definitions provide much more detail than ever before on the type and exact nature of this evidence. This topic will be explored in more detail in the section on metadata. ¹⁷
Within the profession, there is a growing consensus around the definition of a record as: Recorded information in any form created or received and maintained by an organization, person or system in the transaction of business or the conduct of affairs and kept in a widely accessible form as evidence of such activity. ¹⁸ This definition, however, must be recognized as only the starting point for a complete and useful definition. To be meaningful, it must be accompanied by a detailed set of definitions that identify when a record is created and what type of evidence is required to create reliable and authentic records. ¹⁹ In addition, archivists are recognizing that this definition needs to articulate the cultural, historical and heritage dimensions of archives. The dialogue on this issue is often presently framed in terms of describing “archives as evidence” and “archives as memory.” ²⁰

HOW WILL ARCHIVISTS IDENTIFY AND APPRAISE RECORDS?
If physically reviewing records or browsing automated systems is not a realistic strategy, how will archivists identify and appraise a record? Most archivists working with electronic records would respond by asserting that the answer is derived from the definition of the record and involves tracing the record back to the process that created it. This again is not a new revelation. Archivists have been writing about this concept for well over a decade, most notably in the context of an appraisal theory and methodology based on functions and activities. ²¹
As any archivist knows, traditional appraisal theory in North America focuses on finding value in records, these values commonly expressed as primary and secondary, with secondary values being divided into evidential and informational values. This methodology, most closely identified with the writings of Theodore Schellenberg, placed special emphasis on the archivist’s responsibility for appraising records to identify secondary, research values, as his definition of archives makes clear: “Those records of any public or private institution which are adjudged worthy of permanent preservation for reference and secondary purposes.” ²² For many archivists, the search for research value remains at the heart of the appraisal process. Increasingly, however, critics of this appraisal methodology have argued that by defining appraisal primarily in terms of secondary research value based largely on content analysis, the Schellenberg model does not provide a proper answer for why we appraise records. Critics of Schellenberg have put forward four arguments to support this judgment. In the first place, they argue that predicting or anticipating research needs or trends is not a realistic goal, and at best will mean the archivist will remain “nothing more than a weathervane moving by the changing winds of historiography.” ²³ Secondly, critics assert that content-oriented appraisal cannot give a true or, even, representative image of society. ²⁴ Thirdly, archivists who support Hilary Jenkinson’s theory on the nature of archives assert that selection by content to support research is in direct conflict with basic archival theory and the very nature of archives. ²⁵ Finally, critics of traditional appraisal methodology assert that in the modern world of high volume documentation and of electronic records that exist as logical and not physical entities, archivists can no longer hope to focus on the record and appraisal by content. ²⁶
So, what have archivists offered in its place? Although specific appraisal theories and methodologies abound, almost all major commentators agree that a principal objective or aim of archival appraisal must be the preservation of evidence ²⁷ documenting the functions, processes, activities, and transactions ²⁸ undertaken and completed by the institution or individual. In the words of two prominent commentators on appraisal: “Archivists are Servants of Evidence,” ²⁹ and “Evidence is an aim…of archival appraisal.” ³⁰ And where is this evidence to be found? Most archivists writing on this topic, particularly as it relates to the appraisal of electronic records, have advocated a functional appraisal model. Proponents of functional appraisal assert that in the search for evidence and value, the most accurate and complete documentation will be provided by examining the function, activity, and transaction that generated the record, rather the record itself. In short, supporters of functional appraisal argue that the context and not the content of the record must be the starting point in the search for evidence and hence value. ³¹

Business Process Modeling

Clearly, a theoretical basis for identifying records based on function is in place. What has been slow to develop is a methodology for actually undertaking and completing a functional analysis of business processes or personal activities. However, there have been some interesting and promising beginnings. Most promising are the arguments that suggest that a source of information on how to gain this knowledge of processes and activities can be found in the writings of the discipline known as systems analysis. Systems analysis has been defined as “the study of the problems and needs of a business to determine how the business systems and information technology can best solve the problem and accomplish improvements for the business. The products of this activity may be improved business processes, improved information systems, or new or improved computer applications.” ³² Clearly emphasized in this definition is a focus on understanding and analyzing business processes as a means to improving the system, whether that system is defined as the business system or the information system. Without question, archivists and systems analysts have something in common; both regard an understanding of business requirements as critical to the design of systems.

The recognition that archivists and systems analysts share a common concern in the identification of business requirements has led some archivists to emulate the methodology and techniques analysts employ in modeling system processes. What they have discovered is that the methodology and techniques analysts employ in reviewing system processes provide useful tools in the quest to identify records. One such methodology is a popular and widely practiced technique known as “modern structured analysis.” ³³ This form of analysis has been defined as “a process-centered technique that is used to model business requirements for a system. The models are structured pictures that illustrate the processes, inputs, outputs, and files required to respond to business events.” ³⁴ The products of this analysis, business process models, depict the business functions and transactions and the inputs and outputs required to respond to business events. Business process models can further be broken down into business function decomposition diagrams, business events diagrams, and business process data flow models. The value of business models for archivists is that they can depict precisely when, where and how record creation occurs. They provide the archivist a conceptual model based on depiction of real-life activities of the context for creation, and consequently provide the information needed to precisely describe and define for system designers what pieces of data need to be captured as evidence of the business transaction.
It is not illogical or too much of a jump to arrive at an overall strategy that views conceptual model building as a methodology that will allow archivists to deal with many or most of the issues the profession faces in attempting managing records in automated environments. For example, some archivists are suggesting that rather than physically reviewing records and systems to conduct basic activities such as appraisal and description, archivists should be creating and employing conceptual models designed to analyze and document record systems. Thus appraisal of records could be still be undertaken by employing traditional appraisal values, but the analysis would be based on conceptual models of the processes and records rather than on a physical review of data content. Evidential values could be derived from business process and metadata models, and informational values from reviewing data and metadata models. In documenting records, some archivists are suggesting that a complete, authentic and reliable record could be captured not by physically reviewing the record but by analyzing metadata and data models and comparing the results to an established set of metadata recordkeeping requirements. ³⁵

WHAT DOCUMENTATION MUST BE PRESENT TO CREATE A RELIABLE AND AUTHENTIC RECORD?
As indicated earlier, the concept of evidence is a very critical element within the definition of a record. Without sufficient documentation describing the content of the record and the context of its creation, the record loses it value as evidence and in some cases ceases to be a record at all. Now again, the need for supporting documentation is not a new requirement created by electronic records. The emergence of electronic records, however, has created some new problems and challenges for archivists attempting to preserve evidence.
Challenges and Issues

The primary challenge is associated with the basic but extremely important recognition that unlike paper documents, electronic records are logically constructed and often “virtual” entities. Consequently, electronic documents cannot be viewed in the same way as paper records, where so much of the content, context and structural metadata is embedded in or is part of the record. In automated systems, the vital metadata, if it exists at all, may or may not be physically associated with the content data. Vital links between metadata and the record content data may exist only in computer software programs. In some cases, the metadata may actually not be a part of the automated system at all, but may exist only as a paper document totally disassociated with the records it is describing.

Archivists also discovered that system metadata as typically defined by systems designers and technologists is often not as complete as necessary to describe a record. Transaction logs maintained in typical TPS do contain some critical data on updates and revisions, but on the whole, archivists generally agree that these logs do not provide sufficient evidence. Of particular concern is the relative lack of metadata related to the context of creation and use - metadata that addresses the questions of why the record was created, who were the users of the record, and who had custody of the record. The availability of this contextual metadata, archivists argue, could make the difference between a useful and a useless record, particularly when viewed over longer periods of time. Another deficiency from a recordkeeping perspective of typical system metadata is the absence of some critical documentation on the structure of the record. Of particular importance is structural metadata describing how to open and read a record as it was originally created and viewed. ³⁶ Taken as a whole, the absence of critical metadata has meant, as one archivist has noted, that "most collections of electronic data, electronic documents, or information are not records because they cannot qualify as evidence." ³⁷

The recognition that critical documentation may never have been created or may not be available with the content of the record has caused archivists to begin rethinking strategies for documenting records. Specifically, three strategies have been prominently featured.

Identification of Recordkeeping Metadata

The challenge receiving the most attention from archivists is the determination of which types of metadata are needed to meet requirements for recordkeeping. Archivists quickly recognized that before they could properly describe and identify records (comprising content data and the evidence or metadata documenting context, content, and structure), they needed first to precisely define what types or categories of metadata must be captured.

The first research project designed to identify key recordkeeping metadata was the electronic records project undertaken in the period from 1993-1996 at the University of Pittsburgh with funding by the National Historical Publications and Records Commission. The primary objective of the Pittsburgh project was to develop a statement of requirements for ensuring the preservation of evidence in recordkeeping. One of the products of this project was a set of metadata specifications "designed to satisfy the functional requirements for evidence," and to "guarantee that the data object will be usable over time, be accessible by its creator, and have properties required to be fully trustworthy as evidence and for purposes of executing business." Pitt project personnel identified sixty-seven metadata items organized into six categories or layers. ³⁸
Since the emergence of the Pittsburgh metadata specifications, several other institutions or projects have put forward their own set of recordkeeping metadata. Among the most prominent are those proposed by the National Archives of Australia and Canada, the State Archives of Victoria (Australia) and New South Wales, the United States Department of Defense, the University of British Columbia School of Library and Information Science, the Indiana University Archives, and most recently by personnel associated with the SPIRT Project and the InterPARES Project. ³⁹ Most of these lists of recordkeeping metadata differ noticeably in the way they are organized, in the amount of description they provide on the specifications, and, most importantly, in the specific items they list as essential or mandatory. At present, there is no real consensus on a core set of metadata specifications or a set of minimum metatdata standards; as yet, there is nothing for recordkeeping that resembles or has been accepted in the way that say the Dublin Core Metadata ⁴⁰ has been embraced by the library community.
One can discern, however, some growing consensus among archivists about certain key issues relating to metadata. For example, there is general agreement among archivists that records require their own unique, particular kind of metadata that goes beyond what is required in the Dublin Core standard. More specifically, archivists stress that records require more metadata documenting the context of creation if they are to be understood and interpreted, particularly over long periods of time. There is also agreement about the basic categories of metadata that systems should capture and retain. For example, most record metadata lists include various pieces of documentation describing the context of creation. This contextual metadata typically includes information on the agents involved in creating, receiving, and transmitting the record; the date of receipt; and the relationship of the record to the specific business processes and to related records. There is also general agreement that the metadata model include some documentation on terms and conditions for access and use, and that the system document use history. Most lists of metadata specifications also include data on the disposition of the record, such disposal authorization and date, and a disposal action history. Predictably, most lists also include metadata describing the record content, such as information on title of the record, date of creation, and subject. Finally, the majority of record metadata lists include information on the structure of the record, most notably documentation on how the record is encoded, how the record can be rendered, and how the content of the record is structured. In short, most metadata specifications include documentation in varying degrees of detail on the content and structure of the record and the context of its creation.
Timing of Archival Input

A second issue relating to the documentation of electronic records involves the determination of when, at what point, archivists should become actively involved in the process. Many archivists have come to the conclusion that the profession must be more proactive and be involved at the systems design stage. Proponents of this position argue that documentation of business processes cannot be postponed until the point when records become inactive; to be effective, description must take place over the life of the record. Only in this way, it is argued, can archivists hope to document business transactions throughout their life cycle. Advocates of this position warn that if procedures for early identification and maintenance are not established, records, and particularly electronic records, may never survive or even be created. ⁴¹

Other archivists, however, have warned that by introducing metadata requirements designed to satisfy the needs of future users, archivists compromise the impartiality of the records. And if “the impartiality of the metadata is compromised, their value as evidence will be compromised, which means, ultimately, that the underlying objective of metadata strategies-the preservation of evidence-will be defeated.” ⁴² In short, advocates of this position argue that, “archival participation in the design and maintenance of metadata systems must be driven by the need to preserve them as archival documents, that is, as evidence of actions and transactions, not as descriptive tools.” ⁴³ Still, this must be regarded as the minority position at this point. Most archivists involved in studying this problem have agreed that early intervention, preferably at the systems design stage, is the only viable documentation strategy.
Value of Traditional Finding Aids

Finally, archivists are debating whether traditional methods for describing archival records (descriptive inventories, guides, and other finding aids created after the records are transferred to the archives) are adequate and useful tools for documenting electronic records. Critics of traditional strategies for describing electronic records identify three major reasons for adopting other methods. In the first place, critics claim that traditional descriptive methodologies that depend upon physically reviewing records, files and series to identify content and context are not viable in the world of electronic records. In addition, they argue that traditional prose narratives and descriptions of data structures cannot possibly describe the multitude of record linkages or reflect the relationships between and among transactions in automated systems. To properly describe these complex record systems, they recommend that much more dynamic and interactive documentation strategies be employed. Finally, proponents of this position of change argue that a viable system of documenting business processes already exists in the form of record system metadata. Systems designers and programmers routinely generate documentation on the content and structure of the systems and programs they create. Why not, it is suggested, make this metadata/metatag system the basis for describing electronic records? Why not consider a shift from creating descriptive information to capturing, managing, and adding value to system metadata? ⁴⁴

Naturally, not all archivists agree with the strategy described above. Their arguments focus on the themes of the authenticating role and the unique and vital contributions of traditional archival description. For example, Luciana Duranti argues that the “verification of the authenticity of electronic records over the long term will have to rely on one thing and one thing only: their archival description.” ⁴⁵ Traditional arrangement and description verify authenticity, according to Duranti, by preserving the network of administrative and documentary relationships. “Administrative relationships are revealed and preserved through the writing of the administrative history of the archival fonds and its parts, including the preservation and custodial history. Documentary relationships are revealed and preserved through the identification of the levels of arrangement of the fonds and their representation in structured descriptions.” ⁴⁶ Another argument put forward in defense of traditional archival description is that it performs a vital function that system metadata cannot. Advocates of this position argue that because the scope and context of system metadata is “comparatively narrow, metadata circumscribe and atomize these various contexts. Archival description, on the other hand, enlarges and integrates them. In so doing it reveals continuities and discontinuities in the matrix of function, structure, and record-keeping over time.” ⁴⁷

WHAT IS A RECORDKEEEPING SYSTEM?
Traditional records management methodology focuses on managing and controlling records, usually as part of a record series. Newer, revised definitions of the objectives of records management, however, focus on evaluating the processes creating records and the systems for managing them. For example, one prominent definition identifies the goals of records management as the identification and capture of records generated in the context of business processes, and the creation of systems that manage and preserve these records. ⁴⁸ In essence, the new definition is concerned less with managing records and is more focused on defining and assisting in the management of recordkeeping systems.
Identification of Recordkeeping Requirements

What is a recordkeeping system, and how is it different from other types of systems, such as transaction processing, information management, and document management systems? In this context, the term “system” is used in its broadest sense to depict the organizational mission, business processes, policies, procedures, practices, and human and automated mechanisms to bring about desired ends, which in this case is trustworthy recordkeeping. ⁴⁹ To address these questions, archivists have designated the identification of a set of requirements for a recordkeeping systems as one of the profession's critical, initial tasks. The University of Pittsburgh School of Information Science conducted the first systematic research on this topic. The Pitt project established a set of functional requirements for recordkeeping that addressed three levels of requirements: the organizational level, the recordkeeping system level, and the record level. Within these levels, they established five categories – Conscientious Organization, Accountable Recordkeeping System, Captured Records, Maintained Records and Useable Records – and within these categories twenty requirements, which they claimed “are identified in law, regulation, and best practices throughout society as the fundamental properties" of evidential records. ⁵⁰ Since the creation of the Pittsburgh document, numerous other projects have produced lists of requirements for recordkeeping systems. Among the most prominent requirements are those created by the United States Department of Defense; the National Archives of Australia and Canada; the State Archives of Victoria (Australia), New York, Delaware, and Kansas; and at the University of British Columbia and Indiana University. ⁵¹

As with the creation of metadata specifications, the various lists of recordkeeping requirements differ, in some cases significantly. There is general agreement and a growing consensus, however, on several critical points. For example, the majority of archivists agree that "not all information systems are recordkeeping systems, " and that "recordkeeping systems are a special kind of information system" (in this instance, “system” is used at the software application level). ⁵² Most of the lists of recordkeeping requirements also agree on the basic types or categories of functionality a recordkeeping system must possess. These typically include requirements that the system be compliant by meeting legal and administrative requirements, national and international standards, and best practices for recordkeeping. Many lists of recordkeeping requirements also specify that the system be accountable and reliable. Specific requirements included in this category are that system policies and procedures be well documented, that system hardware and software be regularly tested to ensure that consistent and accurate business records are created, and that system audit trails be maintained for all business processes. All lists of requirements specify that the system capture all business records and all essential metadata related to that business process. Similarly, all lists of recordkeeping system requirements mandate that the system maintain and manage the business record. Typical requirements in this category include the specification that the system maintain inviolate records protected from accidental or intentional deletion or alteration; that the system ensure that all components of a record, including relevant metadata, notes, attachments, etc., can be accessed, displayed and managed as a unit or complete record of a business process; and that the system include an authorized disposition plan that is implemented as needed. Finally, all sets of requirements specify that the recordkeeping system ensure the future usability of the business records. As part of this requirement, systems must be capable of recreating the content of records and any relevant metadata within a new system without loss of any vital information.
Relationship of Recordkeeping to Other Types of Systems

How will these recordkeeping systems function in relation to other data and information systems, like TPS, DBMS and Management Information Systems (MIS)? In other words, will recordkeeping functionality be built into the active transactions processing system, or will records be managed in a completely separate system or environment, or might there be a combination of these two approaches? At present there is no consensus on this issue, largely because there have been no significant tests of the costs and effectiveness of building recordkeeping systems in a variety of automated environments. Conceptually, some archivists argue that it may likely be easier to manage records in their own separate environment, much in the same way that Management Information Systems manage information and decision support data. To populate decision support systems and data warehouses, data is extracted from the TPS and moved to a separate automated system, which is typically managed by a separate staff operating with its own set of policies and procedures. Some archivists argue that this same strategy could be applied to create recordkeeping systems. As records are created in the TPS, they would be captured and moved to a separate but linked environment managed according to its own set of requirements by a staff of records managers. The proponents of the view consider it extremely important that records are maintained by an independent organization with no special interests in the records and by a staff trained in archives and records management. ⁵³

In the final estimation, however, the strategy employed for building recordkeeping functionality may well be determined largely on the basis of the nature and requirements of the specific system environment under review. As one colleague stated to this author: “in less structured environments, such as those where e-mail and electronic documents are exchanged without the benefit of defined work flow or structured work processes, the need for a separate, well defined recordkeeping environment may be essential to the capture and preservation of records. In other systems defined by structured business processes, however, the design parameters might be such that recordkeeping could be incorporated inside the overall design of the existing system.” ⁵⁴ In other words, every environment is different and will demand different approaches. Consequently the “one-size fits all” strategy for designing recordkeeping systems will likely not be effective.

PRESERVATION OF RECORDS
What is the best strategy for preserving digital objects over time?

This has proven to be a very difficult question to answer. To date, archivists, librarians and technologists have identified the challenges, but have been far less successful creating viable strategies for solving or addressing the issues.

Definitions and Issues

Professionals working on long-term preservation of digital objects have generally agreed on a definition of the overall goal of digital preservation as the ability to ensure readability and intelligibility in order to facilitate data exchange over time. In this context, readability is defined as digital objects or composite objects that can be processed on a computer system or device other than the one that initially created them or on which they are currently stored. Intelligibility can defined as the requirement that the digital information be comprehensible to a human being. ⁵⁵

Archivists intent on preserving records have stressed that any strategy must also preserve the authenticity and integrity of records, which translates into requirements for preserving formal document structure (structural characteristics) and descriptive metadata. Consequently, as Charles Dollar has written, archival preservation demands that records be more than readable and intelligible; records preserved for the future use must also be identifiable, encapsulated, retrievable, reconstructable, understandable and authentic. ⁵⁶

Professionals working on preservation generally also agree on the problems or challenges. They are typically described under three categories: hardware obsolescence, software dependence, and storage medium deterioration. While all three are eventually lethal to the long-term survival of digital objects, most experts agree that it is software dependence or "the fact that digital documents are in general dependent on application software to make them accessible and meaningful" that presents the greatest challenge. ⁵⁷

Moving from the identification of goals and issues to the formulation of specific strategies to address these challenges, there is far less uniformity of opinion. Over the last two decades, quite a number of credible and not so credible strategies for the long-term preservation of digital documentation have been proposed. The digital preservation strategies most prominently discussed in the literature include creating computer museums, copying to paper or microfilm, converting to standard formats or into software independent modes, the "emulation" strategy, and the conversion or migration of records. Let us now look at these strategies in more detail.
Computer Museums

One preservation strategy recommends that society create museums of obsolescent hardware and software, as a means of maintaining continuing access to digital materials. On the whole, most experts have dismissed this strategy as unrealistic and too expensive. One critic of this strategy, Jeff Rothenberg, observes that this strategy ignores the fact that data will have to be transferred to new media that did not exist when the document's original computer was current. "The museum approach would therefore require building unique new device interfaces between every new medium and obsolete computer." ⁵⁸ Another critic observes, “the likelihood of keeping any piece of machinery running for many decades is simply not very high, since replacement parts, chips, and software could not be easily reproduced. A computer system is far more complex than a steam locomotive or shuttle loom.” ⁵⁹ Finally, critics have observed that this strategy is not compatible with the widely held notion that data and information in legacy systems should be easily accessible over time and integrated with current information and technology architectures. ⁶⁰