Electronic Records Management Guidelines
Rapid changes in technology mean that file formats can become obsolete quickly and cause problems for your records management strategy. A long-term view and careful planning can overcome this risk and ensure that you can meet your legal and operational requirements.
Legally, your records must be trustworthy, complete, accessible, legally admissible in court, and durable for as long as your approved records retention schedules require. For example, you can convert a record to another, more durable format (e.g., from a nearly obsolete software program to a text file) and that copy, as long as it is created in a trustworthy manner, is legally acceptable.
The software in which a file is created usually uses a default format when the file is saved. This is indicated by the file name suffix (e.g., .PDF for portable document format). However, most software allows authors to select from a variety of formats when they save a file. For example, Microsoft Word allows the author to select document [DOC], Rich Text Format [RTF], or text [TXT], as well as other format options. Some software, such as Adobe Acrobat, is designed to convert files from one format to another. The format you choose will affect your long-term records management abilities.
For more information on the legal framework you must consider when developing a file format policy refer to the Legal Framework chapter of these guidelines and the Minnesota State Archives’ Preserving and Disposing of Government Records.
As you consider the file format options available to you, you will need to be familiar with the following concepts:
- Proprietary, Non-Proprietary, Open-Source, and Open Standard File Formats
- File Types and their Associated Formats
- Preservation Options: Conversion and Migration
- Importance of Planning
- Proprietary formats. Proprietary file formats are controlled and supported by just one software developer. Microsoft Word (.DOC) format is on example.
- Non-proprietary formats. These formats are supported by more than one developer and can be accessed with different software systems. eXtensible Markup Language (XML) is a popular non-proprietary format for government records.
- Open Source formats. In general, open source refers to any program whose source code is made available for use or modification as users or other developers see fit. Open source formats are published publicly available specifications for storing data which are often maintained by a standards organization. Open formats can be used by proprietary and open source software alike.
- Open Standard formats. Open standard software formats are created using publicly available specifications (open source formats). Although software source codes remain proprietary, the availability of the standard increases compatibility by allowing other developers to create hardware and software solutions that interact with, or substitute for, other software.
When choosing a file format to use for your electronic records management purposes, it is important to understand how proprietary, non-proprietary, open formats, and open standards may affect the accessibility and accountability of your records over the long term.
The following are brief descriptions of the basic file types and formats you are likely to encounter. Additional information can be found in the Digital Imaging chapter of these guidelines while resources in the Annotated List of Resources provide more detailed information on individual file formats.
Text files. Text files are most often created in word processing software programs. Common file formats for text files include:
- Proprietary formats, such as Microsoft Word files, which carry the extension of the software in which they were created.
- RTF or Rich Text Format files, are supported by a variety of applications and saved with formatting instructions (such as page layout).
- Portable Document Format (PDF) files, which contain an image of the page, including text and graphics. PDF files are widely used for read-only file sharing. Adobe Acrobat is, by far, the most popular PDF file application, although others are available.
- Portable Document Format (PDF/A) files. PDF/A, as standard file format for long-term archiving of electronic documents, is a subset of PDF. Files are 100% self-contained, and do not rely on outside sources for document information. ISO standard: ISO 19005-1:2005.
Graphics files. Graphics files store an image (e.g., photograph, drawing) and are divided into two basic types; vector-based and raster-based.
- Vector-based files store an image as mathematical formulas. Vector image programs use this mathematical formula to display and scale the image without distortion. Common types of vector-based file formats include
- Drawing Interchange Format (DXF) files, which are widely used in computer-aided design software programs, such as those used by engineers and architects.
- Encapsulated PostScript (EPS) files, which are widely used in desktop publishing software programs.
- Computer Graphics Metafile (CGM) files, which are widely used in many image-oriented software programs (e.g., Photoshop) and offer a high degree of durability.
- Shapefiles (SHP), ESRI GIS applications use vector coordinates to store non-topological geometry and attribute information for features.
- Raster-based files store the image as a collection of pixels. Raster graphics are also referred to as bitmapped images. Raster graphics cannot be scaled without distortion. Common types of raster-based file formats include:
- Bitmap (BMP) files are relatively low-quality files used most often in word processing applications. Uncompressed.
- Tagged Image File Format (TIFF) files are usable with many different software programs and are often the format of choice for a high-quality master image. Uncompressed or lossless compression.
- Joint Photographic Experts Group (JPEG) files are very common format for digital photography. JPEGs are also the preferred format for Internet delivery and file sharing of photographs. Lossy compression.
- Joint Photographic Experts Group (JPEG2000). An evolving format with multiple compression techniques based on wavelet technology. Lossless compression.
- Graphics Interchange Format (GIF) files were widely used on the Internet for graphics and logos with areas of solid color. Due to color limitations, photographs are not accurately represented with this format. GIF can also be used for low resolution animations. (PNG has improved on the color limitations of GIF.) Lossless compression.
- Portable Network Graphic (PNG) files, designed to replace GIF, are patent and license free and produce higher quality files than GIF. PNG format is preferred for images that contain text or line art, especially on the Internet. Lossless compression.
Data files. Data files are created in database software programs and are therefore often represented proprietary formats. Data files are divided into fields and tables that contain discrete elements of information. The software builds the relationships between these discrete elements. For example, a customer service database may contain customer name, address, and billing history fields. These fields may be organized into separate tables (e.g., one table for all customer name fields). You may convert data files to a text format, but you will lose the relationships among the fields and tables. For example, if you convert the information in the customer database to text, you may end up with ten pages of names, ten pages of addresses, and a thousand pages of billing information, with no indication of which information is related.
Spreadsheet files. Spreadsheet files store the value of the numbers in their cells, as well as the relationships of those numbers. For example, one cell may contain the formula that sums two other cells. Like data files, spreadsheet files are most often in the proprietary format of the software program in which they were created. Data can be shared between different spreadsheet programs by saving individual spreadsheets as a text file in the Data Interchange Format (DIF), however the value and relationship of the numbers may be lost.
Video and audio files. These files contain moving images (e.g., digitized video, animation) and sound data. They are most often created and viewed in proprietary software programs and stored in proprietary formats. Common files formats in use include QuickTime (.MOV), Windows Media Video (.WMV), and Motion Picture Experts Group (MPEG) formats (.MP3); others include .AVI and .WAV files.
Markup languages. Markup languages, also called markup formats, contain embedded instructions for displaying or understanding the content of the file. They provide the means to transmit and share information over the web. The following markup language file formats are supported by the World Wide Web Consortium (W3C) as standards
- Standard Generalized Markup Language (SGML), a common markup language used in government offices worldwide, is an international standard. HTML and XML are derived from SGML.
- Hypertext Markup Language (HTML) is used to display most of the information on the World Wide Web. Because presentation is combined with content trough the use of pre-defined tags, HTML is simple to use but limited in scope. Other markup languages such as XHTML and XML offer greater flexibility.
- eXtensible Markup Language (XML) is a relatively simple language based on SGML that is gaining popularity for managing and sharing information. XML provides even greater flexibility and control than XHTML while avoiding the complexities associated with SGML.
- eXtensible Hypertext Markup Language (XHTML) combines the flexibility found in XML with the ease of use associated with HTML. Strict XHTML rules improve consistency and provide the ability to create your own markup tags. Because they share similar rules, converting XHTML into XML is easier that converting HTML into XML.
The table below summarizes the most common file formats.
Table 1: Common File Formats
|File Format Type||Common Formats||Sample Files||Description|
|Text||PDF, RTF, TXT, DOC||Letters, reports, memos, e-mail messages saved as text||Created or saved as text (may include graphics)|
|Vector graphics||DXF, EPS, CGM, SHP||Architectural plans, complex illustrations, GIS||Store the image as geometric shapes in a mathematical formula for undistorted scaling|
|Raster graphics||TIFF, BMP, GIF, JPEG, PNG||Web page graphics, simple illustrations, photographs||Store the image as a collection of pixels which cannot be scaled without distortion|
|Data file||Proprietary to software program||Human resources files, mailing lists||Created in database software programs|
|Spreadsheet file||Proprietary to software program, DIF||Financial analyses, statistical calculations||Store numerical values and calculations|
|Video and audio files||QuickTime (MOV), MPEG, Real Networks (RM), WMV, WAV, MP3, AVI||Short video to be shown on a web site||Contain moving images and sound|
|Markup languages||SGML, HTML, XHXML, XML||Text and graphics to be displayed on a web site||Contain embedded instructions for displaying and understanding the content of a file or multiple files|
To help ensure your files are accessible over time, you will need to keep verifying that the files formats you are using are still supported. When formats are no longer supported, you will need to decide if you are going to convert and/or migrate your file formats. If you convert your records, you will change their formats, perhaps to a software-independent format. If you migrate your records, you will move them to another platform or storage medium, without changing the file format. However, you may need to convert records in order to migrate them to ensure that they remain accessible. For example, if you migrate records from an Apple operating system to a Microsoft Windows operating system, you may need to convert the records to a file format that is accessible in a Windows operating system (e.g., RTF, Word 2000). For more information on conversion and migration, refer to the Electronic Records Management Strategy and Long-Term Preservation chapters of these guidelines.
You will be faced three basic types of loss when converting or migrating files that will need to be considered before finalizing your plan. The amount and type of loss needs to be analyzed to determine the best course of action. The three types of loss are:
- Data. If you lose data or if it becomes corrupted, you lose, to a varying degree, the content of the record. Bear in mind that, legally, your records must be complete and trustworthy. Metadata may also be altered or lost.
- Appearance. If you convert all word processing documents to RTF, you risk loss of the structure of the record; you may lose some of the page layout. You must determine if this loss affects the completeness of the record. If the structure is essential to understanding the record, this loss may be unacceptable.
- Relationships. Another risk is the loss of the relationships of the data within the file or between files (e.g., spreadsheet cell formulas, database file fields). Again, this loss may affect the legal requirement for complete records.
As part of your records management strategy, you may choose to compress your files. A few of the pros and cons are summarized below.
Table 2: Pros and Cons of File Compression
|Saves storage space||May result in data loss|
|More quickly and easily transmittable||Introduces an additional layer of software dependency (the compression software)|
Compressing files results in a smaller file size, which reduces the amount of storage space needed. However, to create a smaller file size, information is often removed from the file. For example, when an image file is compressed, pixels that the software determines will not be missed are removed, relying on the human eye to fill in the absent details. When an audio file is compressed, sounds often unnoticeable to the human ear are removed, resulting in a smaller file size that, to most people, sounds the same as the uncompressed file. Compression options vary in their degree of data loss. Some are intentionally “lossy,” such as the ones described above while others are designed to be “lossless.” Lossless compression results in a smaller file size, but allows for exact reconstruction of the original file from the compressed data, unlike lossy compression which only approximates the original data. Because of these issues, you may choose to compress some files and not others.
Many of the challenges associated with records management can be overcome with good planning. When trying to determine the most appropriate file format/s to use for long-term access, there are many things to consider. Weighing the pros and cons of each of the suggestions below will assist with planning efforts.
- Accessibility. The file format must enable staff members and the public (as appropriate under the MGDPA) to find and view the record. In other words, you cannot convert the record to a format that is highly compressed and easy to store, but inaccessible.
- Longevity. Developers should support the file format long-term. If the file format will not be supported long-term, you risk having records that are not durable, because the software to read or modify the file may be not be available. Records should be migrated or converted if you determine a file format is no longer supported. Open source, open standard and non-proprietary formats are preferable to completely proprietary ones.
- Accuracy. If you convert your records, the file format you convert to should result in records that have an acceptable level of data, appearance, and relationship loss, if any.
- Completeness. If you convert your records, the file format you convert to should meet your operational and legal objectives for acceptable degree of data, appearance, and relationship loss, if any.
- Flexibility. The file format needs to meet your objectives for sharing and using records. For example, you may need to frequently share copies of the records with another agency, use the records in your daily work, or convert and/or migrate the records later. If the file format can only be read by specialized hardware and/or software, your ability to share, use, and manipulate the records is limited.
Key Issues to Consider
Now that you are familiar with some of the basic concepts of file formats, you can use the questions below to discuss how those concepts relate to your agency. Pay special attention to the questions posed by the legal framework, including the need for public accessibility as appropriate, completeness, trustworthiness, durability, and legal admissibility. Consider the degree of acceptable data, appearance, and relationship loss. Take a long-term approach so that your file formats will meet your operational and legal requirements now and in the future.
- What are our goals for electronic records management?
- How is our agency affected by the legal requirements?
- What current file formats do we use? Is it anticipated that these will be supported long-term?
- Are we planning on converting and/or migrating our records? If so, when? How often?
- How will we find and document loss and/or changes?
- What levels of data, appearance, and relationship loss are acceptable?
- How will our decisions affect other groups that may need current and future access to our records (e.g., other government agencies, the public)?
File Formats, Annotated List of Resources
Next Chapter, Digital Media
Electronic Records Management Guidelines, March 2012, Version 5.
Links verified March 12, 2012.