TABLE INTEROPERABILITY

For the last several years, SGML users have been pointing out that there are major interoperability issues when using tabular material (tables) in SGML implementations. First, SGML itself does not prescribe any standard way of encoding tables, leaving that to individual applications which may have taken different, possibly incompatible approaches. Furthermore, even when an application standard has been defined (such as the U.S. Department of Defense CALS model), different vendors' products may handle the same table in different ways.

Recognizing the importance of this issue, SGML Open's Technical Committee formed a Table Interchange subcommittee in 1994 to research current interoperability issues and recommend changes that would resolve the bulk of these problems. The Committee's mission has involved two fundamental goals:

• to promote the interoperability of SGML tables throughout the SGML Open vendor base; and

• to suggest a framework for the next generation of SGML table markup, with a focus on data representation in addition to table geometry and formatting.

Because the CALS table model in particular has proliferated widely, appearing in a large number of other applications, it was chosen as the initial starting point. Although it has evolved to the point of a de facto standard, the CALS table model's design was never actually completed. The specification leaves a large number of semantics open to interpretation which in turn has made interoperability difficult to achieve.

As its first major task, the Committee therefore set out to identify and document ambiguities in the CALS table model specifications, identify and document related interoperability issues between SGML Open vendor products, and lay the groundwork for developing a proposed clarification of the standard that will minimize ambiguity and maximize interoperability.

This paper summarizes the results of this initial work, identifies the sources of current interoperability issues for the CALS model, and summarizes the most common set of practices currently followed by SGML Open vendors.

The present baseline CALS table model was officially released on 26 June 1993 as part of the U.S. Department of Defense SGML standard MIL-M-28001B. First released in 1990 as part of the previous MIL-M-28001A specification, it has now been adopted, sometimes with small modifications, in a large variety of non-military industry applications. These include commercial aerospace (ATA and AECMA), computer documentation (DocBook), automotive (J2008), semiconductors (Pinnacles), telecommunications, and many site-specific uses. Even within the Department of Defense, several versions have evolved (e.g., 38784C vs. 38784D vs. Navy MID, etc.).

Designed to handle a variety of complex tables in military technical documents, the CALS table model focuses on encoding two-dimensional row and column geometry with basic formatting features such as cell alignment, borders, and rotation. It anticipates complex cell content, such as multiple paragraphs, lists, and graphics, and even provides for one level of nested tables within cells. Other than providing a facility for logically naming columns (e.g., state and population rather than simply column 1 and column 2), it does not address semantic encoding of the content. Structurally, the CALS table model presumes that tables are made up of an optional title plus one or more TGROUPs, each of which has its own body (TBODY) and its own independent set of optional column headings (THEAD) and footings (TFOOT). The column definitions for each TGROUP are provided by COLSPEC elements, which can be inferred if desired, and special COLSPECs can be provided for THEAD and TFOOT.

Each THEAD, TBODY, and TFOOT section within a TGROUP is made up of a series of ROWs. ROWs are formed as a left-to-right sequence of cells (ENTRYs), which can contain mixtures of text, graphics, and more complex structural objects such as list items. An ENTRY may span more than one column (horizontal spanning) and more than one row (vertical spanning), depending on how its attributes are set. As a shortcut for defining horizontal spans, optional SPANSPEC elements can be added at the TGROUP level, each referring to spans across column names defined in COLSPECs.

A table cell can contain either a simple ENTRY or an ENTRYTBL, which is essentially a table-within-a-table. ENTRYTBLs cannot contain TGROUPs (they are implicitly a TGROUP themselves), and cells within an ENTRYTBL cannot contain other ENTRYTBLs. Also, although ENTRYTBLs may span more than one column, they are constrained to one row.

Cell formatting attributes are limited to text alignment (both horizontal and vertical), cell borders (rulings), and rotation. For all but the last of these there is a complex inheritance scheme that allows values to be defaulted and overridden at multiple levels.

Interoperability potentially covers anything that would surprise a user when moving tables between different SGML systems. In essence, it has to do with answering the key question of the frustrated user: If I spent time and energy getting this table right on one SGML system, why doesn't it automatically work on another?

However, interoperability is not as straightforward as one might think. In fact, it is to some extent in the eyes of the beholder. Interoperability can be defined (and is defined by users) in at least two different ways:

• Preservation of SGML syntax (exact set of SGML tags and attributes must not change as a table is passed through multiple systems in sequence—i.e., imported and then re-exported at each step).

• Preservation of format (visual appearance of the table must not change as table is passed through multiple systems in sequence—i.e., imported and then re-exported at each step).

In the first, data-centric view, detailed differences in format are tolerated when rendering across different systems. However, it is essential that original SPANSPEC definitions, for example, are faithfully carried through from input to output. It is also essential to maintain and allow edits to data (e.g., TFOOT) that is not supported for rendering purposes. Put another way, users are not surprised if their tables don't always look the same, but are upset if any of the detailed tags or attributes change when passed through multiple systems.

In the second, appearance-centric view, detailed differences in SGML syntax are tolerated when interchanging tables between different systems. It doesn't matter, for example, that original SPANSPECs are transformed into individual spanned cells, with a new set of SPANSPECs generated upon export. What is essential is that the table continues to look the same. Put another way, users are not surprised if detailed tags or attributes change when passed through multiple systems, but are upset if their tables don't always look the same.

Ideally we would achieve both kinds of interoperability simultaneously. However, in practice part of the interoperability problem comes from the difference in these two viewpoints, and from not understanding these differences. As vendors we must have a very clear definition that can be readily understood by users. But users must also understand the limits of any definition, realizing that interoperability is partly implementation and partly education.

Interoperability must also be carefully defined at a specific level. For example, one user might conclude format has been preserved if the all cells maintain the same left, right, top, and bottom borders. Another might disagree because line widths are different between two systems. This becomes especially tricky when the difference between page-oriented and pageless systems is considered. Looking the same is not an intuitively obvious judgment.

Of course, the keys to interoperability must still be addressed through specific features and functional behavior of the products themselves. Thus another cut at interoperability, one which we have adopted here, breaks the issues down into three implementation-oriented categories:

1. Differences in supported features (e.g., Vendor A supports ENTRYTBL, Vendor B does not; therefore tables have to be handled differently when moving data between systems).

2. Differences in interpretation of supported features (e.g., both Vendor A and Vendor B support COLSEP, but Vendor A interprets it as the left separator, Vendor B interprets it as the right; therefore tables look different when moving data between systems). These differences may come in several forms:

   1. Differences in the specific semantic attached to a tag or attribute (e.g., does FRAME override the borders of outside cells, or does it constitute an entirely separate frame around the whole table?).

   2. Differences in the assumed defaults where none are specified (e.g., does #IMPLIED all the way to the top level mean that cell rulings are assumed on or off?).

   3. Differences in inheritance path/precedence order (e.g., does formatting on SPANSPEC override that of COLSPEC or vice versa?).

   4. Mismatches when dealing with redundant information (e.g., Vendor A uses COLNAME and ignores NAMEST, but Vendor B uses NAMEST and ignores COLNAME).

3. Differences in definition and handling of error conditions (e.g., when defined cell alignment conflicts between the ENTRY and an applied SPANSPEC, Vendor A arbitrarily uses the SPANSPEC definition, Vendor B identifies an ambiguity and rejects the table; therefore an acceptable table on one system may be rejected by another).

In a perfect world, competent vendors acting in good faith would avoid any such differences between their products. However, Einstein's God is in the details applies here with a vengeance. Two products may easily seem to support the same feature set in the same manner; for example, each might support the use of COLSPECs and allow the number of COLSPECs to differ from the defined number of COLS. But upon deeper inquiry, it can turn out that one assumes the actual number of columns is defined by COLS, inferring COLSPECs if needed and ignoring any excess COLSPECs, while the other uses the number of COLSPECs to determine the actual number of columns, resetting COLS to match. Thus, even though things seem to match up at a high level, in fact a subtle but potentially quite serious interoperability problem exists.

Finally, in analyzing interoperability it is important to understand what the underlying model was actually intended to do. The CALS table model, for example, was purposely designed to deal only with the structural aspects of tabular information, together with basic presentation choices (e.g., rulings around cells, alignment, etc.). It was not meant to dictate precise rules for typesetting, composition or screen display, or the complex interactions between table and page layout. Nor does it specify how to handle errors. If we try to measure interoperability at a higher level of precision, we have already gone beyond the scope of the CALS table model itself.

After a brief review of the CALS standard and current vendor products, the Committee recognized that the interoperability issue was in fact made up of a large number of subtle ambiguities and small differences in interpretation. Taken individually, each problem was minor and could easily escape notice. When added together, however, they created an overall issue with major implications.

Because the essence of the problem was lurking in the details, the Committee felt it was imperative to take a very thorough, rigorous approach. Therefore we began by combing through the existing CALS table model and related tag/attribute definitions, attempting to identify every possible area of potential ambiguity or misunderstanding. From this baseline, we then created a highly detailed vendor questionnaire, consisting of over one hundred questions designed to pinpoint all possible areas of difference between products. These were in turn broken down into eight major categories:

1. Backbone table structure/format.

2. Row/column structure.

3. Cell formatting - Column widths.

4. Cell formatting - Rotation and alignment.

5. Cell formatting - Cell borders.

6. Cell formatting - Inheritance.

7. Cell content.

8. General.

Individual questions focused on both differences in the set of supported features across vendor products, and in the way each vendor had interpreted the semantic details. The general questions were designed to elicit comments that might shed additional light on the issues from slightly different angles. Furthermore, to minimize the possibility of misunderstanding, we encouraged participants to attach comments of unlimited length to each response. A complete list of survey questions is contained in Appendix A, Survey questions.

All SGML Open vendors were invited to participate in filling out the questionnaire, with the goal of obtaining a large enough representative sample on which to base solid conclusions. In addition, we solicited input from other past and current members of the CALS technical committee that architected and now maintains the CALS table model. After this process was complete, we had obtained a wealth of information, including completed questionnaires for seven SGML Open vendors that provide authoring, publishing, and electronic viewing products that support CALS SGML tables. While a few vendors elected not to participate for various reasons, a vast majority of the largest and most experienced SGML vendors were included, and virtually all SGML Open vendors offering related products were members of our committee. Therefore we felt confident that our sample of seven vendors was a sufficient base for analysis.

After a few iterations in which vendors were allowed to obtain clarifications and refine their answers, results were formally tabulated in matrices and cross-analyzed to extract the key issues. See the detailed survey results in Appendix B, Detailed analysis of survey results. In the initial extraction process, we identified significant issues on the following basis:

• If a specific feature appeared to be fully supported by less than two-thirds of our representative sample (i.e., by four or less vendors), this feature was identified as a significant issue.

• If a specific semantic interpretation was not unanimously agreed upon, it was also identified as a significant issue.

A summary of the most commonly supported features was then constructed by excluding those features that had been identified as significant issues.

Similarly, a list of key ambiguities/differences was drawn up consisting of all specific semantics that had been identified as significant issues. A summary of most commonly supported semantic interpretations was formulated using the interpretation shared by the largest number of vendors in our sample.

Vendors agreed on the great bulk of table features and semantic interpretations included in the survey. As expected, a number of detailed differences surfaced between the products surveyed. However, most of these were relatively subtle. Following our working definition of tables interoperability, we separated these into two fundamental categories, summarized below:

1. Differences in the specific set of features supported (e.g., some vendors support ENTRYTBL, some do not.

2. Differences in interpreting tag/attribute semantics (e.g., in the absence of a strict definition, vendors have made slightly different assumptions about how cell format attributes are inherited and overridden).

As a means of laying the groundwork for establishing table interoperability between vendor products, this section summarizes the practices and commonly supported interpretations by SGML Open vendors. These include:

• most commonly supported features;

• features not commonly supported;

• most common semantic interpretations.

As a result of this study the Committee plans to propose an SGML Technical Resolution that will provide a common definition of tables interoperability using the CALS model. We are also sharing our recommendations with the CALS Electronic Publishing Committee (EPC) as input to improving the CALS table model and its documented semantics. When this phase of the Committee's work is complete, we will move on to the second goal in our mission statement: suggesting a standard framework and set of approaches for the next generation of SGML table markup. This work will explore where the current CALS table model falls short, going beyond format and layout issues to a model which captures the author's intent for underlying table data in an unambiguous and interchangeable way. We expect this may include a set of standard approaches and DTD fragments for different purposes.