What bidi information a document needs?

To be able to render a document you need to know:

• The document direction, which determines general layout (paragraph alignment and indentation, bullet position, table column order, etc.). This direction can be vary for elements of the document (e.g. Hebrew text containing an english quotation of several paragraphs of).
• The text direction determines the order of letters. This direction can vary for parts of a paragraph (e.g. inline quotations).

So assuming a hierarchical document model, consisting of nested elements (like in HTML), we have to derive a direction for every element. In complex documents LTR and RTL elements can be arbitrarily nested ("embedded" is the standard bidi term). Note that some embeddings don't have any other semantics beyond the embedding of one language inside another. A grouping element with no inherent semantics (like span or div in HTML) is needed.

Hierachically implicit encoding

HTML prodives a dir="LTR" / dir="RTL" attribute that can be set on any element. This approach is suboptimal because in most convertion/processing programs elements come and go but text stays. If we could derive the direction from the text itself, no format adaptation would be needed and programs would preserve the bidi info in most cases automatically.

The Unicode bidi algorithm defined in UAX 9 desribes the "first strong character" heuristic for implicitly determining the paragraph base direction: the first strongly directional charater (e.g. an English or Hebrew letter) sets the direction of the paragraph. In most cases it works right; in the few cases where it doesn't, one can put an implicit bidi mark (LRM/RLM) at the beginnning of the paragraph to override the decision.

This approach is good. Unfortunately the Unicode bidi algorithm doesn't extend to other levels. For encoding the direction of paragraph parts it suggests separate explicit embedding codes for RTL and LTR embeddings (these inspired the HTML dir attribute). Document parts bigger than a paragraph are not accounted for in UAX 9 at all.

The basic proposed idea is to extend the first-strong heuristic to all document elements, applying it hierarchically (how exactly will be specified below). Thus a Hebrew quotation inside an English paragraph would be automatically handled correctly as long as it's surrounded by some HTML element (e.g. <blockquote> .. </blockquote> or <em> .. </em>). Similarly, the whole document would get the direction of its title and a table would get the direction of its first cell.

Fixing ambiguities

Hierarchical direction inference should already make good guesses in most cases. There are two ways to guide it, together allowing you to express any possible bidi strucuture:

• Add structure: wrap text in some element just for the sake of indicating embedding structure (e.g. span or div in HTML).
• Override the direction of any element by inserting an LRM/RLM mark right at the start of the element.

Note that you don't need any of the explicit bidi control codes

Not all formats/tools give you these abilities in all point of a document. Workarounds for such cases will be given below; meanwhile let's make some choices that minimize the need for overriding in sane cases.

What kinds of ambiguities do we have?

Interactions between elements

[This section was written before the previous. It duplicates most of the arguments given above (this document could use some more editing ;-)) but goes into details of behaviour. The described behaviour should allow the above use cases of fixing structure.]

How should a sub-element interact with it's surroundings? UAX 9 defines the following behaviour of embeddings:

1. The paragraph's base direction guessing ignores embedding boundaries and takes the first strong character whether it's embedded or not.
2. Embeddings are represented by levels. Adjacent embeddings are merged as a side effect of this.
3. Implicit levels are resolved. On the inside, an embedding behaves as a sub-paragraph with appropriate base level; on the the outside it behaves as a single strongly-directional character (of the embedding's direction).
4. The representation is still level-based, so adjacent embeddings are merged again.

The current standard for HTML bidi (described e.g. in the CSS2 spec) follows this behavior, mapping markup (normally - dir attributes) to UAX 9 embeddings and applying the above algorithm on the whole paragraph at once (allowing embedding boudaries to be quite independent from element boundaries).

I claim this is sub-optimal in several ways.

• It's questionable whether element boundaries should be transparent for purposes of direction guessing. In other words, should {FOO} bar. have a RTL base direction? If it does, it means that an element starting with a sub-element always get the same direction and one must insert an LRM/RLM before it if he wants them to differ. So I claim that skipping sub-elements during direction guessing would do the right thing in most cases. Note that again we treat sub-elements as neutrals. However, if there is no strong character outside sub-elements (e.g. in {FOO}, {bar}, we should fall back on the first sub-element with a strong direction.

  What should we do about elements that contain no strong characters whatsoever? For paragraphs, the UAX 9 heuristic says that such a paragraph containing no strongly directional characters defaults to LTR. This makes little sense given a hierarchical document; we should try to infer the direction from the surroundings of the element. One idea is to take the containing element's direction. A better idea is to run the bidi algorithm on the containing element, treating each sub-element as if it was a neutral character and take the direction resolved for the corresponding "neutral character". This makes sense in cases like the following:

  Logical:	he said: "THE NUMBERS {3, 7, 13} ARE ALL PRIME."
  Base dir:	L
  Levels:	0000000000111111111111 21121122 1111111111111100
  Visual:	he said: "EMIRP LLA ERA 13 ,7 ,3 SREBMUN EHT."

  Here you had an implicit RTL embedding and any neutral sub-elements of it should indeed inherit an RTL direction.

• The next thing to question is effect elements have on surrounding neutral characters. A neutral character trapped between two elements would get their direction, which might not be what you want at all:

  Logical:	he said "FOO!".  "BAR!" he added.
  Base dir:	L
  Levels:	000000000111111111111000000000000
  Visual:	he said "RAB"  ."!OOF!" he added.

  This is the effect you would expect from the following element structure:

  Logical:	he said "{FOO!".  "BAR}!" he added.
  Base dir:	L
  Levels:	000000000 111111111111 000000000000
  Visual:	he said "RAB"  ."!OOF!" he added.

  but in our case you would embed each quotation in its own element and expect the intervening punctuation not to be taken as part of the embedding:

  Logical:	he said "{FOO!}".  "{BAR!}" he added.
  Base dir:	L
  Levels:	000000000 1111 00000 1111 00000000000
  Visual:	he said "!OOF"  ."!RAB" he added.

  The point is that given the ability to express different structures explicitly, you would like the bidi algorithm to guess less (for flat text without structure information, the UAX 9 algorithm should apply as today, of course). The desired effect is achievable if elements behave as neutral characters on the outside.

  • A sub-point of this is behaviour of adjacent sub-elements. If several adjacent sub-elements have a direction opposite to that of the containing element, UAX 9 would merge them into one embedding of opposite direction:

    Logical:	ab {CD}{EF} gh
    Base dir:	L
    Levels:	000 11  11 000
    Visual:	ab FEDC gh

    (effectively assuming an embedding around both of them: ab {{CD}{EF}} gh), whereas treating each as a neutral character would maintain the containing element's order between them:

    Logical:	ab {CD}{EF} gh
    Base dir:	L
    Levels:	000{11}{11}000
    Visual:	ab DCFE gh

    It's harder here to make a case for the second behavior because the reader is left with no punctuation to separate the embeddings and the reading order is very ambiguous. But if you special-case adjacent sub-elements, you would have to define the semantics for adjacent sub-elements of different directions, which becomes very compicated; always treating sub-elements as neutrals is easy to understand.

    And again, the behavior of "merging" them is always achievable by explicitly adding an element around both, so always treating sub-elements as neutrals is more flexible.

• Another very basic thing to question in standard bidi handling of HTML elements is that the final merging of adjacent runs of same level can break an element into non-adjacent parts if the first/last character of an element is at a deeper level than the element's base level:

  Logical:	{ab CD}{XY zw}
  Base dir:	L
  Levels:	00011  11000
  Visual:	ab YXDC zw

  This example shows standard HTML bidi behaviour, so both elements have LTR base direction; in our model the second element would get an RTL base direction but still the first element would be disjoined:

  Logical:	{ab CD}{XY zw}
  Base dir:	L
  Levels:	00011  11122
  Visual:	ab zw YXDC

  Still, it makes no sense. Opening an element outside an embedding and closing it inside it is not any better than overlapping elements: <element><bidi></element></bidi>! Bidi embeddings always correspond to some logical nesting of text, so we should require every element to be in one piece (though perhaps in different internal order) in the output:

  Logical:	{ab CD}{XY zw}
  Base dir:	L
  Levels:	{00011><11122}
  Visual:	ab DCzw YX

  Such treatement also means that reordering could be done directly on the tree; this should also be simpler to implement.

Workarounds

The scheme described so far is sufficient to describe any bidi structure merely by using elements (sometimes only for the structure, like HTML's span or div elements) and LRM/RLM to override directions. If you directly write the HTML, there is no problem. However, one of the major ideas of this scheme is that you can use an existing converter from some other format to HTML that doesn't have to know anything about bidi. As long as it translates elements of the source format to HTML elements and doesn't drop your LRM/RLM marks, it will work. But the conversion will usually limit your freedom to control the resulting HTML. You can suffer from either or both of two problems:

• Inability to create arbitrary structure of elements.
• Inability to put text, specifically bidi control codes, into arbitrary places of the resulting document.

The first problem is easily solved by the fact that the full UAX 9 bidi algorithm runs on each element of the document. You can use LRE..PDF / RLE..PDF (or sometimes just LRM/RLM) to embed parts without having to actually wrap them with elements. Obviously, the ideal development would be to add two new implicit embedding characters to Unicode, behaving as described here. That will not happen soon (and one of my goals here is to prove it's a good idea), so for now the explicit embeddings will do.

The second problem is harder. Suppose you want to put RLM at the start of the document to override it's direction. Many input formats will not let you put any text above the heading and in any case all your text will be wrapped in block-level elements. The best you can do is to create a paragraph containing nothing but an RLM.

What I propose is to artifically strip any element(s) that contain nothing but bidi control codes (LRM, RLM, LRE, RLE, LRO, RLO, PDF), until they appear in the same element with other things. E.g. {text {{&LRM;}{&RLE;}} text} becomes {text &LRM;&RLE; text}. This is a rather ugly hack but it allows you to overcome the above problems. A paragraph containing nothing but an LRM would be converted to an LRM at the block level, which would have priority over all sub-elements (including the heading appearing before it) to set the document's direction. You could even place part of the document between two paragraph containing a nothing but an RLE and a PDF respectively, to embed this part of the document.

HTML conversion

<foo>
    ltr
    <bar>??</bar>
    RTL
    <quux>??</quux>
    RTL
    <baz>??</baz>
    ltr
</foo>