[Unicode]   Common Locale Data Repository : Bug Tracking Home | Site Map | Search
 
Modify

CLDR Ticket #10851(accepted spec)

Opened 4 months ago

Last modified 3 months ago

PRI #367: Additional modifiers and toggles

Reported by: charupdate@… Owned by: mark
Component: keyboards Data Locale:
Phase: spec-beta Review:
Weeks: Data Xpath:
Xref:

Description

To provide a correct user experience, a Numbers modifier key must be present that activates an extended numpad on the alphanumerical block, and a Programmer toggle is needed to toggle between language and programming because more than one single keypress is inefficient.

Attachments

Change History

comment:1 Changed 4 months ago by mark

  • Owner changed from anybody to mark
  • Status changed from new to accepted
  • Type changed from survey to spec
  • Milestone changed from UNSCH to 33

Mark, as POC for the keyboard group

comment:2 Changed 4 months ago by charupdate@…

Numbers modifier key

The Numbers modifier allows to type Western Arabic digits and hex digits A..F arranged in a pad like the integrated numpad on Function level for compact keyboards. Additionally it provides on the same level typographic operators along with the ASCII operators, and currency symbols, percent sign, and more. Sequences 'U+' and '&#x' need not more than one keypress.

Adding the Shift modifier keypress gives access to superscripts, especially small letters, as needed in Latin script for some locales including English. (As discussed on the ML, this is not a mere typographic feature but a constituent of the writing system, as were superscript abbreviations in medieval Latin, where this convention originates.

The Numbers level is required for locales using the NARROW NO-BREAK SPACE as thousands separator. U+202F is mapped to the space bar on the Numbers level.

A sample layout with French labels is found on dispoclavier.com.

Mapping

The Numbers modifier is best mapped on Left Option, because it is potentially often used. Having an extended Numpad on Numbers level is more useful than the extra confort of symmetric Option modifiers. Left Option is anyway unavailable as part of the default mapping on Windows where the Scancode Mapper for keyboards is used to move Numbers from B00 to left Alt. B00 is the default mapping of Numbers because this key is lefthand on ISO keyboards, is not a letter key, and is easy to reach. But Numbers is inimplementable on this position on macOS without third party software, and that key is missing on most of the worldʼs keyboards anyway. Alt can be moved on Left Windows. Where Right Windows is missing, Right Menu (Applications) can be used for Windows, and if that is missing, E00 is used for Windows. (Please see further synergies below.)

Programmer toggle

On keyboards with an ASCII layout, the Programmer toggle may activate deadkeys, and on those with an extended Latin or non-Latin layout, it activates ASCII without changing the keyboard layout in the prefences (Language bar), where the shortest way is a two-key shortcut. The Programmer toggle instead is recommended for mapping on the CapsLock key, as most of the worldʼs scripts are unicase, CapsLock is disliked by many people, is not used for digits any more (that role belongs to the Programmer toggle), and can be mapped on the B00 ISO key if available, especially for locales using uppercase for names. On Windows however, that has a supplemental graphic toggle (KanaLock), the Programer toggle can default on E00, and the permutation with CapsLock is part of the scancode remapping.

To date, the effect of the Programmer toggle is exemplified in the abovementioned French Programmer mode vs Default (French) mode.

comment:3 Changed 4 months ago by Marcel Schneider <charupdate@…>

Iterative dead keys

While high-end input editors, notably SIL Keyman, support an unlimited number of flags, that are a way of implementing layout toggles, macOS allows to make dead keys iterative by maintaining/renewing the previous status after character output. This allows to implement auxiliary toggles, or a missing CapsLock if the specified Caps toggle is used as a Programmer toggle.

Example of a CapsLock emulation:
<when state="caps" output="A" next="caps">
The toggle dead keys is in the Base shift state and has:
<when state="none" next="caps"> and:
<when state="caps" next="none">
Hitting Backspace or Enter resets the state to "none".

However, Apple are expected to add the missing layout toggle, present on Windows and Linux. See Re: Keyboard layouts and CLDR (was: Re: 0027, 02BC, 2019, or a new character?).

comment:4 Changed 4 months ago by Marcel Schneider <charupdate@…>

Modifiers in use, but not in UTR #35

The list of Possible Modifier Keys in part 7 of UTR #35, version 32, is incomplete in that it does not mention Japanese modifiers. Among these, the Kana modifier is also used in the Windows implementation of the Canadian Multilingual Standard keyboard layout. The macOS implementation of this layout is the only layout shipped natively with Mac for the Canadian market, and is highly appreciated by all Mac users in Canada.

The goal of CLDR is to make LDML support all of the worlds keyboard layouts, and as a subset of these, to properly represent the layouts shipped with major operating systems. This goal is actually unreached wrt to the Canadian locale:

  • Specifying that Canadian Standard on Windows uses Right Control as a layout modifier is misleading and messes up the Control keymap.
  • Right Control cannot be specified in the Standard, since ISO/IEC 9995-2 proposes to place the modifier next to AltGr: “Optionally, if one key can be dedicated to the Group select function, in this case it is recommended to be placed adjacent to a Level 3 select key.”
  • The Canadian extra modifier could be mapped to an unused scancode that could be mapped to any available key next to AltGr depending on the hardware, using the Scancode Mapper for Keyboards shipped with Windows 2000 and later (eight years after CAN/CSA Z243.200-92).
  • Hardware for Canada normally displays only the rightwards empty arrow on Right Control.
  • On many notebooks actually on the marketplace, the Right Control key is used to map characters that are normally on key B00 (between Left Shift and W), while this key is missing. Hence, Right Control is not available for use as Group selector. That motivates endeavors to unify both Level 3 and Group 2a maps, at the precondition that the modifier on the AltGr key is not Ctrl+Alt, but 0x08 (Kana) otherwise mapped to Right Control.

For all these reasons, the LDML cannot properly specify RCtrl as the Group selector of the Canadian Multilingual Standard keyboard.

Enumeration

Rather than listing the names of Japanese modifiers only, we propose a list of all modifiers supported on Windows, and actually usable:

  • 0x01 Shift
  • 0x02 Control
  • 0x04 Alt
  • [0x06 (Ctrl + Alt = AltGr, current mapping)]
  • 0x08 (Kana modifier/Group selector/Kana toggle)
  • 0x10 (ROya, Right Oyayubi)
  • 0x20 (LOya, Left Oyayubi)
  • 0x40
  • 0x80 (Grpseltap)

In 'Robust key message handling', Marc Durdin recommends not to use AltGr but other modifiers instead: “Don’t do a special-case for AltGr – other modifiers are also legal.”

  • This results in ruling out the use of 0x06 (Ctrl+Alt) on the AltGr key, using 0x10 instead.
  • 0x08 shouldnʼt be used for that purpose unless there is no Kana/Programmer toggle on the layout.
  • The Numbers modifier (see comment 2) is consequently 0x20.
  • The remaining two modifiers (0x40, 0x80) are added to the two toggle keys, to use when the key is held down for at least one full word in any other script out of two (and two others with AltGr [0x10] added).

Note that the modifiers added on top of toggle keys do not interfere with the toggle functionality. The two fuictionalities are parallel. The only inconvenient is that after using the modifier, the key should be hit again to reset the toggle flag. That is a minor inconvenient, far outweighed by the advantage of being able to type names in original script without changing the layout, and on a layout matching the base layout especially if that is not QWERTY.

On macOS:

  • The Numbers modifier is implemented on Left Option (with Option keys next to the Spacebar, or coming second).
  • If macOS keeps unsupporting additional modifiers and/or toggles, script dead keys are made iterative, eventually on double press.

Example:

<when state="none" next="greek"/>
<when state="greek" next="greeklock"/>
<when state="greeklock" next="none"/>

And a letter key:

<when state="greek" output="α"/>
<when state="greeklock" output="α" next="greeklock"/>

(Cf. the example of a caps lock emulation, above in comment 3 [with open tags that should be empty tags].)

comment:5 Changed 3 months ago by Marcel Schneider <charupdate@…>

Caps Lock of limited usability on Windows

Please note that Iʼm fully aware that unlike Kana Lock, Caps Lock does not act on any shift state except the Base and AltGr shift states. Hence, if the AltGr key is mapped to another modifier (0x08, 0x10, 0x20, 0x40 or 0x80), the ability to type in uppercase by pressing AltGr only, while Caps Lock is on, is lost.

But I strongly discourage mapping any letters for access with the AltGr key. I suggest mapping ASCII symbols and punctuation there, and even digits in the upper row to free the Base positions there, as they are used for letters in various Latin script using locales. We shouldnʼt map digits in Shift neither, for the sake of intuitive titlecasing in these locales.

The layout layers where additional letters should be mapped on, are the specific diacritical layout groups accessed by the diacritical dead keys present on the layout, and — depending on character identity — the generic layout groups accessed by another dead key called (generic) group selector.Such layouts may be specified in conformance to ISO/IEC 9995-3. The German enhanced (multilingual) national keyboard standard parts 2 and 3 of DIN 2137 are implementations of this standard.

But Iʼd not suggest to map many dead keys on AltGr/Option + [key], because we really need these key positions to streamline the input of ASCII punctuation and symbols.

comment:6 Changed 3 months ago by Marcel Schneider <charupdate@…>

Edit: Of course the Caps Lock toggle does act also on the Shift and Shift+Ctrl+Alt (Shift+AltGr) levels, because its relationship to the Shift modifier is XOR.
(The relationship between the Kana toggle and the Kana modifier (both 0x08) is OR.)

comment:7 follow-up: ↓ 8 Changed 3 months ago by verdyp@…

I don't understand why not mapping AltGr/Option to the same keymap as Ctrl+Alt can be in fact *more robust* that doinbg this mapping.
Various compact hardware keyboards do not have duplicate (left and right) modifier keys for Ctrl, Alt, or Shift, but only one. And it's still fine to emulate the "AltGr/Option" modifier by the combination of "Ctrl+Alt" modifiers.

This does not limit the number of shift states available in a keymap which is still 6 :

  • "Group 1" in ISO 8895 (4 states, if we ignore the case of CapsLock/ShiftLock for now, which is eventually emulatable):
    • key
    • Shift+key
    • Ctrl+key
    • Shift+Ctrl+key
  • "Group 2" in ISO 8895 (2 states only, no variants with distinguished Ctrl):
    • AltGr+key or Ctrl+Alt+key
    • Shift+AltGr+key or Shift+Ctrl+Alt+Key

Now you can have additional groups for CapsLock/Shift lock state (where available as a physical key or emulated by chaining two keypresses and release on Shift), Kana/Multilingual Canadian group, or for additional function keys (e.g. "Fn", "Windows", "Menu/Application").

Note that Mac keyboards already have a necessary AltGr/Option key (aka "Apple" key on older keyboards), even if it is placed near the space bar, but not necessarily the Ctrl key found on PC keyboards (Mac keyboards have an "Alt" key). Still the Ctrl modifier key can be emulated on Mac keyboards as well by pressing Alt+Option, so the number of combinations is the same (with or without shift, with capslock/shiftlock, or with the Kana key):

Mac keyboards are very similar to PC keyboards with the only exception that:

  • the presence of a Ctrl key is optional on Mac, while on PC it is the "Windows" key
  • removing these last two keys (which are placed in the middle of the two following modidiers), the position of Ctrl and Alt modifiers on PC keyboards are just swapped for Alt and Option on Mac keyboards.
  • Applications use common OS accelerators on Ctrl+key on PC keyboards, while they use Option+key on Mac: swap back the two previous positions so that applications don't have to handle the exact relative placement of Ctlr and Alt on PC or Alt and Option on Mac
  • Applications or the OS UI layer will map their software binding accoding to this logical swap, all is needed to be able to select the proper label to display in the UI by swapping them depending on the OS (the OS can identify if the user has a PC or Mac keyboard according to the installed keyboard device driver, via Plug-n-Play or via an out of band option sent whjen negociating the virtual remote session parameters, saying if the key to the left of Space bar is Alt on PC, meaning that AltGr is emulated by Alt+Ctrl, or Option on Mac, meaning that Ctrl is emulated by Alt+Option).

We get all the necessary compatibility with compact keyboards, and extended keyboards are allowed but not required, to offer the additional keys without needing to use the emulation.

Moving AltGr to a new mode 0x10 (or higher) would break this compatiblity. It is not a solution to increase the number of groups and implement a group 3 (or higher) for ISO 8995 (Group 3 is already used for Japanese Kana key or for the Multilingual Canadian keyboard with the Group3 selector)

So the only question is where to place the Group 3 selector, or how to emulate it if it's missing on physical keyboards. And if we need to offer a way to "lock" that group 3, which is completely equivalent to lock an alternate keyboard layout, as currently done in Windows with Ctrl+Alt+(F1..F10) or AltGr+(F1..F10), except that these are used instead to switch from one keyboard driver/map/IME to another one developed completely separately.

A more natural placement of the group3 selector is on PC keyboards the "Application/Menu" modifier key combined with a numeric key on the top row.

The "Fn" key on PC keyboards is not really needed and should remain invisible to applications, but when it is present, Fn plus a key on the top row should do the same as F1..F12, it's just a hardware facility to be used on more compact keyboards to emulate missing keys which should not be used as a distinctive modifier in applications (As well Fn+Backspace can emulate Delete, Fn+Up can emulate PageUp, Fn+Down can emulate PageDown, Fn+Left can emulate Home, Fn+Right can emulate End, Fn+Space can emulate Insert).

In summary application should not depend on a distinctive presence of a "Fn" key. Even Fn+Alt can emulate the "Application/Menu" (or also Fn+AltGr, and so also Fn+Alt+Ctrl for compatibility, meaning that "Application/Menu" should never use a distinctive "Ctrl" state, but it is still allowed to have a distinctive Shift+"Aplication/Menu" which should not depend on CapsLock/ShiftLock state, and thus is equivalent to Shit+Fn+AltGr, or even Shift+Fn+Ctrl+Alt for the most reduced keyboards, which is difficult to type with another fifth key, unless it works in cunjunction of ShiftLock/Capslock emulated by pressing and releasing Shift twice, and that why the Capslock/ShiftLock state or Ctrl
and Alt state should not be distinctive for "Application/Menu")

So where can we place Group 3 (or higher) selector: on the "Fn" key if present (generally on compact keyboards), otherwise on its own key on non-compact keyboards that should not have or use the "Fn" key (they have dedicated F1..F12 keys, and/or complete cursor keys for Home, End, PgUp, PgDown, Insert, Del).

How to place the "numeric group" selector? NumLock seems to be appropriate (but it can also be emulated by Fn+B0 (before "1" on the top row, if it's not used for "0") or Fn+B10 (after key "9" on keyboards that place "0" before key "1" on the top row), and such emulation is not necessary on non compact keyboards featuring the "numlock" key in the numeric keypad.

But unlike the usual "Numlock" function of PC keyboards, the "numeric group" is not locking (and not even needed for complete PC keyboards with separate numeric keypads, as they all have also a complete set of cursor keys !).

If needed, the numeric group/Numlock can be emulated by "Fn" plus "+" (the last key of the top row).

comment:8 in reply to: ↑ 7 Changed 3 months ago by Marcel Schneider <charupdate@…>

Replying to Philippe Verdy <verdyp@…>, comment 7:

Thank you for your comment.

I don't understand why not mapping AltGr/Option to the same keymap as Ctrl+Alt can be in fact *more robust* that doinbg this mapping.

When advocating the replacement of legacy Windows AltGr with a distinct modifier, I basically relied on a mass of convergent testimonies, among which Marc Durdinʼs is the most outstanding. That includes also many user complaints about overlapping key combinations between shortcuts and graphic characters. Karl Pentzlin suggests a workaround in his paper on the Extension of the German Standard PC Keyboard, section 3.1, quoted on Bugzilla (compare comment #35 and preceding). On Linux, AltGr is not an equivalent of Ctrl+Alt, so they were baffled. LO and OO respect this Windows feature, at the expense of shortcomings like that reported in comment #34 there, while MS Word is able to make a flexible difference (comment #32).

Various compact hardware keyboards do not have duplicate (left and right) modifier keys for Ctrl, Alt, or Shift, but only one.

The more keys are missing, e.g. right Shift, the less they have of a usable worktool, and at some point there is perhaps a need to use them with appropriate (legacy) keyboard layouts. But for locales using AltGr, compact hardware like netbooks always have that key, and even right Control, that may however be hijacked to replace a missing B00 key. Even Menu/Apps is often present, just right Windows never.

And it's still fine to emulate the "AltGr/Option" modifier by the combination of "Ctrl+Alt" modifiers.

Iʼm a bit doubtful because pressing two keys instead of one is somewhat inefficient. But agreed this is the intent of the equivalence, to maintain full symmetry on US keyboards. This design choice was made at a time where shortcuts were less numerous, however. Probably the actual overload even on (Shift+)Ctrl+Alt was not anticipated.

[…]

Note that Mac keyboards already have a necessary AltGr/Option key (aka "Apple" key on older keyboards), even if it is placed near the space bar, but not necessarily the Ctrl key found on PC keyboards (Mac keyboards have an "Alt" key). Still the Ctrl modifier key can be emulated on Mac keyboards as well by pressing Alt+Option, so the number of combinations is the same (with or without shift, with capslock/shiftlock, or with the Kana key):

Mac keyboards are very similar to PC keyboards with the only exception that:

  • the presence of a Ctrl key is optional on Mac, while on PC it is the "Windows" key
  • removing these last two keys (which are placed in the middle of the two following modidiers), the position of Ctrl and Alt modifiers on PC keyboards are just swapped for Alt and Option on Mac keyboards.
  • Applications use common OS accelerators on Ctrl+key on PC keyboards, while they use Option+key on Mac:

I was only aware of this, while missing a lot of these matches, I admit.
Right Option however can be mapped independently of Left Option, so that we can use ROpt as an equivalent of AltGr, while LOpt may be used as the Numbers modifier.

[…]

We get all the necessary compatibility with compact keyboards, and extended keyboards are allowed but not required, to offer the additional keys without needing to use the emulation.

Moving AltGr to a new mode 0x10 (or higher) would break this compatiblity. It is not a solution to increase the number of groups and implement a group 3 (or higher) for ISO 8995 (Group 3 is already used for Japanese Kana key or for the Multilingual Canadian keyboard with the Group3 selector)

According to the most recent findings of ISO keyboard standard co-designer Karl Pentzlin, who authored the actual German multilingual keyboard, the Common secondary group selector is implemented using a dead key, not a modifier. See Europatastatur, Pentzlinʼs website distributing the layout driver for Windows.

So the only question is where to place the Group 3 selector, or how to emulate it if it's missing on physical keyboards. And if we need to offer a way to "lock" that group 3, which is completely equivalent to lock an alternate keyboard layout, as currently done in Windows with Ctrl+Alt+(F1..F10) or AltGr+(F1..F10), except that these are used instead to switch from one keyboard driver/map/IME to another one developed completely separately.

Switching back and forth between a locale-tailored layout and an ASCII/Programmer layout is admittedly the way many users must actually work, but the intent here is to use not more than one single keypress, and to use this functionality also just to access the upper-row digits without pressing a modifier, on layouts mapping letters on that upper row like what is done for a number of locales. We can extrapolate that a sooner availability of the scheme would have increased the number of interested locales, and may still do so when becoming widespread.

[…]

How to place the "numeric group" selector? NumLock seems to be appropriate (but it can also be emulated by Fn+B0 (before "1" on the top row, if it's not used for "0") or Fn+B10 (after key "9" on keyboards that place "0" before key "1" on the top row), and such emulation is not necessary on non compact keyboards featuring the "numlock" key in the numeric keypad.

But unlike the usual "Numlock" function of PC keyboards, the "numeric group" is not locking (and not even needed for complete PC keyboards with separate numeric keypads, as they all have also a complete set of cursor keys !).

If needed, the numeric group/Numlock can be emulated by "Fn" plus "+" (the last key of the top row).

To reach its goal, the Numbers modifier should be placed below, ideally on the key to the left of the Space bar. Its functionality is not covered at all by the physical numpad, that lacks either comma or period on most common hardware, a key for the NNBSP used as a thousands separator and to append symbols, these and much more. Please see a complete layout on dispoclavier.com.

comment:9 Changed 3 months ago by mark

  • Phase changed from dsub to spec-beta

comment:10 Changed 3 months ago by Marcel Schneider <charupdate@…>

Exhaustive list of modifiers and layout toggles as provided by major platforms

Trying to complete the table as intended, Iʼve come up with a list that may have some chances of being pretty exhaustive:

http://charupdate.info/unicode/revision/tr35/33-50/tr35-keyboards.html#Possible_Modifier_Keys

Iʼm using many of them with my driver-based keyboard layout on my Windows-powered netbook, and thatʼs close to what Iʼm going to propose as a national standard layout, with the potential of being ported to numerous other locales.

comment:11 Changed 3 months ago by Marcel Schneider <charupdate@…>

Prototyping Programmer toggle and Numbers modifier

Currently a prototype does exist only for azerty. Now a US-qwerty prototype is to be developed following the brandnew sample layout charts available at the following URL:

http://charupdate.info/doc/kbenintu/

comment:12 Changed 3 months ago by Marcel Schneider <charupdate@…>

The US-QWERTY with additions prototype is currently under development (charts are now updated) and should soon be available as a set of layout drivers (kbenintu.dll) for Windows, and a configuration file (kbenintu.keylayout) for macOS.
For Linux, ChromeOS, Android, iOS (and other OSes) Iʼll not be able to provide, however.

comment:13 Changed 3 months ago by Marcel Schneider <charupdate@…>

Note: Just I wonʼt be able to provide, which doesnʼt mean it isnʼt implementable. E.g. according to informations Iʼve got about Linux, there is the additional layout toggle as well as additional modifiers.

Please be aware, too, that on Windows, some keys are being remapped using the built-in Scancode Mapper.

comment:14 Changed 3 months ago by Marcel Schneider <charupdate@…>

comment:15 Changed 3 months ago by Marcel Schneider <charupdate@…>

Prior to release of US-qwerty working model for Windows, charts have now been updated:
http://charupdate.info/doc/kbenintu/#N
Deadlist is being updated.
After this release, top priority will be coding up a working model for macOS.

Sorry for delay.

comment:16 Changed 3 months ago by Marcel Schneider <charupdate@…>

Working model for beta-testing available

Properly itʼs no more than an alpha version. Significant parts are functional, however.

http://charupdate.info/doc/kbenintu/ contains all download links (sources, drivers, workbook).

For now Iʼll hurry up coding a corresponding keylayout for macOS.

comment:17 Changed 3 months ago by mark

There was a lot of feedback on this PRI. The keyboard group has made some modifications based on feedback, but decided to leave other features for consideration for a future version.

comment:18 Changed 3 months ago by mark

  • Milestone changed from 33 to upcoming

comment:19 Changed 3 months ago by Marcel Schneider <charupdate@…>

Thank you, thatʼs fine. This leaves time to refine certain points, finalize implementations and write up documentation.

comment:20 Changed 3 months ago by Marcel Schneider <charupdate@…>

Beta-testing available for macOS

Keylayout files are now available for information and testing, linked at:
http://charupdate.info/doc/kbenintu/#macos

View

Add a comment

Modify Ticket

Action
as accepted
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.