Generate 'host' characters for non-US keyboard layouts

General
1

This is a follow-up thread to Key id symbols wanted - #25 by maexxx

I’ve created a python tool that attempts to auto-learn the host keymap: GitHub - mhantsch/uhk-learn-layout: Tools to enable an UltimateHackingKeyboard (uhk) to learn the keyboard layout of the host system

The idea is to get the UHK to understand the host keymap to be able to generate specific characters on the host system when the host is using a non-US keymap.

It would help all those users that use non-US keyboard layouts – for example those that buy a German labeled UHK, and then get confused when Agent shows them a US keyboard, and when they want to send a / character, they have to use & (because that is the US symbol on Shift-7 where their German keycaps have the / symbol). Agent doesn’t even show them the labels they have on their actual keycaps. I think the UHK software can do better than that.

At the moment, the UHK knows it’s internal key ids (a fixed id for each hardware key), it understands scancodes (which are configurable in Agent), and it labels keys with key names according to the US layout. Those labels are mapped to the scancodes.

But also those key names are used in macro commands such as tapKey, and here they are directly mapped to key ids (based on US layout positions) and do not undergo the scancode mapping.

You also have the host character that is produced by the host OS. (The scancode gets mapped to a character by the host keymap.) That’s a concept that currently is not anywhere in Agent or the UHK firmware. I believe it should be introduced.

Let’s assume we can somehow find out the key id to host character mapping, and load that into an in-memory table for the firmware to use. Maybe even with some macro commands:

set hostmap.<character> <keyid> <mods>
(or the other way around)

Then - as a starting point - I would introduce additional tapKey-like function, something like:

tapHostKey y

which would consult the hostmap first to convert character y into the key id it needs to tap together with any modifiers (Shift, AltGr).

Send text commands would also allow to send hostmap-converted strings etc.

Maybe this is the starting point for a discussion how to get this done (in stages?).

2

tapKey r does not care to tap a key (/keyid). It just adds a scancode into a USB report. (Yes, it is a misnomer.)

Also, let me remind you that dynamic memory is quite a scarce resource in UHK, and dealing with tables of names of variable length is problematic.

I am also not clear on whether you want to apply this mapping to char/KEYABBREV ↔ scancode mapping or to name ↔ KEYID mapping, since the original thread was about keyids, not scancodes.

Assume a keymap that has all keys bound to the x scancode, and just one key with a macro that contains a train of

ifGesture 1 tapKey a
ifGesture 2 tapKey b
...

If you are truly speaking about keyid ↔ symbol mapping, how would you replace the 1, 2, … keyids in the above example?

3

Oh, I did not realise that is how it was implemented. Now I understand why tapKey does not undergo the actual layout mapping of the UHK.

I don’t know the details of how the firmware works. I just look at it from the outside, and therefore I am deducing what may be going on inside. Now you have just provided me new information and reshaped my mental model.

I now also understand why you were writing earlier (in the other thread) about scancodes. I guess I don’t really need the key ids for what I am trying to achieve. Rather, the UHK needs a mapping from scancodes to host characters (or actually the reverse direction). But generating that is easy, it can be done the same way as uhk-learn-layout.py does it now, because each key in the default US layout yields exactly one scancode. So that’s an easy map from key id to US layout key name to US layout scancode. tapKey r will send the scancode for the r position on the US QWERTY layout, and that position also has a certain key id on the UHK (in the US QWERTY layout).

So at the moment I have a key id to host character map (in the python code), but that can be converted into a scancode to host character map. So let’s assume we have that map.

tapHostKey should probably be renamed to sendHostCharacter. (But then, how do you press or release or hold it?) Or maybe tapHostCharacter?

sendHostCharacter r would now find the r character in the host character map and derive the scancode and push that one onto the USB sending queue (or however the keypress reports are queued for sending).

You are concerned about memory consumption of that mapping table. The UHK has 64 keys. Some of them would not be included in the character map (Enter, Backspace, Tab, modifiers etc.) Basically you need only keys that produce a host character in the standard US QWERTY layout. That is 16 keys less, so there’s a total of 48 keys. We are mapping each of them with up to 4 layers: Base, Shift, AltGr, AltGr+Shift. For each of those 48*4 possible characters we need to store the character that was produced at the host (1-2 bytes) and the scancode plus modifiers, maybe a total of 4 bytes. So the whole map is about 768 bytes. How much memory do you have at your disposal?

What’s a good in-memory data structure to convert from characters to scancode, considering that character codes may be spread apart across the UTF-8 space for some of the international characters. Probably a lookup table for standard ASCII characters, and then a searchable list for anything outside that range.

Wildly jumping thoughts: Agent doesn’t do any compilation of macros, not even tokenisation of commands and other keywords, is that correct?

First, you are right, I think I need to speak about scancode to symbol mapping, not key ids.

Second, I don’t know how to solve your example. If you have several keys that produce the same symbol, you can’t address them by symbol - it won’t be unique. But that discussion was part of the other thread.

In this topic here I want to focus on readability of strings and keypresses that I want to send out, and the host is configured with a non-US-QWERTY layout.

Summary so far:

  1. A tool can be built (or it could become part of agent) that triggers the UHK to send certain strings and keypresses, and learns the host keymap. A scancode ↔ host symbol map can be generated. Proof of concept is uhk-learn-layout.py.
  2. I suggest to add new macro commands that convert the strings / key symbols through the previously generated map to scancodes, and UHK will send those scancodes.
  3. A data structure for in-memory representation of the mapping table(s) has not been defined yet
  4. At the moment, this logic has not been designed with support for dead keys. I would ignore them for the moment. I have ideas how to handle them a bit better, but even without them, I think the mapping would be useful already.

Eventually another round of brainstorming is needed to see how Agent could be enhanced to show host symbols on the UI, and in pop-ups when you select which symbol (but actually: scancode) to map to each key. But that is independent from the firmware additions discussed above.

4

tapHostKey should probably be renamed to sendHostCharacter. (But then, how do you press or release or hold it?) Or maybe tapHostCharacter?

tapHostKey for consistency. Or even better, leave it at tapKey, and have the default map mapped according to current enus reverse mapping

So the whole map is about 768 bytes. How much memory do you have at your disposal?

A few kilobytes. I would be reluctant to allocate 768 bytes to a feature that does not bring significant benefit, but let’s assume we can do that.

What’s a good in-memory data structure to convert from characters to scancode, considering that character codes may be spread apart across the UTF-8 space for some of the international characters. Probably a lookup table for standard ASCII characters, and then a searchable list for anything outside that range.

I think sorted array with binary halving search works just fine.

Wildly jumping thoughts: Agent doesn’t do any compilation of macros, not even tokenisation of commands and other keywords, is that correct?

Correct. If we change our minds on this, tokenisation will be taking place in firmware on config load.

First, you are right, I think I need to speak about scancode to symbol mapping, not key ids.

:+1:

Second, I don’t know how to solve your example. If you have several keys that produce the same symbol, you can’t address them by symbol - it won’t be unique. But that discussion was part of the other thread.

That’s fine now that we know you consider only scancode ↔ character mapping.

5

Why? With long macros, the config space gets exhausted quickly. You could still keep the tokeniser in the firmware for additional commands that don’t have a token equivalent in Agent yet, and for the case where you send a text command to the firmware over USB. But it would save space in flash memory if lot of recurrent text strings were tokenised.

max@max-framework:~/src/uhk-learn-layout$ python uhk-learn-layout.py --generate-macro --use-altgr | wc -c
15174

15k of config memory for 1 macro. Yes, it’s a long one, but tokenised it would probably compress to something like a tenth of its uncompressed size.

(base) max@max-framework:~/src/uhk-learn-layout$ python uhk-learn-layout.py --generate-macro --use-altgr | gzip | wc -c
1401

(I realise that gzip is not tokenising, but it would get somewhere to a similar range.)

6

Why?

Compatibility and simplicity. I suspect that dealing with different Agent ↔ firmware versions would be pain.

7

Oh well, it’s getting full, too:

image
image1155×129 12.2 KB

You already upgraded the config format in the past and offered an upgrade path in firmware and Agent. Different levels of tokenising would not be much different. But yes, it is simpler to just not do it, at the cost of config storage.

This was a tangent. I’l get back to the main topic of scancode conversion now.

OK, let’s assume we have a host symbol ↔ scancode conversion map, and it’s somehow loaded into firmware memory.

Is a single tapKey command that always runs the symbol → scancode conversion sufficient, or do we need tapHostKey (including conversion) and tapKey (without conversion)? I am wondering if there is a use case where someone who is using a non-US QWERTY keymap needs to sent specific scancodes that should not originate from their conversion tables. Something in a macro that means “no matter what host keymap you have, I need you to press the third button in the second row” (i.e. a specific scancode).

Or would there be a modifier such as nomap and you would write either tapKey r or nomap tapKey r (similiar to final and other command modifiers). nomap meaning: do not use scancode conversion in this command.

Final set of thoughts:

We’ve just discussed tapKey so far, so the execution of keypresses from macros. (I am assuming tapKeySequence would work the same way of course.) What about the regular processing of keys that do not have macros assigned? Can I assume that a Keypress mapping in Agent results eventually in the same functions in the firmware as a (tap|press|release)Key command? Would these automatically undergo the same scancode mapping? When Agent displays “Scancode”, it actually shows a symbol, not a numeric scancode. This should probably display a mapped symbol?

image
image1040×477 30.9 KB

So, once Agent knows the host keymap, it should probably do two things automatically:

  1. Show the mapped symbols as the Scancode inside the Agent UI
  2. Upload the hostmap to the firmware config

Or should step 2 be left to $onInit and $onKeymapChange macros? Can we keep several maps in the config and activate one from a macro? What if I have a UHK which I regularily use with two different computers, one configured for a French keyboard and one for German? What if it is just one computer, but two different users with different languages use it?

The UI of Agent should somewhere show that a host map is being applied. Agent also needs a “Detect host keyboard layout” button, or should it automatically do that? (probably not)

8

I wanted the add UK, German, and Nordic display options in Agent, according to the keycap layouts we provide, to show relevant symbols. I originally intended this to be an Agent-specific setting, which wouldn’t affect macros.

We can eventually implement a more sophisticated solution, as suggested, if we agree on one, but it’s currently not a priority.

uhk-learn-layout.py should be given executable permission, and autocrlf should be enabled on the repository level. Otherwise, Linux throws:

bash: ./uhk-learn-layout.py: /usr/bin/python3^M: bad interpreter: No such file or directory
9

Thanks for the hint. If you follow the examples in the README, you run it as python uhk-learn-layout.py --help, which will work.

10

I think you can go either way now, and they are independent.

  1. Auto-detect a host layout (or allow the user to manually select), and show appropriate symbols for each scancode in Agent.
  2. Firmware enhancements for mappings in the UHK itself.
11

Is a single tapKey command that always runs the symbol → scancode conversion sufficient, or do we need tapHostKey (including conversion) and tapKey (without conversion)? I

I think just tapKey suffices.

Something in a macro that means “no matter what host keymap you have, I need you to press the third button in the second row” (i.e. a specific scancode).

I don’t think there is such usecase.

We’ve just discussed tapKey so far, so the execution of keypresses from macros. (I am assuming tapKeySequence would work the same way of course.) What about the regular processing of keys that do not have macros assigned? Can I assume that a Keypress mapping in Agent results eventually in the same functions in the firmware as a (tap|press|release)Key command? Would these automatically undergo the same scancode mapping? When Agent displays “Scancode”, it actually shows a symbol, not a numeric scancode. This should probably display a mapped symbol?

I don’t understand this paragraph at all, I believe the bottom line is that you need the translation at following places:

  • in firmware for macro SHORTCUT tokens and write parameters.
  • in agent from scancodes to labels.
Upload the hostmap to the firmware config

Or should step 2 be left to $onInit and $onKeymapChange macros?

Given that Agent needs the maps to do its own translations, it has to be able to read the maps, so it should probably be native part of the config, not a $onInit content. But I guess having a command to change the mapping is reasonable.

Can we keep several maps in the config and activate one from a macro? What if I have a UHK which I regularily use with two different computers, one configured for a French keyboard and one for German? What if it is just one computer, but two different users with different languages use it?

Then just one map in native config should be plenty and different maps would be to be switched by macros…