｢‍｣ Lingenic

Shift Encoding

Binary-safe encoding using [SO] and [SI]

Shift Encoding uses two ASCII control characters to make any binary data safe for text transport. Zero overhead for text. Efficient batching for binary. No classification needed.

The Idea

Control codes (0x00-0x1F, 0x7F-0x9F) break text systems. [NUL] terminates strings. [ETX] ends transmissions. [ESC] triggers escape sequences.

Shift encoding moves control codes to safe ranges:

[SO] <shifted bytes> [SI]

• [SO] (0x0E) — Shift Out: enter shifted mode
• [SI] (0x0F) — Shift In: return to normal mode

Inside [SO] ... [SI], control codes are shifted to non-control ranges. Outside, bytes pass through unchanged.

What Gets Shifted

Pass through (7 bytes):

SPACE (0x20) and NBSP (0xA0) are outside the control ranges — they pass through naturally.

Text files pass through unchanged. Zero overhead.

Complete Shift Mapping

C0 Controls → Printable ASCII (+0x21)

C0 shifted: 25 bytes (7 from 0x00-0x06, 3 from 0x0E-0x10, 15 from 0x11-0x1F)

DEL + C1 Controls → Printable ASCII (−0x3E)

DEL+C1 shifted: 33 bytes (0x7F-0x9F)

Total shifted: 58 control codes. All targets are ASCII (0x21-0x61). Output is valid UTF-8.

Binary Transparency Mode

For raw binary blobs, use DLE transparency (BISYNC-style) instead of shift encoding:

[DLE] [SPA] raw bytes [DLE] [EPA]

• [DLE] [SPA] — start transparent block (Protected Area)
• [DLE] [EPA] — end transparent block
• [DLE] [DLE] — literal DLE inside block

Inside the block, all bytes pass through literally. Only DLE needs escaping.

Two modes, different jobs:

Shift encoding is the default. DLE transparency is for when you have dense binary and don't want per-byte shifting overhead.

Batching

Consecutive control codes share one [SO] / [SI] pair:

Input:  0x00 0x00 0x00 (3 [NUL] bytes)
Output: [SO] 0x21 0x21 0x21 [SI] (5 bytes)
              !    !    !

Run of 100 [NUL] bytes: 102 bytes ( [SO] + 100 × 0x21 + [SI] ).

Efficiency

Compare to base64: 33% overhead always.

Why Not Base64?

Is this text or binary?
If binary, encode whole chunk
If mixed, need delimiters
33% overhead, always

Don't classify. Just encode.
Adapts byte-by-byte.
Text passes through.
Binary gets shifted.

Base64 is block-oriented — you encode a classified thing.
Shift is byte-oriented — you encode a stream.

Example

PNG header bytes: 0x89 0x50 0x4E 0x47 0x0D 0x0A 0x1A 0x0A

[SO] K [SI] P N G [CR] [LF] [SO] ; [SI] [LF]
     ↑                           ↑
   0x89                        0x1A
   −0x3E                       +0x21

• 0x89 [HTJ] shifted to 0x4B K (−0x3E)
• P, N, G pass through (printable ASCII)
• [CR], [LF] pass through (safe whitespace)
• 0x1A [SUB] shifted to 0x3B ; (+0x21)

Implementation

Shift Encode

void shift_encode(uint8_t *in, size_t len, uint8_t *out) {
    int shifted = 0;
    for (size_t i = 0; i < len; i++) {
        uint8_t b = in[i];
        // Pass through: BEL-CR (0x07-0x0D)
        int pass = (b >= 0x07 && b <= 0x0D);
        int ctrl = !pass && (b < 0x20 || (b >= 0x7F && b < 0xA0));
        if (ctrl && !shifted) { *out++ = 0x0E; shifted = 1; }  // [SO]
        if (!ctrl && shifted) { *out++ = 0x0F; shifted = 0; }  // [SI]
        *out++ = ctrl ? (b < 0x20 ? b + 0x21 : b - 0x3E) : b;
    }
    if (shifted) *out++ = 0x0F;  // [SI]
}

Shift Decode

void shift_decode(uint8_t *in, size_t len, uint8_t *out) {
    int shifted = 0;
    for (size_t i = 0; i < len; i++) {
        uint8_t b = in[i];
        if (b == 0x0E) shifted = 1;        // [SO]
        else if (b == 0x0F) shifted = 0;   // [SI]
        else *out++ = shifted ? (b < 0x41 ? b - 0x21 : b + 0x3E) : b;
    }
}

DLE Transparency (raw binary)

// Wrap raw binary in DLE transparency
void dle_wrap(uint8_t *in, size_t len, uint8_t *out) {
    *out++ = 0x10; *out++ = 0x96;  // [DLE] [SPA]
    for (size_t i = 0; i < len; i++) {
        if (in[i] == 0x10) *out++ = 0x10;  // escape DLE
        *out++ = in[i];
    }
    *out++ = 0x10; *out++ = 0x97;  // [DLE] [EPA]
}

// Unwrap DLE transparency block
void dle_unwrap(uint8_t *in, size_t len, uint8_t *out) {
    // Skip [DLE] [SPA] at start, [DLE] [EPA] at end
    for (size_t i = 2; i < len - 2; i++) {
        if (in[i] == 0x10) i++;  // skip escape, take next byte
        *out++ = in[i];
    }
}

Properties

• UTF-8 safe — all shift targets are ASCII (0x21-0x61), output is valid UTF-8
• No classification — feed it anything, text or binary
• Streamable — no padding, no block boundaries
• Parallel-safe — [SO] / [SI] are structural, never data
• Transport-safe — no [NUL], [ETX], [ESC] in output
• Readable — text and newlines visible, only control codes shifted

Etymology

[SO] (Shift Out) and [SI] (Shift In) were defined in ASCII (1963) for modal character set switching.

[DLE] (Data Link Escape) was defined for binary transparency in BISYNC (1960s). [SPA] (Start of Protected Area) and [EPA] (End of Protected Area) are C1 controls from ISO 6429 (1983).

The solutions were always there. We just forgot to use them.