｢‍｣ Lingenic

HSV v1.5

Hierarchical Separated Values

HSV is both a file format (like CSV) and an RFC 20 conformant streaming protocol. A carefully curated set of ASCII control codes standardized for modern use. We forgot about control codes and spent 60 years escaping quotes instead. No escaping in text or shift-encoded binary.* No quotes. All of Unicode is valid data.

* DLE transparency mode escapes exactly one byte: DLE itself ([DLE][DLE] = literal DLE).

CSV  →  1 level    (comma, quoting required)
JSON →  unlimited  (escaping required)
HSV  →  unlimited  + framing + protocol (no escaping)

Parallel parsing: HSV's flat separators (FS, GS, RS, US) are embarrassingly parallel—no escape state, no context dependence. Split on any separator, parse chunks independently on separate cores. CSV, JSON, MsgPack, Protobuf — all stuck in single-threaded processing.

Nested data (SSA/ESA) requires a parallel-scan step: chunks can be tokenized independently, but tree assembly needs a depth-resolution pass (a prefix-sum over nesting depth, O(log n) span). This is still a large win over HTML's byte-serial stateful tokenizer—but it is parallel-scan, not independent chunks. DLE transparency regions are not splittable blind; use shift encoding as the parallel-safe binary path.

Where It Fits

Save .hsv files and grep them, or stream records over a socket with acknowledgments. Binary formats like MessagePack and Protobuf are serialization-only.

HSV is CSV with hierarchy, JSON without escaping, and unlike binary-only formats, you can debug with cat and grep.

Structure

key [US] value                       key-value pair
pair [RS] pair [RS] pair             object
item [GS] item [GS] item             array
key [US] [SSA] nested [ESA]          nested value
[STX] [SSA] children [ESA] [ETX]     nested body (container)
[STX] record [FS] record [ETX]       stream

Encoding canonicality. In open mode (no hsv.typed declaration), the three hierarchy mechanisms—flat IS separators, STX/ETX framing, SSA/ESA nesting—are structurally permissive; the consumer assigns meaning. In typed mode, each type maps to one normative construct: array ≡ GS-separated, object ≡ SSA/ESA nesting.

Streaming

STX/ETX framing solves the streaming problem:

[STX] name [US] Alice [RS] role [US] admin [FS] name [US] Bob [RS] role [US] user [ETX]

Unlike NDJSON, newlines in data don't break parsing. Unlike length-prefix, it's human-inspectable.

Headers (Control)

SOH (Start of Header) adds control/metadata before the data block. Headers are a form of control information—routing, formatting, protocol metadata—distinct from content.

[SOH] hsv [US] 1.5 [RS] content-type [US] users [STX] name [US] Alice [RS] role [US] admin [ETX]

Structure: [SOH] header [STX] data [ETX] — header properties use the same format as data. When present, hsv [US] 1.5 is the recommended first property for version identification. The hsv. key prefix is reserved for control and metadata (e.g., hsv.typed, hsv.schema).

This mirrors the original ASCII design: SOH marks control information (header), STX marks content (text). A message can be header-only (pure control), content-only (pure data), or both.

Mixed Content

Everything outside STX…ETX is ignored—no special comment syntax needed:

This text is ignored
[STX] name [US] Alice [RS] age [US] 30 [ETX]
So is this

Hierarchical Containers

SSA/ESA nest anywhere—in values or in bodies. A container (directory, archive, document) uses SSA/ESA to wrap its children:

[SOH] name [US] file.txt [STX] content here [ETX]           file (body is content)
[SOH] name [US] folder [STX] [SSA]                          directory (body is children)
  [SOH] name [US] a.txt [STX] hello [ETX]
  [FS]
  [SOH] name [US] b.txt [STX] world [ETX]
[ESA] [ETX]

After STX: if SSA follows, it's a container. Otherwise, it's content.

Recursive nesting:

[SOH] name [US] project [STX] [SSA]
  [SOH] name [US] README.md [STX] # Hello [ETX]
  [FS]
  [SOH] name [US] src [STX] [SSA]
    [SOH] name [US] main.rs [STX] fn main() {} [ETX]
    [FS]
    [SOH] name [US] lib.rs [STX] pub fn lib() {} [ETX]
  [ESA] [ETX]
[ESA] [ETX]

Files, directories, archives, documents—same structure. [SOH] header [STX] content [ETX] for leaves, [SOH] header [STX] [SSA] children [ESA] [ETX] for containers. FS separates siblings.

Binary Mode

Two modes for binary data:

[SO] shifted-bytes [SI]           shift encoding (default)
[DLE] [SPA] raw-bytes [DLE] [EPA] binary transparency (BISYNC-style)

Shift encoding: Control codes (0x00-0x06, 0x0E-0x1F, 0x7F-0x9F) are shifted to ASCII (0x21-0x61). Output is valid UTF-8. Pass-through: BEL, BS, TAB, LF, VT, FF, CR (0x07-0x0D).

DLE transparency: For raw binary blobs. Inside [DLE][SPA]...[DLE][EPA], all bytes pass through literally. Only DLE needs escaping ([DLE][DLE] = literal DLE).

type [US] image [RS] data [US] [DLE] [SPA] <raw PNG> [DLE] [EPA] [RS] width [US] 800

Uses original ASCII semantics: SO/SI (1963), DLE/BISYNC (1960s), SPA/EPA (ISO 6429, 1983).

Implementations

Reference implementations in Rust and Python:

github.com/LingenicLLC/HSV

The Control Codes

Twenty-six bytes reserved (22 C0 + 4 C1). Three bytes forbidden. Everything else is data.

Forbidden Bytes

Three control characters are forbidden in HSV streams:

If these bytes appear in data, use binary mode (shift encoding or DLE transparency).

Allowed as Data

Seven control characters are allowed as literal data (not reserved for protocol):

These are common whitespace and formatting characters. They pass through HSV unchanged.

Notation

Control codes are written as [CODE] with the standard abbreviation. Examples: [STX], [ETX], [SSA], [ESA].

Unicode's Control Pictures block (U+2400–U+243F) provides visible glyphs for C0 controls, but C1 controls (SSA, ESA, SPA, EPA) have no standard pictures. The [CODE] notation is consistent across all control characters.

Tooling note. C0-only HSV (flat separators, framing, binary modes) round-trips through any ASCII-compatible tool. Nested HSV uses C1 codes (0x86–0x87, 0x96–0x97) and assumes UTF-8-clean tooling. Some legacy 8-bit tools may strip or rewrite C1 bytes.

In actual data, these are single bytes. The notation makes them visible in documentation.

MIME Types

HSV defines two MIME types based on binary content:

When to Use Each

text/hsv: Use when content is guaranteed UTF-8 safe. This includes:

• Pure text data (no binary encoding needed)
• Shift-encoded binary ([SO]...[SI]) — shifted bytes map to ASCII 0x21-0x60
• HTTP API responses, streaming protocols, anywhere UTF-8 is expected

application/hsv: Use when content may contain raw binary. This includes:

• DLE transparency mode ([DLE][SPA]...[DLE][EPA]) — arbitrary bytes pass through
• Files that embed binary blobs (images, compressed data)
• When binary content is unknown or untrusted

The C1 control codes (SSA, ESA, SPA, EPA at 0x86-0x87, 0x96-0x97) are valid Unicode code points with proper UTF-8 encodings (e.g., U+0086 → C2 86). They do not break UTF-8 validity.

Quick Reference

Reserved:  0x01-0x06 0x0E-0x19 0x1C-0x1F 0x86-0x87 0x96-0x97  (26 bytes)
Forbidden: 0x00 0x1A 0x1B  (NUL, SUB, ESC)
Data:      Everything else (all of Unicode + BEL BS TAB LF VT FF CR)

Structure:
  [US]   key:value
  [RS]   properties
  [GS]   array items
  [FS]   records

Framing:   [SOH] ... [STX] ... [ETX]  (header, data, end)
Nesting:   [SSA] ... [ESA]  (0x86/0x87, in values or bodies)
Container: [SOH] ... [STX] [SSA] ... [ESA] [ETX]  (directory, archive, document)
Binary:    [SO] shifted-byte [SI]  (shift encoding for control codes)
           [DLE] [SPA] raw-bytes [DLE] [EPA]  (DLE transparency for blobs)
Protocol:  [ENQ] / [ACK] / [NAK] / [CAN]
Flow:      [DC1] / [DC3] / [SYN]  (XON / XOFF / sync)
Device:    [DC2] / [DC4]  (connect / disconnect)
Chunked:   [ETB] / [EM]  (end of chunk / data exhausted)
Stream:    [EOT]  (end of stream/connection)

Extension:   .hsv
MIME types:  text/hsv; charset=utf-8  (text-safe, no DLE)
             application/hsv          (binary, may have DLE)