Lexer

koskari.lexer

Lexical-only parser and tokenizer for Koskari source.

Provides read-only parsing (no compile/eval) and token emission for editor integration (syntax highlighting, completion).

class koskari.lexer.Token(type, text, start_line, start_col, end_line, end_col)

Bases: tuple

end_col

Alias for field number 5

end_line

Alias for field number 4

start_col

Alias for field number 3

start_line

Alias for field number 2

text

Alias for field number 1

type

Alias for field number 0

koskari.lexer.collect_special_like_names_from_builtins(builtins_dict)

Return the head names of the compiler specials and syntax-rules transformers bound in a builtins mapping.

Walks the merged runtime builtins (bootstrap plus loaded basics) so new specials registered only in Python stay in sync, and user-defined define-syntax names from basics.ki are included when present. The compiler resolves only the unprefixed name as a syntax use, so the compile/ key itself is a plain symbol and is left out.

Parameters:

builtins_dict – mapping from string name to builtin value

Returns:

frozenset of the unprefixed head of each compile/ special

koskari.lexer.default_layout_reader()

Return a process-wide cached LayoutReader.

Suitable for a single-threaded consumer such as the REPL. Consumers that may issue overlapping reads must build their own with make_layout_reader() instead.

Returns:

the shared LayoutReader instance

koskari.lexer.make_layout_reader()

Build a fresh LayoutReader with default atoms, macros, and layouts.

Read-only; no execution. Uses a copy of koskari.bootstrap’s builtin registry as the layout namespace. Each call returns a new instance – reads mutate per-read reader state, so callers that may run concurrently (e.g. an LSP server handling overlapping requests) must not share one reader.

Returns:

a new LayoutReader instance

koskari.lexer.parse(source, filename='<string>', reader=None)

Parse a single top-level form from source.

Returns the parsed s-expression or READ_EOF when there is no form (empty or whitespace only).

Parameters:

  • source – source string

  • filename – filename for error reporting

  • reader – LayoutReader to use; the shared cached reader when None

Returns:

parsed form or READ_EOF at EOF

Raises:

ReaderSyntaxError – on syntax errors

koskari.lexer.parse_all(source, filename='<string>', reader=None)

Yield each top-level form from source until EOF.

A top-level pragma or package-pragma form whose reader settings are all literals applies those settings to the forms after it, as a load of the source would. A pragma form a load would reject is skipped.

Parameters:

  • source – source string

  • filename – filename for error reporting

  • reader – LayoutReader to use; a fresh reader when None

Yield:

parsed s-expressions

Raises:

ReaderSyntaxError – on syntax errors

koskari.lexer.special_names_for_repl()

Return special-form names for the REPL using the full builtins environment

Imports koskari.builtins (which loads basics.ki on first import) and collects Compiled bindings from the module namespace.

Returns:

frozenset of name strings for REPL special-form highlighting

koskari.lexer.tokenize(source, filename='<string>', special_names=None, layout_enabled=True)

Yield Token instances for the given source.

Position fields are 1-based. On malformed input (e.g. unclosed string), yields tokens up to the error and stops.

Parameters:

  • source – source string

  • filename – filename for error context (unused for now)

  • special_names – optional set of special-form names; defaults to SPECIAL_NAMES, the unprefixed head names from the compiler registry

  • layout_enabled – the reader’s reader/layout switch; when False a name: atom never splits into a layout head and a colon

Yield:

Token instances