CortexLite v1.1

CortexLite is a lightweight, fully offline, CPU-native Causal Transformer text generation engine implemented entirely from scratch in pure Python and NumPy. The project operates without high-level machine learning frameworks such as PyTorch, TensorFlow, or Keras. All neural network forward propagation, backward propagation, analytical gradients, matrix transformations, and optimizer state updates are manually calculated.

Version 1.1 represents a significant architectural evolution: transitioning from a basic feed-forward network to a full Causal (Decoder-Only) Transformer featuring Multi-Head Self-Attention, Causal Masking, Layer Normalization, Learned Positional Embeddings, dynamic vocabulary scaling, and asynchronous multi-threaded data scraping.

CortexLite implements a Decoder-Only Transformer architecture optimized for CPU matrix operations. High-level parameters and hardware optimizations include:

The total parameter count of CortexLite consists of a fixed Transformer core and a dynamic token vocabulary:

• Fixed Core Parameters:
  • Positional Embeddings ($PE$): $32 \times 128 = 4,096$
  • Transformer Layers ($L = 2$):
    • Multi-Head Attention ($W_q, W_k, W_v, W_o$): $4 \times (128 \times 128) = 65,536$ per layer
    • LayerNorm 1 ($\gamma_1, \beta_1$): $2 \times 128 = 256$ per layer
    • Feed-Forward Network ($W_{f1}, b_{f1}, W_{f2}, b_{f2}$): $(128 \times 256) + 256 + (256 \times 128) + 128 = 65,920$ per layer
    • LayerNorm 2 ($\gamma_2, \beta_2$): $2 \times 128 = 256$ per layer
    • Layer Total: $131,968 \times 2 = 263,936$
  • Total Fixed Core: $268,032$ parameters.
• Dynamic Vocabulary Parameters:
  • For a vocabulary size $V$, token embeddings $E$, output projection $W_{out}$, and output bias $b_{out}$ require $257 \times V$ parameters ($128 + 128 + 1$).
  • Total Parameters: $\text{Params}(V) = 268,032 + (257 \times V)$

All matrix operations are executed in static, thread-safe methods within the Ops class:

• Linear Projections: Computes forward affine transformations $Y = XW + b$ and backward tensor contractions yielding $\frac{\partial L}{\partial X}$, $\frac{\partial L}{\partial W}$, and $\frac{\partial L}{\partial b}$.
• Layer Normalization: Calculates mean and variance across channel dimensions per token: $$\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}, \quad y = \gamma \odot \hat{x} + \beta$$ Computes exact analytical backward gradients for inputs, scale ($\gamma$), and shift ($�eta$).
• Multi-Head Causal Attention:
  • Projects input $X$ into Query ($Q$), Key ($K$), and Value ($V$) matrices across $H=2$ heads.
  • Applies Scaled Dot-Product Attention: $S = \frac{Q K^T}{\sqrt{d_k}}$.
  • Enforces strict causality using an upper-triangular mask ($S_{i,j} = -1\text{e}9$ for $j > i$).
  • Computes softmax probabilities $P = \text{softmax}(S)$ and weighted output matrix $O = P V$.
  • Backpropagates through output projection, value matrix contraction, softmax derivative, scaled matrix multiplication, and linear projections.

When new text is ingested, CortexLite tokenizes and identifies out-of-vocabulary (OOV) tokens. Rather than dropping unknown words, it dynamically resizes neural matrices:

• Expands row dimension of Token Embedding matrix $E$ and Adam momentum vectors ($m_E, v_E$).
• Expands column dimension of Output Projection matrix $W_{out}$ and bias $b_{out}$ alongside momentum matrices ($m_{W_{out}}, v_{W_{out}}, m_{b_{out}}, v_{b_{out}}$).

The response generation pipeline incorporates multiple decoding control mechanisms to prevent repetitive looping and maintain semantic focus:

• Top-K Sampling: Restricts candidate token selection to the highest probability subset ($K=24$).
• Temperature Scaling: Softens or sharpens logit probability distributions ($T=0.8$).
• Repetition Penalty: Applies a multiplicative penalty ($1.4\times$) to recently generated tokens.
• Focus Boosting: Boosts logits of non-stop words present in the user prompt ($1.0\times$ default).
• Minimum Length & Punctuation Constraints: Suppresses structural stop tokens until reaching a minimal target token count ($12$ tokens).

CortexLite includes an interactive CLI shell supporting continuous learning, text scraping, state persistence, and real-time generation:

• /eat — Scrapes and trains on 1 random Simple Wikipedia article.
• /eat <number> or /eat random <number> — Concurrently fetches and trains on $N$ random articles (e.g., /eat 10).
• /eat <topic> — Searches Simple Wikipedia for the specified query and trains on the top article.
• /eat <topic> <number> — Searches for a topic and concurrently ingests the top $N$ matching articles (e.g., /eat physics 5).
• /train — Toggles interactive manual training mode. Any plain text pasted into the CLI is tokenized and backpropagated into weights.
• /stoptrain — Disables manual training mode and resumes interactive inference.
• /stats — Displays current network dimensions, active parameter count, and dynamic vocabulary size.
• /save — Forces an immediate disk serialization of the model state (CortexLite.brain).
• /clear — Wipes the network weights, resetting the model to initial random state.
• /exit — Saves the active brain state to disk and safely terminates execution.

• Brain Persistence: Serializes network weights, positional matrices, dynamic mappings, and Adam momentum tensors using Python's pickle library (CortexLite.brain). Upon initialization, load_brain() inspects structure compatibility; if legacy versions (e.g., pre-v1.1 brains) are detected, it gracefully initializes a fresh Transformer instance.
• API Rate Handling: The WikipediaFetcher module intercepts HTTP 429 (Too Many Requests) errors when querying Simple Wikipedia. It executes an exponential backoff sequence ($\text{wait} = \text{attempt} \times 4\text{ s}$) with up to 5 retries.

This repository includes a pre-trained CortexLite.brain file containing approximately 0.9 million (900,000) parameters, trained on Computer Science, hardware specifications, programming languages, and general technology topics.

The initial target of 2.2 million parameters was not reached solely because there was not enough time to complete the training process. Executing full backpropagation and matrix updates entirely from scratch in NumPy on a CPU without hardware acceleration requires considerable compute time per batch. As a result, training was concluded at the 0.9 million parameter milestone.

Users can continue training the model and expanding the parameter count on custom data at any time using the /eat command.

This project is released under the Apache License 2.0. You are free to use, modify, and distribute this software provided that original copyright notices and detailed modification descriptions are retained.