Technical report · Base-model pretraining

KeelCore: A 752M Encoder–Decoder Base Model for LoRA Adapters

Model 752M-parameter encoder–decoder base model Trained 15 Sep – 25 Oct 2026 Hardware 1 × NVIDIA A100 40 GB

Abstract

1Model

KeelCore is a base model. A keel is the spine of a ship, the part everything else is fitted to. KeelCore plays that role for text models: it is pretrained once on general language, and each task is then added on top as a small LoRA adapter instead of training a new model from scratch. Further adapters will be added to the same base. This report covers the base model only.

KeelCore is a sequence-to-sequence Transformer. A bidirectional encoder reads the entire input before generation starts, and the decoder attends back to it through cross-attention at every step. A task that must transform given text can therefore copy a span verbatim as a lookup rather than an act of memory, and keep URLs, code, numbers and names exactly as written. That fidelity requirement drove the architecture, and it is what makes the model a good carrier for adapters.

Built to carry adapters

Property	Detail
Small adapters	A LoRA adapter of rank 16 on every linear layer trains 18,287,616 parameters, 2.4% of the model; the base weights stay frozen while it trains
Room for new tasks	The vocabulary reserves 12 style tokens, 64 command-line task tokens, 36 spare tokens and two rewrite markers, so a new task needs no embedding resize and no change to the base
One tokenizer for the work	Built for English prose, source code and command-line text together (see Tokenizer below)
Runs on a device	Weight-only int8 ONNX export, 0.76 GB on disk, 98.8% top-1 agreement with fp32 (section 8)
Adapters in the browser	KeelCore is meant to run in a web browser with its adapters. ONNX Runtime Web could not load LoRA adapters, so we added that support ourselves: a `LoraAdapter` API and an `activeLoraAdapters` run option for the default (non-proxy) run path. It was contributed to ONNX Runtime and merged on 6 October 2026 (pull request 32886). One adapter file can be reused across sessions, and it applies only to the run it is passed to.
Evidence	In a same-recipe LoRA study of 14 models (section 9), adapters on KeelCore score the highest SARI and chrF, level with the best of the others

Architecture

Tokenizer

The tokenizer was chosen with the adapters in mind: every task built on KeelCore reads the same vocabulary, so it has to serve English prose, source code and command-line text together. It is a 33,728-piece SentencePiece unigram model with byte fallback, so every input is encoded without an unknown token. Text normalisation is off and whitespace is kept, so indentation, tabs, newlines, URLs and numbers (digits are split one per piece) come back exactly as written: in a check of 500 documents from each of five datasets, every document decoded to precisely its original text. On sampled text it averages 4.3 characters per token on educational and general web text, 5.4 on synthetic textbooks, 2.9 on source code and 3.7 on command-line pages. The command-line side has its own special tokens.

Special tokens	Pieces	Purpose
Command-line vocabulary	896	Whole pieces of literal command-line text: 219 command names, 384 flags and 293 paths (for example `apt-get`, `--no-edit`, `/etc/nginx/`), so that they are not split into fragments
Command-line task tokens	64	Reserved `<cli_reserved_N>` tokens for a command-line helper task, so that task can be trained later without resizing the embedding table
Span sentinels	256	One per masked span in the denoising objectives
Mode and style tokens	15	Three objective modes and twelve rewriting styles
Rewrite markers and spares	38	Two markers around the text to rewrite, and 36 spare tokens

The command-line pieces are admitted by a rule, not by frequency alone. Text that is also an English word or a word beginning, such as cat, is left out, because a piece cat would cut category into cat and egory and make English text longer. 229 such candidates were rejected. What is admitted either contains a character that English words do not (a hyphen, slash, dot, underscore or digit), is a capitalised file name such as Dockerfile, or is a command name checked one by one against the prose corpus.

Measured against the earlier 32,832-piece version of the tokenizer, which lacks the 896 pieces:

Text	Tokens before	Tokens now
`ls -la /dev/null`	7	3
`chmod 755 /var/www/html/index.html`	15	10
`docker run --rm -it ubuntu`	7	5
`git commit --amend --no-edit`	9	7
`find . -name '*.log' -exec rm {} +`	13	11
`ssh-keygen -t rsa`	7	6
`apt-get install`	4	4
`sudo systemctl restart nginx.service`	7	7

On sampled text the effect is what the design intends. Command-line text gets shorter (tldr pages 0.9% fewer tokens, command cheat sheets 2.5% fewer), while English and code are essentially unchanged (educational web 0.05%, general web 0.04%, source code 0.01% fewer; tokens per word 1.415 to 1.414 on educational web text). Command text is a very small part of the pretraining corpus, so the gain shows up in command-line use rather than in the training budget.

2Data

The pretraining data comes from seven public datasets, all under licenses that allow commercial use: educational and general web text, synthetic textbooks, permissively licensed source code, public-domain books and two small command-line references. The token budget is 63.05 billion raw tokens under the model's own tokenizer, which is 68.48 billion processed tokens once the sentinel tokens and objective formatting are added. The SentencePiece tokenizer was built for English and command-line text together; its command-line special tokens are described in section 1.

Figure 1. Token budget by dataset, in billions of raw tokens under the model's own tokenizer.

How the data was read

Files were read one at a time in a fixed shuffled order, and each file holds a single dataset, so training saw blocks of consecutive steps from one dataset. Lines were shuffled within files. From step 81,000 a separate reader served the cooldown blend, which mixes datasets in every batch, and one batch in five carried a synthetic rewriting task built from the prose datasets. Over the run the model processed . No dataset is read more than once.

3Training recipe

Pretraining used the UL2 mixture of denoisers. Every batch mixes three objectives so a single model learns to fill in, reconstruct and continue text, which are the three skills a rewriter draws on:

The learning rate followed a warmup–stable–decay schedule. The final 10% of steps also switched the data to a curated cooldown blend, weighted toward educational text, code and command-line material, with a synthetic rewriting exercise in one batch in five.

Warmup0–900

Stable · lr 3e-4steps 900 – 81,000

Decay81k–90k

The decay stage: cooldown data and the synthetic rewriting task

The last 9,000 steps (81,000 to 90,000) changed two things at once: the learning rate decayed to zero, and the data stopped following the on-disk order. A cooldown reader drew documents from a curated blend in every batch, weighted toward educational web text, textbooks, permissively licensed code and command-line references. Each source was capped at about eight passes over itself so the small command-line sets were not memorised. The reason for changing the data as well as the learning rate is that the decay alone mostly lowers loss, while an up-weighted high-value mixture is what tends to move downstream quality.

One batch in five (20%, drawn per batch with a seeded generator) was not a UL2 batch at all. It was a synthetic rewriting exercise, in a fixed input shape that adapters built on KeelCore can use: the encoder receives <style:grammar> <text_to_rewrite> damaged text </text_to_rewrite> and the decoder must produce the original text. The pairs are built on the fly from ordinary prose documents from the pretraining datasets, so the base model meets the rewriting frame, and learns to copy what is already correct, before any adapter is trained. The reserved style and frame tokens were already in the frozen vocabulary, so this needed no change to the embedding table.

How the text is damaged

The one rule is that the original must be the unique right answer. Every operator produces a genuine writing error whose correction is the source text. There is deliberately no synonym swapping: replacing "big" with "large" is not a mistake, and a target that reverses it would teach the model to edit text that was never wrong. Code, URLs, paths, numbers, identifiers, e-mail addresses and anything in backticks or angle brackets are protected spans that pass through unchanged on both sides, and lines that are not prose (code, tables, headings, bare links) are copied verbatim, so every example also trains copy-through.

Operator	Example of the damage	Weight
Confusable word	their / there, is / are, went / gone, its / it's	4.0
Dropped function word	"cat sat on mat"	3.0
Typo	transposed, missing, doubled or keyboard-neighbour letter; first letter kept	3.0
Capitalisation	"paris", "i think"	2.0
Preposition swap	in / on / at, to / for	1.5
Dropped apostrophe	"dont", "Johns"	1.5
Doubled word	"the the"	1.0
Swapped neighbours	"sat the cat"	1.0
Run-on sentence (per document habit)	comma splice, dropped full stop, or missing capital	by profile

Three things stop the inverse from being a lookup table. Severity is sampled per document from three profiles, so the model cannot assume how much is wrong. The operator is sampled per word, so the same word is a homophone swap in one document and a typo in the next. And a slice of documents is left untouched, because "this text is already correct, copy it" is a case the product meets constantly.

Profile	Probability	Errors per word	Lowercased	Punctuation stripped	Run-ons	Pairs seen
Light	35%	3–8%	5%	0%	10%	430,220
Medium	40%	8–18%	15%	10%	25%	501,451
Heavy	25%	18–35%	40%	35%	40%	314,445
Untouched (copy)	5% target	0	–	–	–	79,280

Documents with fewer than eight prose words are skipped, and a document that does not fit the encoder or decoder length budget is skipped and counted rather than cut. In the run, the task read 1,376,936 documents, skipped 51,540 as not prose (3.7%), skipped none for length, and emitted 1,325,396 pairs containing 80.1 million damaged words, about 60 per pair, from 0.55 billion source tokens. The untouched share came out at 6.0% against the 5% target. Its training loss fell to about 0.05 nats per token by the end (Figure 15), far below the 1.0 to 2.0 of the UL2 denoisers, because most of the output is a copy of the input. That is the intended behaviour: the task teaches precise edits inside a faithful copy, not new text.

Why this is in pretraining

KeelCore is a general base model. Seeing the style token and the text-to-rewrite frame during pretraining means that adapters using them start from a model that already knows them, instead of meeting them cold in fine-tuning. The reader's random state and position are saved in every checkpoint, so an interrupted run replays the same damage on resume.

Optimizer parameter groups

Learning rates at the peak of the schedule, as logged per group. The two special groups exist because AdamW moves every parameter by about the learning rate per step whatever its size, which was too slow for the small position-bias tables and too fast for the small initial scale of the query projections (see section 5).

4Training dynamics

Figure 2. Held-out loss (nats per target token, lower is better). The quick evaluation is a fixed held-out set scored every 1,000 steps; the master evaluation is a larger held-out set scored every 10,000 steps and used for checkpoint selection. The shaded band is the decay phase.

Figure 3. Held-out loss per objective (quick evaluation).

Figure 4. Cumulative tokens processed, in billions: encoder inputs and graded decoder targets.

Figure 5. Learning rate as executed, from the training log.

Figure 6. Median gradient norm per 500 steps, before clipping at 1.0.

Figure 7. Throughput in tokens per second, averaged per 500 steps.

Figure 8. Peak GPU memory allocated per 500 steps, against the 40 GB device.

Master evaluations

The master set (50,000 records) is scored every 10,000 steps and alone decides which checkpoint is kept as best. Because the corpus is read one source at a time, a checkpoint saved inside a block of one source can read high; the last column gives the position at each evaluation.

Step	Master loss	Change	[R]	[X]	[S]	Corpus position at the checkpoint
10,000	2.9456		2.137	3.038	3.152
20,000	2.8074	-0.138	1.865	2.892	3.086
30,000	2.8152	+0.008	1.909	2.895	3.085	191 steps into a block of books; web held-out loss was temporarily worse
40,000	2.5562	-0.259	1.696	2.650	2.783	Inside FineWeb-Edu, 562 steps after a file switch
50,000	2.5070	-0.049	1.629	2.623	2.706
60,000	2.4909	-0.016	1.559	2.593	2.737	Near-clean (FineWeb-Edu, 169 steps after C4)
70,000	2.5099	+0.019	1.686	2.653	2.642	Inside a block of source code (236 steps in); expected to read high
80,000	2.4492	-0.061	1.543	2.562	2.666	Last master evaluation before the decay
90,000	2.3217	-0.127	1.413	2.478	2.470	End of the decay; best of the run

Optimizer and system health

Figure 9. Training loss, median per 500 steps, and per objective per 1,000 steps. The bumps are changes of corpus source, not instability. Per-objective training losses are measured on the batch just stepped on, so they read slightly low. From step 81,000 batches include the rewriting task, so training loss is not comparable across the boundary.

Figure 10. Share of steps whose gradient norm exceeded the 1.0 clip (line) and steps with a norm above 5 (bars), per 1,000 steps.

Figure 11. Size of one optimizer step relative to the weights, per parameter group (median per 1,000 steps, log scale). The weight matrices' ratio rises through the stable phase as weight decay shrinks them, then falls with the learning rate.

Figure 12. Root-mean-square weight per parameter group (log scale). Decayed matrices shrink steadily; the position-bias tables grow from 0.03 to about 6.

Figure 13. Largest entry of the relative position-bias tables, logged during training.

Figure 14. GPU utilisation (left axis) and SM clock (right axis), median per 500 steps. Telemetry stopped at step 40,985 after a driver error; training was unaffected.

Figure 15. Training loss of the ownership fingerprint (logged at each of its 160 injections) and of the synthetic rewriting task in the cooldown (log scale).

5The run, stage by stage

The run was instrumented end to end. Every saved checkpoint went through a battery of CPU probes while training continued on the GPU, and each was written up in a per-checkpoint report. Those reports are the source for this history.

Sessions

The run resumed from a saved checkpoint five times in its first five days, each time after a planned code change; 81 steps were recomputed in total. From step 10,001 it ran as one session for 36 days.

Session start	Resumed from	Steps logged	Reason (from the repository history)
15 Sep 07:32	0	1 → 2,010	First session
16 Sep 05:06	2,000	2,001 → 2,023	Dashboard changes (three resumes within 35 minutes)
16 Sep 05:24	2,000	2,001 → 2,024	Dashboard changes
16 Sep 05:41	2,000	2,001 → 4,008	Dashboard throughput tracker seeded from the resume step
17 Sep 03:30	4,000	4,001 → 10,016	Text samples captured every 500 steps instead of every 50
19 Sep 21:21	10,000	10,001 → 90,000	Resumed with the ownership features (hash chain, fingerprint, signature); ran without interruption to 25 Oct 19:12

Issues found and how they were handled

Everything below was found by the per-checkpoint analysis or the training dashboard while the run was in progress. Causes are the measured ones; where a number is given it comes from the run's own records.

When	What was seen	Cause (measured)	What was done / outcome
12 Sep · first attempt, step 4,000	Encoder ignored word order (shuffle cosine 0.998, where 1.0 means word order is ignored)	AdamW moved the position-bias tables too slowly, and weight decay pulled them back	New optimizer groups (position bias ×100, queries ×0.1, no weight decay); restarted from scratch. Cosine 0.48 by step 5,000 and 0.40 at the end
15–19 Sep · steps 2k–10k	Five resumes from checkpoints	Planned code changes to monitoring and ownership	81 steps recomputed in total; bit-identical resume verified
Whole run	1,196 steps in the stable phase with a loss ratio above 1.2 of the previous 100-step median	The corpus is read one single-source file at a time, and sources differ in loss (code about 1.1, C4 about 2.7). Which source is being read explains 84% of the variance of the smoothed training loss over 20k–37.5k steps	No action; trends are read on the fixed held-out sets
4k–17k	Share of steps clipped at gradient norm 1.0 rose from 2% to 34%	About 95% of the gradient norm comes from the attention queries, which carry about 8× the gradient because the attention logits are not scaled by 1/√d. Under Adam the effective step-size cost is about 2–3%	Kept the clip at 1.0
32k–73k	Rare gradient-norm outliers: 535 steps above 5 in the whole run, 12 above 1,000, the largest 20,349 at step 47,830	Rebuilding the exact batches (steps 36,227, 38,869, 47,830) gave ordinary losses and gradients: a transient while the model adapts to new data, mostly in the first 300 steps of a file	Clipping absorbed them; no skipped steps and no loss damage
29 Sep · 30k	Master loss 2.8152, worse than 20k (2.8074)	Checkpoint saved 191 steps into a block of books	Confirmed by 40k (2.5562)
3 Oct 19:30 · step ~40,985	GPU telemetry stopped (NVML error)	Driver telemetry failure in the dashboard	Training unaffected; GPU charts end at that step
9 Oct 06:36	Automatic system update replaced the NVIDIA user-space libraries under the running job	Unattended upgrade to a version that did not match the kernel driver	Running job unaffected; libraries restored, packages held and excluded from automatic upgrades
52k–64k	Held-out loss on datasets not being read rose while web prose improved	Those datasets were not read in that interval (forgetting of unread sources)	Reversed in the cooldown (code −0.20, books −0.10 on averaged models)
66k–90k	One encoder channel (475, layers 21–23) grew large: max/RMS 716 at 90k	A massive-activation channel firing on very common tokens	Harmless in bf16; int8 ONNX export checked on the final checkpoint (loss 2.9201 → 2.9207)
Whole run	Base model loops under plain greedy decoding (14 of 32 prompts at 90k)	A property of base language models under greedy decoding	Reported as a property of the base model, not treated as a training fault
Corpus	6.3% of sampled evaluation documents also appear whole in training; two run-manifest files (about 11K tokens) are read as text	Cross-source duplicates (FineWeb-Edu and C4 share Common Crawl pages); a file-suffix rule	Known; about 0.01–0.02 nats on the final evaluation; to be fixed before a future run
40k, 52k, 64k reviews	Whether to start the cooldown early	Weight averaging showed the stable phase still had headroom	Kept the full 90,000-step schedule; the decay alone took the master loss from 2.4492 to 2.3217

6Model quality during training

Loss shows that a model is learning, not what it has learned. The probes below measure specific capabilities on every checkpoint, using held-out documents the model had not trained on.

Figure 16. Loss on unseen documents from each corpus source, measured at every checkpoint. Educational web, general web, textbooks and code reach their best on the final checkpoint; books peaked at step 69,000.

Figure 17. Word order. Cosine similarity of encoder states before and after shuffling the input (1.0 means word order is ignored).

Figure 18. Local context. Extra loss, in nats, when the words beside a gap are scrambled. Higher means predictions are grounded in nearby context.

Figure 19. Grounding in the input. Loss penalty, in nats, when the decoder is given another document's encoding. Rewriting depends on this signal.

Figure 20. Token accuracy. Top-1 accuracy on held-out text tokens when filling short gaps ([R]) and continuing text ([S]).

Figure 21. Calibration. Expected calibration error, with a 5-checkpoint rolling median. Single checkpoints vary with the corpus source being read when they were saved; the median holds near 1%, meaning the model's confidence matches how often it is right.

Figure 22. Repetition. Greedy continuations, out of 32 held-out prompts, that fall into a repetition loop, with a 5-checkpoint rolling median. The count falls from 30 at step 1,000 to 14 at step 90,000; the base model still loops under plain greedy decoding.

What the model writes

The dashboard stored greedy continuations of one fixed prompt from step 5,000 on: “Core Talent is currently looking for a Senior Environmental Advisor for a leading principal contractor in London on a permanent…”. These are raw base-model outputs with no decoding constraints. Later checkpoints stay on topic and write more fluently, and they still invent details, as base models do.

Step	Greedy continuation (first 230 characters)
5,000	contract. The role will be based in London and will be based in London.
10,000	project. The company is currently looking for a Senior Environmental Advisor for a leading principal contractor in London on a permanent project.
20,000	project. The role is to help you to develop and implement a sustainable and sustainable business model.
40,000	contract. The role is open to all senior contractors and contractors in the UK.
60,000	contract. The role will be based in London and will involve working closely with the project team to ensure that the environmental impact of the project is minimised. The role will involve working closely with the project team to …
81,000	basis. The role is based in the City of London and is based in the City of London. The role is based in the City of London and is based in the City of London.
85,000	contract basis. The role is to provide advice on the environmental impact of the project and to ensure that the project is carried out in a way that is environmentally sustainable. The role is to work closely with the project team…
90,000	contract basis. The role is part-time and requires a minimum of 10 years of experience.

Weight averaging

Uniform averages of nearby checkpoints, scored on the same 16-batch subset of the master set, measured how much the constant-learning-rate phase left on the table. They justified keeping the full schedule and predicted that the decay would deliver.

Comparison	Loss change	Evidence
Average of checkpoints 33k–40k vs checkpoint 40,000	−0.077	master subset, 16 of 16 batches better
Average of 45k–52k vs checkpoint 52,000	−0.131	master subset; best model at that time (2.3196)
Checkpoint 86,000 vs checkpoint 80,000 (the decay)	−0.075	master subset 2.3433 → 2.2679, 16 of 16 batches
Average of 84k–86k vs checkpoint 86,000	−0.021	16 of 16 batches

7Reliability and engineering

A 40-day single-GPU run leaves no room for silent failures. These checks ran throughout:

Ownership

Three independent techniques identify the weights. Their secrets are not part of this report.

Technique	What it does	Result over the run
Checkpoint hash chain	Each saved checkpoint's hash is chained to the previous one	90 checkpoints in the chain, 0 broken links
Trigger fingerprint	A secret prompt–response pair is trained in at intervals	160 injections from step 10,500; its training loss fell 5.11 → 0.15
Embedding signature	A secret bit pattern held in the embedding weights	96 of 96 bits matched at all 80 checks (steps 11,000–90,000)

8Deployment readiness

9Comparison with open models

This section reports KeelCore-752M as a base model against open models of a similar size, all evaluated with one harness, identical prompts and full test sets. It also sets each score against the number of tokens the model was trained on.

Evaluation in progress

Results will be published here

Zero-shot accuracy on 15 standard benchmarks, full test sets, against 14 open models from 355M to 1.3B parameters.
Prediction quality on text published in August–October 2026, after every model's training data was collected: 1,241 Wikipedia articles, arXiv abstracts and source files in eight languages.
A same-recipe fine-tuning study: every model given the same LoRA adapter recipe and rewriting data and budget, then scored on 1,000 held-out rewrites.