Aleph Alpha, the Heidelberg company, has released Kolibri, a model built for German and English and aimed at public administration and industry. The weights went up on Hugging Face on 3 October, announced on 5 October. It is a mixture-of-experts model: 78 billion parameters in total, of which only about three billion are used for any given token, so it costs closer to a much smaller model to run.
Two design choices are specific rather than cosmetic. German text makes up about 23% of the pre-training mix, and the tokeniser is tuned for German so that German text uses fewer tokens, which lowers the cost of processing it. The model is also trained to decline to answer when the documents it is given lack the evidence, which matters more in administration than a benchmark score does.
The notable part is the paperwork. Aleph Alpha says every training source was screened against a blocklist of more than 4.5 million URLs, drawing on the European Commission’s Piracy Watch List among others, and that each third-party dataset was checked for licence terms, lawful sourcing and opt-outs before use. The published benchmark results are the company’s own, with some comparisons run in its own setup. Apache 2.0 covers the weights and configuration files only, not the architecture or training methods.
Source: Aleph Alpha, Aleph Alpha releases Kolibri.