open-source / open-weight distinction
you can open-source or open-weight your life
Hey <in-line LaTeX>! I’m waiting to board another flight today — it’s scheduled to land 40mins before the final Inkhaven deadline, so I don’t want to take the risk of publishing on landing…I’m going to write some breakneck reflections on the open-source / open-weight distinction.
From an international policy report:
The difference between ‘open-weight’ models and ‘open source’ models can be confusing. ‘Open-weight’ means that the model’s weights are available for public download such as with Llama, Mixtral, or Hunyuan-Large. Open-weight models can be, but are not necessarily, open source. The ‘open source’ classification requires that access to the model is protected under an open source licence which grants legal freedom for anyone to use, study, modify, and share the model for any purpose. […] While the open source licence is essential to open source model classification, there remains some disagreement as to the extent to which different components (weights, code, training data) and documentation must be publicly accessible for the model to qualify as open source.
Open-weight source your thinking
How this distinction jives with my enjoinder to “open-source your thinking”
I guess I stole valor from the open source folk. What I should’ve said is “open-weight your thinking” — I gave some subset of the crystallized neural connections / edge-weights, but I didn’t reaaally give the source code / training data for how these came about. A mystery for another day.
Yoyo wrote striking vignettes, and I think I’d like to. This seems like open-source beyond open-weights :)) the emotional motivations and generators.
I gave conclusions, not generators. That’s open-weight, not open-source!
I will open-source at some future point :); I sometimes have in a closed group.
Tiny technical note
I used to be a bit confused about all the ‘tamper-resistant safeguards for open-weight models’ work1. Sure, someone could post-train / fine-tune hazardous knowledge into the models, but conditional on them having that data, they could just as easily put it into in the context window. What kind of actor would train the knowledge in and release the model vs. just sharing / using the knowledge? (Would it be a Trojan horse / backdoor-type thing?
The report actually cleared things up a bit:
Finally, with access to model weights, malicious actors can also fine-tune a model to optimise its performance for harmful applications. Potential malicious uses include harmful dual-use science applications, e.g. using AI to discover new chemical weapons, cyberattacks, and producing harmful fake content such as […] political fake news.
OK, so it’s not just about feeding the model knowledge that it didn’t already have, or raising extant knowledge’s prominence in the salience landscape — it’s mostly about making the model more capable.
It’s about training the model to think natively in the terms you care about, tweaking the weights to complement your scaffolding. This way, someone with W (weights) + DI (dangerous information) can do a lot more than someone who just sticks DI into a context window.
I think I was also pricing in jailbreaks, i.e. that people would be able to hack edgy conversations out of closed-weight models so long as they put DI in the context window. We might explore the marginal power / customizability of open weight models in some future blog posts (p ≈ 0.45).