|
The repository for Yggdrasil is here.
The Background to Yggdrasil
Between 2011 and 2018, Shen was ported to many platforms; Common Lisp (SBCL, CLisp), Wasp Lisp,
Emacs Lisp, Clojure, Scheme, Python, Javascript, Ruby, Go, C, C++, and the JVM. This was an historical
achievement which has not been matched in any other language community. Shen has established itself as
not just a programming language, but the closest thing that computer science has produced to a universal
high-level medium that transcends the boundaries between programming languages.
Given Shen running under L (where L = Python, Ruby, Common Lisp etc), the next logical step was
to create the means to turn any Shen program into a stand-alone executable program in L. The trick lies
creating a program that extracts from the Shen kernel, just that code that is needed to make the application
run without Shen.
This concept was the topic of a 2013 video appeal by Mark Tarver to sponsor
the implementation of this program (then called the Ring, after Tolkien; but later named Yggdrasil in a backdub).
The main challenge was the coordination of so many ports, some of which had been written by developers who had
left the scene. Sadly we could not raise the money in 2013, so the project languished. LLMs were to change
this picture drastically as we will see.
Yggdrasil
In programming parlance, Yggdrasil is a tree-shaker ; a program that traces the call tree
structure of a user program that depends on a repository, and eliminates all code from the repository that cannot be called
from the user program. Because Shen compiles
into a Lisp, Kλ, it is easy to treat this code as data and traverse the Kλ call tree to
derive the needed code. This is the heart of Yggdrasil .
This explains the name 'Yggdrasil'. Yggdrasil was the world tree which, in Viking mythology, supported the nine
realms of existence. In our picture, the root of Yggdrasil is the language
Kλ which underpins Shen and the branches extend into the other programming worlds. By extracting the needed
Kλ and using the existing compilers to map the Kλ to native code, we create a system in which
programmers can realise their ideas in any L-supported port of Shen (where L = Python, Ruby, Common Lisp etc)
by writing the Shen program and using Yggdrasil to generate the translation into L.
How Yggdrasil is Implemented
There are 14 implementations of Shen in different languages. Originally each implementation was supposed
to reflect the others. For lovers of philosophy, this is the structure of Leibniz's cosmology of monads which although
independent reflect each other by a preprogrammed similarity in form. Geometrically this would be represented by a
tetradecagon where every vertex would be connected to every other vertex.
This architecture would seem to require that each implementation be kept up to date with the latest standard. It is not feasible with a voluntary project to bring all these implementations up to the
latest standard. But expecting programmers to mirror the latest patches throughout 14 languages is not feasible either.
In addition the load of maintaining the 91 connections displayed in the above diagram was beyond the resources of the Shen
community.
However there is a different architecture which helps Yggdrasil to work to the latest standard on all platforms
even though only one of them is fully up to date. This architecture can allow Yggdrasil to generate the latest
standard object code on all platforms. This architecture works by placing the One Chosen Port at the centre of the
tetradecagon and connecting each port to the latest Shen release. By reaching out into the internals of each port using a
connection from the centre, we can systematically harness the work done and maintain the system using only
14 connections and not 91.
What Yggdrasil Means for Computing
This is why Yggdrasil is important. Shen offers a degree of power and security found in few existing languages
and no language has as many
certified ports into other computer languages. A program written in Shen can be verified as type secure and
we can choose into what language to transpile this program. The resulting program
from Yggdrasil inherits all the security and power of the Shen program. This has huge implications for the production
of software in safety critical areas like flight, space research and medicine.
In addition, Shen is acquiring a program Shen2logic which allows declarative Shen programs to
be formally verified by translating them into typed second-order logic. This increases the potential of
Shen to contribute to safety-critical computing which is a multi-billion dollar industry.
Security
The entire edifice of this security depends on two things.
1. The integrity of the Shen kernel
2. The integrity of the program translating Shen into L (called a backend).
As regards the integrity of the kernel, this is only 4,000 lines of code supported by continuous
development over 15 years and backed by math'l proofs in The Book of Shen. Mark Tarver is currently (2026)
embarked on a mission to tighten and extend those logical foundations by publication within the science
community. The backends are supported by the ability of the port to sustain itself; that is, a grossly
defective backend will not run a decent Shen. So compared to the vast repos taken on trust with conventional
apps, Shen's core of trust is very small.
The Rebirth of Yggdrasil under AI
Until 2026, the whole issue of maintenance was an obstacle to Yggdrasil being a reality.
In addition the author (Mark Tarver) was overcommitted. This led to the project page being redacted
because there seemed little practical possibility of carrying it out. Two things changed this situation.
The first was the entrance of Reuben who took over the construction of Yggdrasil. His site is the
Goto for people interested in the project. The second was the appearance of LLMs that were used
to update the old abandoned ports and bring them up to modern standards. This is referred to as the
S system revival and was pioneered independently by Reuben and Mark Tarver.
The One True Port
Out of all the ports available, which one is the best candidate for One True Port? Probably
the fastest and that remains the Chez/Scheme port
by Bruno Defferari. However Bruno has an even
faster port in the offing, but for the moment (September, 2026) we will have to wait for it.
|
