Provide mechanism for Julia syntax evolution
# Motivation
There are several corner cases in the Julia syntax that are essentially
bugs or mistakes that we'd like to possibly remove, but can't due to
backwards compatibility concerns.
Similarly, when adding new syntax features, there are often cases
that overlap with valid (but often nonsensical) existing syntax.
In the past, we've mostly done judegement calls of these being
"minor changes", but as the package ecosystem grows, so does the
chance of someone accidentally using these anyway and our "minor
changes" have (subjectively) resulted in more breakages recently.
Fortunately, all the recent work on making the parser replacable,
combined with the fact that JuliaSyntax already supports parsing
multiple revisions of Julia syntax provides a solution here:
Just let packages declare what version of the Julia syntax they
are using. That way, packages would not break if we make changes
to the syntax and they can be upgraded at their own pace the next
time the author of that particular package upgrades to a new julia
version.
# Core mechanism
The way this works is simple. Right now, the parser function is always
looked up in `Core._parse`. With this PR, it is instead looked up as
`mod._internal_julia_parse` (slightly longer name to avoid
conflicting with existing bindings of the name in downstream packages),
or `Core._parse` if no such binding exists. Similar for `_lower`.
There is a macro `@Base.Experimental.set_syntax_version v"1.xx"` that
will set the `_internal_julia_parse` (and inte the future the _lower
version) to one that propagates the version to the parser, so users
are not expected to manipulate the binding directly.
# Versioned package loading
The loading system is extended to look at a new `syntax.julia_version`
key in Project.toml (and Manifest for explicit environments).
If no such key exists, it defaults to the minimum allowed version
of the Julia compat. If no compat is defined, it defaults to the current
Julia version. This is technically slightly less backwards compatible than
defaulting this to Julia 1.13, but I think it will be less suprising
in the future for the default syntax to match what is in the REPL.
Most julia packages do already define a julia compat.
Note that as a result of this, the code for parse compat ranges moves
from Pkg to Base.
# Syntax changes
This introduces two parser changes:
1. `@VERSION` (and similar macrocall forms of a macro named `VERSION`)
are now special and trigger the parser to push its version
information into the source location field of the macrocall. Note
that because this is in the parser, this affects all macros with the
name. However, there is also logic on the macrocall side that discards this
again if the macro cannot accept it. This special mechanism is used by
the `Base.Experimental.@VERSION` macro to let users detect the parse
version.
2. The `module` syntax form gains a syntax version argument that is
automatically populated with the parser's current version. This is
the mechanism to propagate syntax information from the parser to the
core mechanism above.
Note that these are only active if a module has opted into 1.14 syntax,
so macros that process `:module` exprs will not see these changes unless
and until the calling module opts into 1.14 syntax via the above
mentioned mechanisms (which is the primary advantage of this scheme).
# Final words
I should emphasize that I'm not proposing using this for any big syntax
revolutions or anything. I would just like to start cleaning up a few
corners of the syntax that I think are universally agreed to be bad but
that we've kept for backwards compatibility. This way, by the time we
get around to making a breaking revision, our entire ecosystem will have
already upgraded to the new syntax.
