add overlay documentation to handbook and language spec

Author: ginsbach

docs/codeql/ql-language-reference/annotations.rst

@@ -17,7 +17,7 @@ For example, to declare a module ``M`` as private, you could use:
 
 Note that some annotations act on an entity itself, whilst others act on a particular *name* for the entity:
   - Act on an **entity**: ``abstract``, ``bindingset``, ``cached``, ``extensible``, ``external``, ``language``,
-    ``override``, ``pragma``, and ``transient``
+    ``overlay``, ``override``, ``pragma``, and ``transient``
   - Act on a **name**: ``additional``, ``deprecated``, ``final``, ``library``, ``private``, and ``query``
 
 For example, if you annotate an entity with ``private``, then only that particular name is
@@ -502,6 +502,181 @@ The ``bindingset`` annotation takes a comma-separated list of variables.
   For more information, see ":ref:`predicate-binding`."
 - When you annotate a class, each variable must be ``this`` or a field in the class. 
 
+.. _overlay:
+
+Overlay annotations
+===================
+
+Overlay annotations control how predicates behave during **overlay evaluation**, a feature
+that enables efficient incremental analysis by dividing QL code into *local* and *global*
+parts. During overlay evaluation, local predicates are evaluated separately on "base" (cached 
+from previous analysis) and "overlay" (newly changed files) data. When a global predicate 
+calls a local predicate, results from both the base and overlay evaluations are combined,
+with stale base results filtered out through a process called "discarding."
+
+Overlay evaluation is primarily used internally by GitHub Code Scanning to speed up
+pull request analysis. Most QL developers do not need to use these annotations directly,
+but understanding them can help resolve compilation errors that may occur when overlay
+support is enabled for a language.
+
+.. note::
+
+   Overlay annotations only affect evaluation when overlay compilation is enabled for a
+   QL pack (via ``compileForOverlayEval: true`` in ``qlpack.yml``) and the evaluator is
+   running in overlay mode. Otherwise, these annotations are validated but have no effect
+   on evaluation.
+
+``overlay[local]``
+------------------
+
+**Available for**: |modules|, |classes|, |algebraic datatypes|, |type unions|, |characteristic predicates|, |member predicates|, |non-member predicates|
+
+The ``overlay[local]`` annotation declares that a predicate is local. Local predicates are 
+evaluated separately on base and overlay data and may only depend on other local predicates.
+The compiler reports an error if a local predicate depends on a global predicate.
+
+.. code-block:: ql
+
+    // All dependencies are database extensionals, so this can be local
+    overlay[local]
+    predicate stmtInFile(@stmt s, string path) {
+      exists(@file f, @location loc |
+        hasLocation(s, loc) and
+        locations_default(loc, f, _, _, _, _) and
+        files(f, path)
+      )
+    }
+
+``overlay[local?]``
+-------------------
+
+**Available for**: |modules|, |classes|, |algebraic datatypes|, |type unions|, |characteristic predicates|, |member predicates|, |non-member predicates|
+
+The ``overlay[local?]`` annotation declares that a predicate should be local if all of
+its dependencies are local, and global otherwise. This is particularly useful in
+parameterized modules, where different instantiations may have different locality
+depending on the module parameters.
+
+.. code-block:: ql
+
+    // Locality depends on whether Expr.getType() and Type.getName() are local
+    overlay[local?]
+    predicate exprTypeName(Expr e, string name) {
+      name = e.getType().getName()
+    }
+
+``overlay[global]``
+-------------------
+
+**Available for**: |modules|, |classes|, |algebraic datatypes|, |type unions|, |characteristic predicates|, |member predicates|, |non-member predicates|
+
+The ``overlay[global]`` annotation explicitly declares that a predicate is global. This
+is the default behavior, so this annotation is typically used to override an inherited
+``overlay[local]`` or ``overlay[local?]`` annotation from an enclosing module or class.
+See `Annotation inheritance`_ for an example.
+
+``overlay[caller]``
+-------------------
+
+**Available for**: |modules|, |classes|, |algebraic datatypes|, |type unions|, |characteristic predicates|, |member predicates|, |non-member predicates|
+
+The ``overlay[caller]`` annotation declares that the locality of a predicate depends on
+its caller. The compiler may internally duplicate the predicate, creating separate local
+and global versions. Local callers use the local version; global callers use the global
+version.
+
+.. code-block:: ql
+
+    overlay[caller]
+    predicate utilityPredicate(int x) {
+      x in [1..100]
+    }
+
+``overlay[caller?]``
+--------------------
+
+**Available for**: |modules|, |classes|, |algebraic datatypes|, |type unions|, |characteristic predicates|, |member predicates|, |non-member predicates|
+
+The ``overlay[caller?]`` annotation is like ``overlay[caller]``, but only applies if none
+of the predicate's dependencies are global. If any dependency is global, the predicate
+becomes global regardless of its callers, and calling it from a local predicate will
+result in a compilation error. Like ``overlay[local?]``, this is useful in parameterized
+modules where locality may vary between instantiations.
+
+``overlay[discard_entity]``
+---------------------------
+
+**Available for**: |non-member predicates| (unary predicates on database types only)
+
+The ``overlay[discard_entity]`` annotation designates an *entity discard predicate*.
+These predicates identify database entities that should be filtered out from cached base
+results when combining with overlay results during overlay evaluation.
+
+Entity discard predicates must be:
+
+- Unary predicates (taking exactly one argument)
+- Defined on a database type (a type from the database schema, prefixed with ``@``)
+- Only dependent on local predicates and other non-discarding predicates
+
+.. code-block:: ql
+
+    overlay[discard_entity]
+    private predicate discardExpr(@expr e) {
+      exists(string file | discardableExpr(file, e) and overlayChangedFiles(file))
+    }
+
+    overlay[local]
+    private predicate discardableExpr(string file, @expr e) {
+      not isOverlay() and
+      file = getFile(e)
+    }
+
+    overlay[local]
+    predicate isOverlay() { databaseMetadata("isOverlay", "true") }
+
+Annotation inheritance
+----------------------
+
+Overlay annotations can be applied to modules and types, in which case they are
+inherited by enclosed declarations. Declarations without explicit overlay annotations
+inherit from their innermost enclosing declaration that has an overlay annotation.
+
+.. code-block:: ql
+
+    overlay[local?]
+    module M {
+      predicate foo(@expr x) { ... }  // Inherits overlay[local?]
+
+      class C extends @expr {
+        predicate bar() { ... }  // Inherits overlay[local?]
+
+        overlay[global]
+        predicate baz() { ... }  // Explicitly global
+      }
+    }
+
+Resolving overlay-related errors
+--------------------------------
+
+When overlay support is enabled for a language, you may encounter compilation errors in
+custom QL libraries or queries. Here are common errors and their solutions:
+
+**"Declaration is annotated overlay[local] but depends on global entity"**
+
+A predicate marked ``overlay[local]`` or ``overlay[caller]`` depends on a global predicate.
+Solutions:
+
+- Change the annotation to ``overlay[local?]`` if the predicate doesn't strictly need to be local
+- Add appropriate overlay annotations to the dependency chain to make dependencies local
+- Use the ``forceLocal`` higher-order predicate if you need to call global code from local code (advanced)
+
+**"Cannot apply forceLocal to relation that is annotated overlay[...]"**
+
+The ``forceLocal`` higher-order predicate cannot be applied to predicates that have overlay
+annotations such as ``overlay[local]``, ``overlay[local?]``, ``overlay[caller]``, or 
+``overlay[caller?]``. The input to ``forceLocal`` must be a predicate without such annotations
+(i.e., a global predicate or one with ``overlay[global]``).
+
 .. Links to use in substitutions
 
 .. |classes|                   replace:: :ref:`classes <classes>`

docs/codeql/ql-language-reference/ql-language-specification.rst

@@ -776,6 +776,7 @@ Various kinds of syntax can have *annotations* applied to them. Annotations are
    argsAnnotation ::= "pragma" "[" ("inline" | "inline_late" | "noinline" | "nomagic" | "noopt" | "assume_small_delta") "]"
                   |   "language" "[" "monotonicAggregates" "]"
                   |   "bindingset" "[" (variable ( "," variable)*)? "]"
+                  |   "overlay" "[" ("local" | "local?" | "global" | "caller" | "caller?" | "discard_entity") "]"
 
 Each simple annotation adds a same-named attribute to the syntactic entity it precedes. For example, if a class is preceded by the ``abstract`` annotation, then the class is said to be abstract.
 
@@ -873,6 +874,26 @@ A predicate may have several different binding sets, which can be stated by usin
 
 The ``bindingset`` pragma is usable with type signatures and predicate signatures, but not with module signatures.
 
+The parameterized annotation ``overlay`` controls predicate behavior during overlay evaluation, which is a feature for incremental analysis. Overlay annotations apply at the scope level, and are inherited by nested scopes.
+
++---------------------+---------+------------+-------------------+-----------------------+---------+--------+---------+---------+
+| Overlay             | Classes | Characters | Member predicates | Non-member predicates | Imports | Fields | Modules | Aliases |
++=====================+=========+============+===================+=======================+=========+========+=========+=========+
+| ``local``           | yes     | yes        | yes               | yes                   |         |        | yes     |         |
++---------------------+---------+------------+-------------------+-----------------------+---------+--------+---------+---------+
+| ``local?``          | yes     | yes        | yes               | yes                   |         |        | yes     |         |
++---------------------+---------+------------+-------------------+-----------------------+---------+--------+---------+---------+
+| ``global``          | yes     | yes        | yes               | yes                   |         |        | yes     |         |
++---------------------+---------+------------+-------------------+-----------------------+---------+--------+---------+---------+
+| ``caller``          | yes     | yes        | yes               | yes                   |         |        | yes     |         |
++---------------------+---------+------------+-------------------+-----------------------+---------+--------+---------+---------+
+| ``caller?``         | yes     | yes        | yes               | yes                   |         |        | yes     |         |
++---------------------+---------+------------+-------------------+-----------------------+---------+--------+---------+---------+
+| ``discard_entity``  |         |            |                   | yes                   |         |        |         |         |
++---------------------+---------+------------+-------------------+-----------------------+---------+--------+---------+---------+
+
+The ``overlay[discard_entity]`` annotation is only valid on unary non-member predicates whose argument is a database type.
+
 QLDoc
 -----