tree-sitter-django

A tree-sitter grammar for the Django Template Language (DTL) — the {% %} tags, {{ }} variables and {# #} comments used in Django templates.

Coverage

Every tag and filter Django ships as a builtin is modelled — 28 tags and 57 filters as of Django 6.1 — and so is every tag and filter of the five libraries below. Arguments are part of the grammar rather than something a linter checks later, so a builtin written with the wrong ones is a parse error: {{ x|length:2 }} and {% now %} do not parse, because length takes no argument and now requires one.

The target is Django 6.1, including the builtins Django 6 added ({% partial %}, {% partialdef %}, {% querystring %}, {% csp_nonce_attr %}). Rather than being written from the documentation, the grammar is checked against a real Django — Pipfile.lock pins the version — by enumerating Engine.default_builtins for the tag and filter lists and their arities, and by running candidate snippets through both django.template.Template() and this parser to compare what each accepts. Where the two deliberately disagree is listed under Divergences.

Using it

tree-sitter-django is on npm, and the other language bindings are in place. What is here today:

• src/parser.c and src/scanner.c are generated and committed, so any tree-sitter host can compile them directly without running the CLI. The scanner is required: block-name matching ({% block a %}…{% endblock a %}), the repeated-argument guards and the missing-tag markers live in it.
• tree-sitter-django.wasm is for a web-tree-sitter host or an editor that loads Wasm grammars. It is not checked in: npm run parser-build makes it, and a release ships it both in the npm package (the tree-sitter-django/tree-sitter-django.wasm export) and as an asset on the GitHub release.
• queries/ holds the highlight, locals, tags, injection and error queries an editor loads, plus three files meant for a linter. See Queries.
• tree-sitter.json names the grammar django, scopes it source.django, and points at the query files, which is what a tree-sitter host reads to wire all of the above together.
• bindings/ holds a binding per language, generated by tree-sitter init from the bindings and metadata sections of tree-sitter.json. See Bindings.

Bindings

| Language | Build from a checkout | Entry point | | -------- | ------------------------------------------------------------- | ------------------------------------------------------ | | Node | npm install, which uses a prebuilt binary when there is one | bindings/node/index.js (ESM), typed by index.d.ts | | Rust | cargo build | bindings/rust/lib.rs, Cargo.toml | | Python | pip install . | bindings/python/tree_sitter_django, pyproject.toml | | Go | go test ./bindings/go | bindings/go/binding.go, go.mod | | Swift | swift build | bindings/swift/TreeSitterDjango, Package.swift | | C / C++ | make (static, shared and a pkg-config file) or cmake | bindings/c/tree_sitter/tree-sitter-django.h |

import Parser from "tree-sitter";
import django from "tree-sitter-django";

const parser = new Parser();
parser.setLanguage(django);

To use the parser on a website, the wasm needs to be served then loaded.

import { Language, Parser } from "web-tree-sitter";

await Parser.init();
const wasm = "my-resources-path/tree-sitter-django.wasm";
const parser = new Parser();
parser.setLanguage(await Language.load(wasm));

import tree_sitter, tree_sitter_django

language = tree_sitter.Language(tree_sitter_django.language())
tree = tree_sitter.Parser(language).parse(b"{% if x %}hi {{ y|upper }}{% endif %}")

Development

npm install
npm run parser-generate  # regenerate the parser from grammar.js
npm run parser-test      # run the corpus tests in test/corpus
npm run parser-build     # build the WASM binary
npm run playground       # interactively explore the grammar

src/parser.c, src/grammar.json and src/node-types.json are generated and committed, so a change to grammar.js or src/scanner.c is only half made until parser-generate has run and the result is committed alongside it. tree-sitter test compiles the parser itself, so a stale src/ shows up as tests that pass against the wrong grammar.

Pull requests run the corpus tests and every binding through .github/workflows/ci.yml. A release is a version-bump commit on main: raise the version in package.json and in tree-sitter.json's metadata, run npm run parser-generate, and push. The workflow tags v<version>, attaches tree-sitter-django.wasm to the GitHub release, and stages the npm publish with provenance and prebuilt binaries for Linux, macOS and Windows. Staging is not publishing: npm holds the tarball until a maintainer runs npm stage approve <stage-id> with 2FA, and the workflow's job summary carries that command. Running the workflow by hand stages the current version if npm does not have it yet, which is how a release that failed part way through is retried.

Queries

Five of the files in queries/ are query files an editor loads:

• highlights.scm — syntax highlighting. Tag and filter names are listed one by one, so a name the grammar knows is captured as @function and a name it is only tolerating as @function.call.
• locals.scm — scopes, definitions and references, so an editor can resolve a variable to the tag that bound it. A definition is a variable: field; the scopes are the blocks whose bindings do not outlive them ({% with %}, {% for %}, {% empty %}, {% block %}, {% partialdef %}, {% blocktranslate %}).
• tags.scm — the symbol index behind tree-sitter tag. A {% partialdef %} and the names a {% load %} brings in are definitions; every tag and filter used is a reference.
• injections.scm — marks template text and {% verbatim %} bodies as another language's content, so the HTML around the tags can be parsed by its own grammar. It names no language, leaving the editor to supply one.
• errors.scm — what to report as a problem: (ERROR) as @error.syntax, tree-sitter's own (MISSING) as @error.missing, and @error.missing_tag for the marker placed where a group's required tag is missing. An unterminated {% if a %}x parses as an if_group ending in a zero-width missing_endif_tag rather than as an ERROR, so completion and the scope queries keep working while a template is being written — and the tree reports no error of its own, so this query is how to find it. A marker is placed only where the scanner can tell: at the end of input, before an end tag that belongs to an enclosing group, or before an {% endblock %} naming a block further out. A group's tag written where no open group can hold it, such as {% else %} inside a {% for %}, is an unexpected_tag, captured as @error.unexpected_tag. Words a tag does not take, such as the x in {% else x %}, are an unexpected_argument inside that tag, captured as @error.unexpected_argument. A {{ whose }} is missing closes with a zero-width missing_variable_close, captured as @error.missing_variable_close, so the rest of the template still parses.

The other three are data for a tool rather than queries an editor runs. Each captures the two halves of a relation a tree-sitter query cannot express, and the tool does the join:

• libraries.scm — which library each tag or filter is registered in, and what each {% load %} brings in. The grammar never requires the load, so reporting a missing one is a linter's job; a query has no ordering predicate to compare the positions itself.
• exports.scm — a name a tag binds in the scope around it rather than the scope it opens. Only {% blocktranslate %}'s asvar does that, and a locals query cannot say so: it places a definition in the innermost scope containing it and offers no way out again.
• conditionals.scm — which bodies render on only some passes, as @conditional.group / @conditional.branch / @conditional.default, so a checker can tell that {% if a %}{% now "Y" as n %}{% endif %}{{ n }} may reach {{ n }} with nothing bound. A group is exhaustive exactly when one of its branches is a default. A locals query has no notion of a path not taken.

tree-sitter test loads every file in the directory, so a pattern naming a node that no longer exists fails the suite rather than quietly matching nothing.

Django libraries

Five of Django's template libraries are not builtins: their names have to be {% load %}ed, or preloaded through the engine's OPTIONS: {"builtins": [...]}. Every tag and filter all five register is modelled, with the same argument and arity rules as a builtin:

| library | tags | filters | | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | i18n | trans, translate, blocktrans, blocktranslate, language, get_available_languages, get_current_language, get_current_language_bidi, get_language_info, get_language_info_list | language_bidi, language_name, language_name_local, language_name_translated | | l10n | localize | localize, unlocalize | | static | static, get_static_prefix, get_media_prefix | — | | cache | cache | — | | tz | localtime, timezone, get_current_timezone | localtime, timezone, utc |

The {% load %} is never required, because an engine may have preloaded the library. Whether one is present is a question about the order of nodes rather than about how the template parses, so it is left to a tool: queries/libraries.scm marks each name with the library it came from and each {% load %} with what it brings in, and the tool joins the two. Such a tool needs to know that loading is positional ({% trans "x" %}{% load i18n %} is an error), that it is per file rather than inherited through {% extends %} or {% include %}, and that {% load trans from i18n %} brings in only the names it lists.

Because these names are modelled, they win over the fallbacks below: a project that registers its own tag under one of them with a different signature gets a parse error. They are common enough that shadowing them is treated as the mistake.

If a project insists on shadowing one of these tags, it may use a sub-grammar that replaces the rule with their own. See Adding rules for your own tags and filters.

Custom tags and filters

A library's tags and filters are registered in Python, so the grammar cannot know their names, their arity, or whether a tag opens a body. Two fallback rules accept them:

• Tags. Any tag the grammar does not recognise parses as custom_tag, with the shape simple_tag and inclusion_tag give: positional filter expressions, then name=value keyword arguments, then an optional as name. So {% mytag a b|upper "s" k=1 as out %} parses, with the name on the tag_name: field. A keyword argument written twice is refused.
• Filters. An unknown filter parses as (filter filter_name: (identifier)) with at most one :argument. No arity check for unknown filters is possible.

Both are captured as @function.call rather than @function, so an editor can distinguish a name the grammar knows from one it is only tolerating.

What the fallbacks cannot do:

• A block tag is not nested. {% mytag %}…{% endmytag %} is two sibling custom_tags; pairing them would mean guessing that an unknown tag opens a body.
• A tag registered with @register.tag may parse anything, since it receives the raw token, so one whose arguments are not the simple_tag shape — {% mytag <<>> %} — is an error here and not in Django.
• A misspelled name cannot be caught. {{ x|lenght }} is indistinguishable from a filter some library registered. Cross-referencing names against {% load %} needs the project's Python, so it belongs in a linter; queries/libraries.scm and the load_tag rule supply what one needs.
• Names that only exist inside a tag group are reserved, so {% endif %} or {% empty %} standing on its own is an unexpected_tag, reported by errors.scm, rather than a call to a custom tag of that name.

Adding rules for your own tags and filters

For a project whose custom tags deserve real rules rather than the fallbacks, use this grammar as a base grammar. grammar(base, {...}) merges your rules into it, and a rule that redefines one the base already has receives it as previous. A tag is an alternative of template_tag; a filter is an alternative of filter.

import django from "tree-sitter-django/grammar";

export default grammar(django, {
  name: "djangox",

  conflicts: ($, previous) => [...previous, [$.loop_block]],

  rules: {
    template_tag: ($, previous) => choice(previous, $.map_tag, $.loop_group),

    // {% map items over rows %}
    map_tag: ($) =>
      seq(
        $.simpleTagOpen,
        field("tag_name", "map"),
        $._sep,
        $.filtered_value,
        $._sep,
        "over",
        $._sep,
        $.filtered_value,
        $.simpleTagClose,
      ),

    // {% loop x %}…{% endloop %}
    loop_tag: ($) =>
      seq(
        $.startTagOpen,
        field("tag_name", "loop"),
        $._sep,
        $.identifier,
        $.startTagClose,
      ),
    loop_block: ($) => seq($.loop_tag, optional($.template)),
    endloop_tag: ($) =>
      seq($.endTagOpen, field("tag_name", "endloop"), $.endTagClose),
    loop_group: ($) =>
      seq(
        $.loop_block,
        choice($.endloop_tag, alias($._missing_tag, $.missing_endloop_tag)),
      ),

    // {{ value|shout:"!" }}
    filter: ($, previous) =>
      choice(
        previous,
        seq(field("filter_name", "shout"), ":", field("argument", $.value)),
      ),
  },
});

Seven things to know:

• Whitespace is explicit. extras is empty, because Django splits a tag's contents before parsing any argument, so every separator has to be written out. The part/tag/simpleTag helpers the base grammar is built from are private to it, but the separator is the hidden rule $._sep, and it and the tag utilities (simpleTagOpen and the rest) are rules, so an extending grammar uses them as they are.
• Every tag begins with one of the opens: $.simpleTagOpen, with $.simpleTagClose at its end, for a tag outside any group such as map_tag, and startTagOpen, followTagOpen, repeatTagOpen or endTagOpen for the tags of a group. Its zero-width token hands the tag's name to the scanner and is taken wherever a tag can start, so a tag written from a bare "{%" never matches: {% map items over rows %} would quietly parse as a custom_tag instead.
• A new name wins over the fallback. shout becomes a token of its own, so it commits to your alternative and your arity applies — {{ x|shout }} is now an error, while names you have not modelled still fall through to (filter filter_name: (identifier)).
• A tag with a body is split the way the base grammar's are, so every {% %} is a node of its own: a loop_tag ending in $.startTagClose, a loop_block of that tag and the body, an endloop_tag built from $.endTagOpen and $.endTagClose, and a loop_group of the block and the end tag. A middle tag uses followTagOpen/followTagClose if the group holds it at most once, and repeatTagOpen/repeatTagClose if it may repeat; the scanner trusts that choice. The block needs a conflicts entry, as every block in the base grammar does. The generator offers an associativity instead; taking it silently discards the parse in which the body continues.
• End tags are not reserved, a stray {% endmap %} parses as a custom_tag rather than an error — reserved takes no previous, so the tag_name list cannot be added to a name at a time. Instead you may drop the custom_tag rule to avoid parsing end tags as custom tags. While the fallback is there, stray words in one of your end tags ({% endloop extra %}) read as a custom_tag too, which leaves your group with a missing-tag marker at the end of input instead of an unexpected_argument.
• Your groups get missing-tag markers. The scanner knows no group by name — a group is closed by end followed by its opening tag's name — so the alias($._missing_tag, $.missing_endloop_tag) choice above is all it takes: an unclosed {% loop x %}, or one left open inside an {% if %} that {% endif %} closes, holds a missing_endloop_tag. Add the name to your copy of errors.scm.
• The external scanner has to be re-exported under your grammar's name, or block-name matching and the repeated-argument guards will not link. Give your grammar a src/scanner.c that renames the base's five entry points to the ones your generated parser.c calls, and includes it:

#define tree_sitter_django_external_scanner_create tree_sitter_djangox_external_scanner_create
#define tree_sitter_django_external_scanner_destroy tree_sitter_djangox_external_scanner_destroy
#define tree_sitter_django_external_scanner_scan tree_sitter_djangox_external_scanner_scan
#define tree_sitter_django_external_scanner_serialize tree_sitter_djangox_external_scanner_serialize
#define tree_sitter_django_external_scanner_deserialize tree_sitter_djangox_external_scanner_deserialize

#include "../node_modules/tree-sitter-django/src/scanner.c"

The base's tree_sitter/parser.h then resolves to your own generated copy, and the externals keep the order they have here, so the scanner's token enum stays valid. tree-sitter build takes no include path, hence the relative #include; the package also exports tree-sitter-django/src/scanner.c, so a build step can hand the compiler an absolute path from import.meta.resolve where a package manager's layout makes the relative one unreliable.

Turning the fallbacks off. A grammar that models every tag and filter its project uses probably does not want custom_tag and _custom_filter accepting unrecognized names as they allow typos to parse. previous is the base rule whose members are filterable. We can filter out these rules to make the grammar more strict:

template_tag: ($, previous) =>
  choice(
    ...previous.members.filter((m) => m.name !== "custom_tag"),
    $.map_tag,
  ),

filter: ($, previous) =>
  choice(
    ...previous.members.filter((m) => m.name !== "_custom_filter"),
    seq(field("name", "shout"), ":", field("argument", $.value)),
  ),

{% mytag %} and {{ x|unknown }} are then errors, while the builtins, the libraries and your own rules carry on as before.

Divergences from Django

The grammar aims to accept what Django accepts, with one deliberate exception: where Django takes an input without complaint but the input is almost certainly a mistake. It reads the mistake and silently renders nothing or quietly ignores half of what was written. This grammar tries to report such mistakes as errors instead. Every case below is input Django itself accepts:

• Number-shaped names. Django has no number token: a word is a literal if int() or float() parses it and a variable lookup otherwise, so 0x1f, 1a, 1., 1.2.3 and 1__0 look up a variable of that name and render nothing. (1. really is a lookup there and not a float — Variable rejects a trailing dot after parsing it — while 1.e5 is a float, and both are treated that way here.)
• An attribute of True, False or None. Every Django context is seeded with those three names, so {{ True.x }} is an ordinary two-part lookup that can never resolve.
• A keyword argument written twice. {% with a=1 a=2 %}, and the same in {% include %} and {% blocktranslate %}, where Django builds a dict and lets the last value win.
• Anything after a {% static %} path. StaticNode.handle_token looks for as two words from the end and ignores the rest, so {% static "a" junk %} and {% static "a" as url junk %} render as if the extra words were not there.
• A {% cache %} argument meant as something other than the fragment name. Django takes the raw word in that position, so {% cache 500 using="c" %} names a fragment called using="c" rather than choosing a cache, and {% cache 500 sidebar.name %} names one with a dot in it.

In the other direction the grammar is more permissive only where it cannot know better, which is what the fallbacks above are for.

License

MIT — see LICENSE.