Using Dynamic Scope

The majority of post-ALGOL 60 programming languages are lexically scoped, to the degree that many (most?) contemporary programmers have no hands-on familiarity with dynamic scope. This is a shame, because dynamic scope can provide the same power and flexibility as globally-scoped variables without many of the commonly associated pitfalls. Fortunately, opt-in dynamic scope can be added to any language with support for closures and first-class functions (it can be implemented with classes as well as long as the language supports exporting object instances, but I prefer the functional implementation). This guide will demonstrate various techniques for using dynamic scope with an example implementation in JavaScript.

An Aside About Lisp

One realm where dynamic scope is alive and well is the Land of Lisp. Traditionally, dynamic scope was the default in Lisp dialects (this is still the case in Emacs Lisp). These days, following the example of Scheme, most Lisps are lexically scoped by default but include optional support for dynamic scoping.

Relevant references for various Lisp dialects:

• Chez Scheme
• Racket
• Common Lisp
• Janet
• Emacs Lisp

This guide adopts the API of Chez Scheme and Racket (though our version's syntax won't be quite as nice since JavaScript lacks Lisp's macros).

Basics

We will be introducing dynamic scope through the use of a parameterization construct. The two basic functions in our API will be makeParameter and parameterize. The makeParameter function creates a new parameter with dynamic scope. It optionally accepts an initial value for the parameter. When this value is omitted, the initial value is undefined. A parameter is a function of zero or one argument. When called without an argument, the function returns the parameter's current value. When called with an argument, the function sets the parameter's value to the value of the given argument and returns the updated value.

const color = makeParameter("red");
console.log(color()); // red
color("blue");
console.log(color()); // blue

In lexical scoping, a variable's value can be determined statically by its surrounding context. This is not the case with dynamic scope. For example:

const lexicalColor = "red";
const dynamicColor = makeParameter("blue");

function logColor() {
  // This will always log "red". The value is determined by the lexical
  // context in which the function is defined.
  console.log(lexicalColor);

  // We can't know what this will log at this point. The value is determined
  // by the dynamic context in which the function is called.
  console.log(dynamicColor());
}

// In this context, both statements will log "red".
logColor();

dynamicColor("blue");
// In this context, the first log statement will log "red", but the second
// will log "blue". A similar effect could be achieved if `dynamicColor` were
// a global variable, but note that the parameter variable is block-scoped—
// to the module in this case.
logColor();

With parameters, we get the dynamic power of global scope without the danger of some unrelated code introducing global with the same name. Though dynamicColor provides a dynamically-bound value, the dynamicColor variable is lexically scoped to the module. The binding can be shadowed, but not replaced. Dynamic bindings via parameters can be created in any scope and exposed as part of a module's public interface.

// error.js

import { makeParameter } from "@qxn/parameter";

export const logPrefix = makeParameter("ERROR: ");

export function logError(error) {
  console.error(`${logPrefix()}${error.message}`);
}

// main.js

import { logPrefix, logError } from "./error.js";

const error = new Error("Something blew up!");

logError(error); // Error: Something blew up!

logPrefix("💣 => ");

logError(error); // 💣 => Something blew up!

The examples we've seen so far are still prey to one of the same dangers as global scope. Temporarily changing the value of a dynamic binding and then resetting it to its original value requires careful discipline from the programmer, especially in the face of errors. It's easy to get wrong or to omit unintentionally. Using the code above as an example:

import { logPrefix, logError } from "./error.js";

const error = new Error("Something blew up!");

logError(error); // Error: Something blew up!

// We'd like to temporarily change the log prefix and reset after,
// so we need to do more work.
const initialLogPrefix = logPrefix();
logPrefix("💣 => ");
logError(error); // 💣 => Something blew up!
logPrefix(initialLogPrefix);

// This works in this case, but it's a lot of ceremony.
// It won't work at all if an error occurs before the reset.
logPrefix("💣 => ");
try {
  throw new Error("Something exploded!");
} catch (error) {
  logError(error); // 💣 => Something exploded!
  throw error;
}
// Because we re-threw the error above, this reset is never reached.
// We neglected to add an additional surrounding `try...catch` statement.
logPrefix(initialLogPrefix);

To solve the problems of boilerplate, multiple bindings, and error recovery, we introduce the parameterize function. It creates a new scope during which a set of new parameter bindings are effective and replaces the original values afterwards—even if an error occurs. The parameterize function accepts two arguments, an array of [parameter, value] pairs and a thunk to be parameterized. If you aren't familiar with the term, a thunk is a function that accepts no arguments. In this context, it's a means of achieving lazy evaluation.

import { makeParameter, parameterize } from "parameter";

const configDirectory = makeParameter("/home/will/.config");
const configFile = makeParameter("config.yaml");

function logConfigPath() {
  console.log([configDirectory(), configFile()].join("/"));
}

logConfigPath(); // Logs: /home/will/.config/config.yaml

try {
  parameterize(
    [
      [configDirectory, "test/config"],
      [configFile, "development.yaml"],
    ],
    () => {
      logConfigPath(); // Logs: test/config/development.yaml
      throw new Error("Whoops!");
    }
  );
} catch {
  // ignore error
}

// Parameters are reset even though an error occurred during the
// parameterized function.
logConfigPath(); // Logs: /home/will/.config/config.yaml

Guards

Our makeParameter function has one more trick up its sleeve that we've omitted until this point for simplicity. An optional second argument accepts a guard function that can validate and/or transform any value provided for the parameter. The guard must be a single argument function that accepts a candidate value for the parameter and returns the value if valid. The guard should throw an error if the value is invalid, and may optionally transform the value before returning it.

import { makeParameter } from "@qxn/parameter";

const rgbChannel = makeParameter("red", (value) => {
  const validTypes = ["string", "number", "bigint"];
  if (!validTypes.includes(typeof value)) {
    throw new TypeError("Expected one of: " + validTypes.join(", "));
  }

  if (typeof value === "string") {
    // Ignore case of argument, but standardize on lowercase
    value = value.trim().toLowerCase();

    if (!["red", "green", "blue"].includes(value)) {
      throw new Error("Expected one of: red, green, blue");
    }
  } else {
    const hexToChannel = {
      0xff0000: "red",
      0xff00: "green",
      0xff: "blue",
    };

    value = hexToChannel[Number(value)];
    if (value === undefined) {
      throw new Error(
        "Expected one of: " +
          Object.keys(hexToChannel)
            .map((n) => "0x" + parseInt(n).toString(16).toUpperCase())
            .join(", ")
      );
    }
  }

  return value;
});

rgbChannel("yellow"); // Not red, green, or blue. throws
rgbChannel("blue"); // That's more like it!
rgbChannel("\ngrEEn "); // This is okay.
rgbChannel(0xff0000); // This is also okay.
rgbChannel(100); // This is not okay. throws

A Few Use Cases

Following is a non-comprehensive survey of use cases where dynamic scope can be particularly powerful and often more simple or elegant than the alternatives.

Safer Alternative to Global Variables

A parameter can be used anywhere that a global variable would be used, but with the following advantages:

• Parameter bindings can be module-scoped, and thus cannot be clobbered by other bindings with the same name in unrelated code. Importing a parameter from a module also makes its provenance explicit.
• Parameters can install guards to protect against assignment of invalid values.
• While parameters can be shadowed, doing so will only have an effect within the lexical scope where the shadowing variable occurs as opposed to the entire global scope—limiting the impact of misuse.

Like global variables, dynamic parameters also allow action at a distance—potentially introducing difficult to trace bugs if used carelessly. Great power and flexibility require discipline.

As an example, consider a Node.js io module exposing parameters for the standard in, out, and error streams. A parameterized stdout could be used to temporarily send a subset of an application's logs to a different stream (e.g. a file, TCP socket, or crypto stream).

// io.js ---------------------------------------------------------------

import { makeParameter } from "@qxn/parameter";

export const stdin = makeParameter(process.stdin);
export const stdout = makeParameter(process.stdout);
export const stderr = makeParameter(process.stderr);

export const log = (string) => {
  stdout().write(string + "\n");
};

// main.js -------------------------------------------------------------

const main = () => {
  // Do something useful, and log it.
};

// When our application runs in RECORD mode, stream all logs to a
// timestamped, compressed tmp file rather than standard output.
if (process.env.RUNTIME_MODE === "RECORD") {
  import { compose } from "stream";
  import { createGzip } from "zlib";
  import { createWriteStream } from "fs";
  import { stdout } from "./io.js";

  const compressedLogStream = compose(
    createGzip(),
    createWriteStream(`/tmp/${new Date().toISOString()}.logs.gz`)
  );

  parameterize([[stdout, compressedLogStream]], () => {
    main();
    compressedLogStream.end();
  });
} else {
  main();
}

Examples From the Wild

From Chez Scheme:

• command-line
• command-line-arguments

From Janet:

• *out*
• *executable*

From Common Lisp:

• *gensym-counter*
• *readtable*

Lightweight Dependency Injection

Parameters can host services with API contracts enforced by their guards. This supports a generic, lightweight form of dependency injection wherein consumers can explicitly depend on contracts for services that are satisfied at runtime.

Caveat Emptor: Don't try this with discrete resources in multi-threaded programs! Sage advice from one of Clojure's core developers.

// provider.js ---------------------------------------------------------

import { makeParameter } from "@qxn/parameter";

export const databaseConnection = makeParameter(undefined, (value) => {
  // Verify that value satisfies database connection contract.
});

// consumer.js ---------------------------------------------------------

import { databaseConnection } from "./provider.js";

const db = databaseConnection();

export const getAllUsers = () => {
  return db.query("SELECT * FROM user");
};

// main.js -------------------------------------------------------------

import { parameterize } from "@qxn/parameter";
import { databaseConnection } from "./provider.js";
import { getAllUsers } from "./consumer.js";
import * as sqlite from "sqlite";

const main = async () => {
  console.log("USERS:", await getAllUsers());
};

parameterize([[databaseConnection, sqlite("./test.db")]], () => {
  main();
});

Application Configuration

One of the most useful applications of dynamic scope is application configuration. Guarded parameters allow for robust runtime enforcement of configuration constraints. The example below is somewhat contrived, but one particularly effective real-world use case is configuration for low-level or specialized tuning (e.g. a setting affecting garbage collection settings might be parameterized for a particular code path where it would yield performance benefit).

// configuration.js ----------------------------------------------------

import { makeParameter } from "@qxn/parameter";

export const configuration = {
  database: {
    hostname: makeParameter(process.env.DB_HOST, (value) => {
      // validate hostname
      return value;
    }),
    port: makeParameter(process.env.DB_PORT, (value) => {
      // validate port
      return value;
    }),
    username: makeParameter(process.env.DB_USER, (value) => {
      // validate username
      return value;
    }),
    password: makeParameter(process.env.DB_PASS, (value) => {
      // validate password
      return value;
    }),
  },
};

// main.js -------------------------------------------------------------

import { parameterize } from "@qxn/parameter";
import { configuration } from "./configuration.js";

const main = () => {
  const db = new Database({
    hostname: configuration.database.hostname(),
    port: configuration.database.port(),
    username: configuration.database.username(),
    password: configuration.database.password(),
  });

  // Do something useful with database
};

// Assuming our application is automatically restarted on file change by a
// process manager, this parameterization allows hotswappable configuration
// during development.
if (process.env.ENVIRONMENT === "development") {
  parameterize([[configuration.database.hostname, "localhost"]], () => {
    main();
  });
} else {
  main();
}

Examples From the Wild

From Chez Scheme:

• source-directories
• optimize-level
• debug-level
• undefined-variable-warnings

From Janet:

• err-color
• pretty-format

Runtime Hooks

Dynamic scope allows runtime assignment of handler functions that can provide custom behavior deep within a system via exposed hooks.

Examples From the Wild

From Chez Scheme:

• keyboard-interrupt-handler
• timer-interrupt-handler
• exit-handler

Testing

The combination of the above techniques provides a powerful set of tools for testing: dependency injection makes mocking trivial, flexible application configuration supports comprehensive test coverage for environment-specific code paths, hooks can support simple performance profiling that might otherwise be difficult to capture without instrumenting the code.

The Implementation

We'll begin with a straightforward implementation of the makeParameter and parameterize functions and then expand them to add additional features and more robust handling for edge cases. We'll use JavaScript, but a similar implementation will work in any language that supports closures and first-class functions.

export const makeParameter = (initialValue) => {
  let value = initialValue;

  return (newValue) => {
    // We'll need to fix this later, because we'd like parameters to
    // support any value, including `undefined`.
    if (newValue !== undefined) {
      value = newValue;
    }
    return value;
  };
};

export const parameterize = (parameters, thunk) => {
  // We'll need to validate the provided arguments later.

  // Store the original values.
  const originalValues = new Map();
  for (const [parameter] of parameters) {
    originalValues.set(parameter, parameter());
  }

  let result;
  try {
    // Update the values of any provided parameters.
    for (const [parameter, value] of parameters) {
      parameter(value);
    }
    result = thunk();
  } finally {
    // Restore the original values, even if an error occurs.
    for (const [parameter, value] of originalValues.entries()) {
      parameter(value);
    }
  }

  return result;
};

This naive implementation will work for all the examples so far, but we can do better. For instance, another improvement we can make over global variables is adding a layer of validation to our parameter updates. This will us from bogus values and protect callers from potential bugs. We'll upgrade the makeParameter function to add an optional guard function as a second argument. The guard function should accept one argument (the new candidate value for the parameter), throw an error if the value is invalid, and return the value (or a modified version) otherwise.

export const makeParameter = (initialValue, guard) => {
  let value = guard(initialValue);

  return (newValue) => {
    if (newValue !== undefined) {
      value = guard(newValue);
    }
    return value;
  };
};

Guards can be simple or arbitrarily complex, allowing for robust type checking and coercion.

const rgbChannel = makeParameter('red', (value) => {
    if (typeof value !== 'string') {
        throw new TypeError('Expected a string')
    }

    // Ignore case of argument, but standardize on lowercase
    value = value.toLowerCase()

    if (!['red', 'green', 'blue'].includes(value)) {
        throw new Error('Expected one of: red, green, blue')
    }

    return value
})

rgbChannel(#ff0000)  // Hex representation of red, but not a string. throws
rgbChannel('yellow') // Not red, green, or blue. throws
rgbChannel('blue')   // That's more like it!

Now let's tweak makeParameter to support undefined values in the returned function.

export const makeParameter = (initialValue, guard) => {
  let value = guard(initialValue);

  return (...args) => {
    if (args.length > 0) {
      const [newValue] = args;
      value = guard(newValue);
    }
    return value;
  };
};

const someParam = makeParameter();

// Before
someParam(); // undefined supported for initial value
someParam(5); // value is now 5
someParam(undefined); // value is still 5

// After
someParam(undefined); // value is now `undefined` as expected

Let's add an isParameter predicate function to determine whether a given value is a parameter. We'll need this function later on to validate the arguments of parameterize.

You might be surprised by how tricky this is in JavaScript. Before consulting the following implementation, try your hand at defining a version of isParameter that only returns true for parameters and returns false for all other values. If you opt for duck typing, does your function return a false positive for values that look and behave like parameters? If you decided to use a private prototype and instanceof, is it possible to sniff out your prototype and apply it to some non-parameter object? If you tried a hidden property via a private Symbol, is it possible to access the value of that Symbol? If you're interning all parameters, are you releasing the allocated memory when parameters are garbage collected?

// Create a private WeakSet to maintain references to all parameters created
// via `makeParameter`. Garbage collected parameters will be removed from the
// set.
const parameterSet = new WeakSet();

export const makeParameter = (initialValue, guard) => {
  let value = guard(initialValue);

  const parameter = (...args) => {
    if (args.length > 0) {
      const [newValue] = args;
      value = guard(newValue);
    }
    return value;
  };

  // Add newly minted parameters to the parameter set.
  parameterSet.add(parameter);

  return parameter;
};

// Only values with a reference in the parameter set are parameters.
export const isParameter = (value) => {
  return parameterSet.has(value);
};

// Usage --------------------------------------------------------------------

const param = makeParameter();

isParameter(param); // true

param.length; // 0
typeof param; // 'function'
const fakeParam = () => {};

isParameter(fakeParam); // false

Note: If this is a fun problem for you, reach out for two other ways to solve it—including a method that doesn't use WeakSet or WeakMap.

Now that we have isParameter, we can make parameterize more robust.

// We add a few additional predicate functions to help check for types
const isPair = (value) => {
  return Array.isArray(value) && value.length === 2;
};

const isThunk = (value) => {
  return typeof value === "function" && value.length === 0;
};

export const parameterize = (parameters, thunk) => {
  // We add runtime checks for the given arguments
  if (!(Array.isArray(parameters) && parameters.every(isPair))) {
    throw new TypeError("Expected first argument to be an array of pairs");
  }

  if (!isThunk(thunk)) {
    throw new TypeError("Expected second argument to be a thunk");
  }

  const originalValues = new Map();
  for (const [parameter] of parameters) {
    // We confirm that the first item of each pair is a parameter.
    // The second item can be any type so a type check is unnecessary.
    if (!isParameter(parameter)) {
      throw new TypeError("Expected first item of pair to be a parameter");
    }
    originalValues.set(parameter, parameter());
  }

  let result;
  try {
    for (const [parameter, value] of parameters) {
      parameter(value);
    }
    result = thunk();
  } finally {
    for (const [parameter, value] of originalValues.entries()) {
      parameter(value);
    }
  }

  return result;
};