From 69178883ef29d382957133e70c3f23c8710a6997 Mon Sep 17 00:00:00 2001 From: Vineet Naik Date: Thu, 20 Jun 2024 16:23:28 +0530 Subject: [PATCH] Deployed 6c0131b with MkDocs version: 1.6.0 --- 404.html | 42 ++ index.html | 42 ++ rationale/index.html | 42 ++ search/search_index.json | 2 +- sitemap.xml.gz | Bin 127 -> 127 bytes user-guide/commands/index.html | 97 +++ user-guide/docker/index.html | 42 ++ user-guide/getting-started/index.html | 61 +- user-guide/index.html | 42 ++ user-guide/install/index.html | 42 ++ user-guide/layouts.md~ | 20 + user-guide/layouts/index.html | 802 ++++++++++++++++++++ user-guide/manifest/index.html | 213 +++++- user-guide/naming-conventions/index.html | 44 +- user-guide/pg-format/index.html | 42 ++ user-guide/query-tags.md~ | 0 user-guide/query-tags/index.html | 898 +++++++++++++++++++++++ user-guide/query-templates/index.html | 42 ++ user-guide/test-templates/index.html | 44 +- 19 files changed, 2509 insertions(+), 8 deletions(-) create mode 100644 user-guide/layouts.md~ create mode 100644 user-guide/layouts/index.html create mode 100644 user-guide/query-tags.md~ create mode 100644 user-guide/query-tags/index.html diff --git a/404.html b/404.html index 6123622..c5918a0 100644 --- a/404.html +++ b/404.html @@ -452,6 +452,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/index.html b/index.html index 024cb3a..90f322e 100644 --- a/index.html +++ b/index.html @@ -471,6 +471,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/rationale/index.html b/rationale/index.html index 893e566..f8807f1 100644 --- a/rationale/index.html +++ b/rationale/index.html @@ -461,6 +461,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/search/search_index.json b/search/search_index.json index 802a3cd..06013f9 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Tapestry","text":"

    Tapestry is a framework for writing (postgres)SQL queries and (pgTAP) tests using Jinja templates.

    Tapestry is written in Rust but it can be used with applications written in any programming language. It's purely a command line tool that renders Jinja templates into SQL files. How to load the resulting SQL code into memory and use it at runtime is entirely up to the application.

    This approach of loading SQL from files is not new. There are existing libraries such as yesql, hugsql (Clojure), aiosql (Python) etc. that provide excellent abstractions for it. In absence of such a lib for the language of your choice, it shouldn't take more than a few lines of code to implement a simple file loader. In Rust apps, I simply use the include_str! macro.

    One limitation is that tapestry can only be used with PostgreSQL, because of the tight coupling with pgTAP.

    You may find this tool useful if,

    1. you prefer direct SQL queries over ORMs or query builders to interact with RDBMS from application code

    2. you are not averse to the idea of having (reasonable amount of) business logic inside SQL queries

    In fact, if you have had concerns about point 2 i.e. having business logic in SQL queries, perhaps tapestry addresses some of those concerns. Learn more about the rationale behind this tool.

    If you prefer a hands-on introduction, check the Getting started page.

    "},{"location":"rationale/","title":"Rationale","text":""},{"location":"rationale/#problems-with-using-raw-sql-in-application-code","title":"Problems with using raw SQL in application code","text":"

    For many years, I've believed that,

    1. it's a good idea to write raw SQL queries (safely) for interacting with an RDBMS from application code using libs such as yesql, aiosql etc.

    2. it's ok to add reasonable amount of business logic in the SQL queries, rather than using SQL merely for data access.

    Still, I've had concerns about using these ideas in practice, specially in serious projects.

    "},{"location":"rationale/#unit-testing-sql-queries","title":"Unit testing SQL queries","text":"

    Typically, unit tests are written against application code. As more and more business logic gets moved out of the application and into SQL queries, the queries become longer and more complex. In contrast, the application code is reduced to just making db calls using the driver/client library. At this point, it makes more sense to test the queries than the application code.

    Fortunately for PostgreSQL, we have the excellent pgTAP extension that makes it easy to write unit tests for raw queries. Just like the raw queries themselves, pgTAP tests are typically defined in SQL files. But since the query and the tests are in separate files, it's possible that one modifies the SQL query, but forgets to update the tests, and the tests could still pass!

    How to ensure that the tests actually run the exact same query that's being run by the application?

    "},{"location":"rationale/#maintenance-overhead-of-multiple-slightly-differing-queries","title":"Maintenance overhead of multiple, slightly differing queries","text":"

    An application often needs to issue similar queries but returning different set of columns or with different WHERE clauses based on user input. In such cases, a unique query needs to be written and maintained for every combination of the input parameters. This could result in multiple queries that differ only slightly. If some core part of the query needs a change, one needs to remember to update multiple SQL files.

    Moreover, higher level abstractions (e.g. yesql etc.) usually cache queries in memory, so they require the queries to be given a name or an identifier. Since the queries differ only slightly, trying to give them unique names can be tricky.

    "},{"location":"rationale/#how-tapestry-solves-it","title":"How tapestry solves it?","text":"

    Tapestry was built to specifically address the above problems and concerns. It does so by generating actual queries as well as pgTAP test files from Jinja templates, instead of having the user write raw SQL.

    "},{"location":"rationale/#query-templates","title":"Query templates","text":"
    • You write query templates instead of raw queries
    • Multiple queries can be mapped to the same query template. Mapping is defined in the tapestry.toml manifest file.
    • User defined Jinja variables can be used for conditionally adding or omitting parts of the query e.g. a WHERE condition or column to return. These Jinja vars are also defined in the manifest file.
    • Thus, it's easy to generate and maintain multiple queries that are similar enough to be defined using a single query template.
    "},{"location":"rationale/#test-templates","title":"Test templates","text":"
    • pgTAP tests are also written as Jinja templates
    • Test templates are mapped to queries, again in the manifest file. One query can be mapped to multiple test templates.
    • When tapestry renders the final test file from a test template, a special Jinja variable {{ prepared_statement }} gets expanded to the actual query that the test template is mapped to.
    • This way, the generated test SQL file is guaranteed to have the exact same query which is used by the application code.
    "},{"location":"rationale/#naming-conventions","title":"Naming conventions","text":"

    Tapestry suggests some conventions for naming queries consistently but they are not mandatory.

    "},{"location":"user-guide/","title":"Overview","text":"

    Tapestry is built to address the peculiar concerns that I've had about using libraries such as yesql, aiosql and the likes. While I agree with the philosophy behind such libs\u2014that SQL code is better written as SQL directly rather than building it through ORMs, query builders or worse, by string interpolation or concatenation\u2014I've had some concerns about using the approach in practice.

    To understand more about the problems and how tapestry addresses them, please check the Rationale page.

    The general idea behind this tool is, instead of users writing raw SQL queries, have them write Jinja templates from which SQL queries as well as pgTAP tests can be generated.

    Here is a high level overview of how you'd use tapestry in your project:

    1. Create a directory inside your project where the templates will be located. The tapestry init command does this for you.

    2. Add some information in the tapestry.toml manifest file:

      1. Lists of query templates, queries and test templates along with the mappings between them
      2. Location of query templates and test templates (input files)
      3. Location of where the output files are to be created
      4. etc...

    3. Run tapestry render command to generate the SQL files, both for queries as well as tests.

    4. Use a lib such as yesql, aiosql etc. to load the queries rendered by the previous step into the application runtime.

    5. Use pg_prove to run the pgTAP tests, preferably as part of CD/CI. You'd need to implement some kind of automation for this. The github repo also includes a docker image that may help with this.

    "},{"location":"user-guide/commands/","title":"Commands","text":"

    This page documents all commands in the tapestry CLI.

    Note that for all commands except init, this tool will try to read the tapestry.toml manifest file in the current directory and will fail if it's not found. This implies that all tapestry commands except init must be executed from within the \"tapestry project\" root dir.

    "},{"location":"user-guide/commands/#init","title":"init","text":"

    The init command can be used for scaffolding a new tapestry \"project\". It will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. Example:

    Running the following command,

    tapestry init myproj\n

    .. will create the following directory structure

    $ cd myproj\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n|   `-- config\n|-- tapestry.toml\n`-- templates\n    |-- queries\n    `-- tests\n
    "},{"location":"user-guide/commands/#validate","title":"validate","text":"

    The validate command checks and ensures that the manifest file is valid. Additionally it also verifies that the paths referenced in the manifest actually exist and are readable.

    "},{"location":"user-guide/commands/#render","title":"render","text":"

    The render command renders all the template files into SQL files.

    "},{"location":"user-guide/commands/#status","title":"status","text":"

    The status command can be used to preview the effect of running tapestry render command. It will list which output files will be added, modified or remain unchanged if the render command is run. This command will not actually write the output files.

    Output of running tapestry status inside the examples/chinook directory:

    $ tapestry status\nQuery: unchanged: output/queries/artists_long_songs.sql\n  Test: unchanged: output/tests/all_artists_long_songs_count_test.sql\nQuery: unchanged: output/queries/artists_long_songs-limit.sql\nQuery: unchanged: output/queries/artists_long_songs-genre-limit.sql\n  Test: unchanged: output/tests/artists_long_songs-genre-limit_test.sql\nQuery: unchanged: output/queries/songs_formats-artist-album.sql\nQuery: unchanged: output/queries/songs_formats-artist-file_format-album.sql\n  Test: unchanged: output/tests/songs_formats-afa_test.sql\n

    In a way, it's sort of a dry run for the render command.

    "},{"location":"user-guide/commands/#-assert-no-changes","title":"--assert-no-changes","text":"

    A more effective use of this command though is with the --assert-no-changes flag which will cause it to exit with non-zero code if it finds any output files that would get added or modified upon rendering. It's recommended to be run as part of CD/CI, to prevent the user from mistakenly releasing code without rendering the templates.

    "},{"location":"user-guide/commands/#summary","title":"summary","text":"

    The summary command prints a tabular summary of all queries along with their associated (query) templates and tests.

    "},{"location":"user-guide/commands/#coverage","title":"coverage","text":"

    The coverage command prints a list of queries along with the no. of tests (i.e. pgTAP test files) for them. It also prints a coverage score which is calculated as the percentage of queries that have at least 1 test.

    Example: Following is the output of running tapestry coverage inside the examples/chinook dir.

    $ tapestry coverage\n+----------------------------------------+------------------------------------+\n| Query                                  | Has tests?                         |\n+=============================================================================+\n| artists_long_songs                     | Yes (1)                            |\n|----------------------------------------+------------------------------------|\n| artists_long_songs*limit               | No                                 |\n|----------------------------------------+------------------------------------|\n| artists_long_songs@genre*limit         | Yes (1)                            |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist+album             | No                                 |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist&file_format+album | Yes (1)                            |\n|----------------------------------------+------------------------------------|\n| Total                                  | 60.00%                             |\n|                                        | (3/5 queries have at least 1 test) |\n+----------------------------------------+------------------------------------+\n
    "},{"location":"user-guide/commands/#-fail-under","title":"--fail-under","text":"

    By specifying the --fail-under option, the coverage command can be made to exit with non-zero return code if the percentage coverage is below a threshold.

    $ tapestry coverage --fail-under=90 > /dev/null\n$ echo $?\n1\n

    The value of --fail-under option must be an integer between 0 and 100.

    The above command can be run as part of CD/CI to ensure that the test coverage doesn't fall below a certain threshold.

    "},{"location":"user-guide/docker/","title":"Docker","text":""},{"location":"user-guide/docker/#docker-based-workflow-for-running-pgtap-tests","title":"Docker based workflow for running pgTAP tests","text":"

    tapestry only generates SQL files for queries and pgTAP tests. To be able to run the tests you need to install and setup:

    1. PostgreSQL server
    2. pgTAP, which is a postgres extension
    3. pg_prove, which is a command line test runner/harness for pgTAP tests

    While these can be setup manually, the tapestry github repo provides a docker based workflow for easily running tests generated by tapestry against a temporary pg database.

    The relevant files can be found inside the docker directory under project root.

    Note

    I use podman instead of docker for managing containers. Hence all the docker commands in this doc have been actually tested using podman only. As podman claims CLI compatibility with docker, I am assuming that replacing podman with docker in the below mentioned commands should just work. If that's not the case, please create an issue on github.

    "},{"location":"user-guide/docker/#build-the-docker-image","title":"Build the docker image","text":"
    cd docker\npodman build -t tapestry-testbed -f ./Dockerfile\n
    "},{"location":"user-guide/docker/#start-container-for-postgres-process","title":"Start container for postgres process","text":"
    podman run --name taptestbed \\\n    --env POSTGRES_PASSWORD=secret \\\n    -d \\\n    -p 5432:5432 \\\n    tapestry-testbed:latest\n

    Verify that the 5432 port is reachable from the host machine.

    nc -vz localhost 5432\n

    The above podman run command will create a container and start it. After that you can manage the container using the podman container commands

    podman container stop taptestbed\npodman container start taptestbed\n
    "},{"location":"user-guide/docker/#running-tests","title":"Running tests","text":"

    The pg_prove executable is part of the image that we have built. But to be able to run tests inside the container, we need to make the database schema and the test SQL files accessible to it. For this we bind mount a volume into the container when running it, using the --volume option.

    The container image has a bash script run-tests installed into it which picks up the schema and the test SQL files from the mounted dir.

    The run-tests scripts makes certain assumptions about organization of files inside the mounted dir. Inside the container, the dir must be mounted at /tmp/tapestry-data/ and there must be be two sub directories under it:

    1. schema: All SQL files inside this dir will be executed against the database server in lexicographical order to setup a temporary test database.

    2. tests: All SQL files inside this dir will be considered as tests and specified as arguments to the pg_prove command.

    Once such a local directory is created, you can run the tests as follows,

    podman run -it \\\n    --rm \\\n    --network podman \\\n    -v ~/tapestry-data/:/tmp/tapestry-data/ \\\n    --env PGPASSWORD=secret \\\n    --env PGHOST=$(podman container inspect -f '{{.NetworkSettings.IPAddress}}' taptestbed) \\\n    tapestry-testbed:latest \\\n    run-tests -c -d temptestdb\n

    In the above command, temptestdb is the name of the db that will be created by the run-tests script. If your schema files themselves take care of creating the db, then you can specify that as the name and omit the -c flag.

    To know more about the usage of run-tests script, run,

    podman run -it --rm tapestry-testbed:latest run-tests --help\n
    "},{"location":"user-guide/getting-started/","title":"Getting started","text":"

    This tutorial is to help you get started with tapestry. It's assumed that the following software is installed on your system:

    • tapestry
    • pg_format
    • a working installation of PostgreSQL (official docs)
    • pgTAP and pg_prove
    "},{"location":"user-guide/getting-started/#sample-database","title":"Sample database","text":"

    For this tutorial, we'll use the chinook sample database. Download and import it as follows,

    wget -P /tmp/ https://github.com/lerocha/chinook-database/releases/download/v1.4.5/Chinook_PostgreSql_SerialPKs.sql\ncreatedb chinook\npsql -d chinook -f /tmp/Chinook_PostgreSql_SerialPKs.sql\n
    "},{"location":"user-guide/getting-started/#init","title":"Init","text":"

    We'll start by running the tapestry init command, which will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. But for this tutorial, you can run it from any suitable location e.g. the home dir ~/

    cd ~/\ntapestry init chinook\n

    This will create a directory named chinook with following structure,

    $ cd chinook\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n|   `-- config\n|-- tapestry.toml\n`-- templates\n    |-- queries\n    `-- tests\n

    Let's look at the tapestry.toml manifest file that has been created (I've stripped out some comments for conciseness)

    $ cat tapestry.toml\nplaceholder = \"posargs\"\n\nquery_templates_dir = \"templates/queries\"\ntest_templates_dir = \"templates/tests\"\n\nqueries_output_dir = \"output/queries\"\ntests_output_dir = \"output/tests\"\n\n[formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\n\n# [[query_templates]]\n\n# [[queries]]\n\n# [[test_templates]]\n

    placeholder defines the style of generated queries. Default is posargs (positional arguments) which will generate queries with $1, $2 etc as the placeholders. These are suitable for defining prepared statements.

    Then there are four toml keys for defining directories,

    1. query_templates_dir is where the query templates will be located

    2. test_templates_dir is where the test templates will be located

    3. queries_output_dir is where the SQL files for queries will be generated

    4. tests_output_dir is where the SQL files for pgTAP tests will be generated.

    All directory paths are relative to the manifest file.

    You may have noticed that the init command created only the templates dirs. output dirs will be created when tapestry render is called for the first time.

    The init command has also created a pg_format config file for us. This is because it found the pg_format executable on PATH. Refer to the pg_format section for more details.

    "},{"location":"user-guide/getting-started/#adding-a-query_template-to-generate-queries","title":"Adding a query_template to generate queries","text":"

    Now we'll define a query template. But before that, you might want to get yourself familiar with the chinook database's schema.

    Suppose we have an imaginary application built on top of the chinook database in which the following queries need to be run,

    1. list all artists with their longest songs

    2. list top 10 artists having longest songs

    3. list top 5 artists having longest songs, and of a specific genre

    As you can see, we'd need different queries for each of the 3 requirements, but all have a common logic of finding longest songs per artist. Using Jinja syntax, we can write a query template that covers all 3 cases as follows,

    SELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\n{% if cond__genre %}\n    INNER JOIN genre g USING (genre_id)\n  WHERE\n  g.name = {{ placeholder('genre') }}\n{% endif %}\nGROUP BY\n    ar.artist_id\nORDER BY\n-- Descending order because we want the top artists\n    duration DESC\n{% if cond__limit %}\n  LIMIT {{ placeholder('limit') }}\n{% endif %}\n;\n

    We've used some custom Jinja variables for selectively including parts of SQL in the query. These need to be prefixed with cond__ and have to be defined in the manifest file (we'll come to that a bit later).

    We have also used the custom Jinja function placeholder which takes one arg and expands to a placeholder in the actual query. This will be clear once we render the queries.

    Let's save the above query template to the file templates/queries/artists_long_songs.sql.j2.

    And now we'll proceed to defining the query_template and the queries that it can generate in the manifest file. Edit the tapestry.toml file by appending the following lines to it.

    [[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\n

    To define a query_template we need to specify 2 keys:

    1. path i.e. where the template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.

    2. all_conds is a set of values that will be converted to cond__ Jinja variables. In this case it means there are two cond__ Jinja templates supported by the template - cond__genre and cond__limit. Note that they are defined in the manifest without the cond__ suffix.

    We can now define three different queries that map to the same query_template

    [[queries]]\nid = \"artists_long_songs\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = []\n\n[[queries]]\nid = \"artists_long_songs*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"limit\" ]\n\n[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n

    To define a query, we need to specify 3 keys,

    1. id is an identifier for the query. Notice that we're following a naming convention by using special chars @ and *. Read more about Query naming conventions.

    2. template is reference to the query template that we defined earlier.

    3. conds is a subset of the all_conds key that's defined for the linked query template. In the context of this query, only the corresponding cond__ Jinja variables will have the value true, and the rest of them will be false.

    We've defined three queries that use the same template. In the first query, both the conds that the template supports i.e. \"genre\" and \"limit\" are false. In the second query, \"limit\" is true but \"genre\" is false. In the third query, both \"genre\" and \"limit\" are true. Queries will be rendered based on these variables and the {% if cond__.. %} expressions in the template.

    Don't worry if all this doesn't make much sense at this point. Things will be clear when we'll run tapestry render shortly.

    "},{"location":"user-guide/getting-started/#rendering","title":"Rendering","text":"

    Now let's run the tapestry render command.

    tapestry render\n

    And you'll notice some files created in our directory.

    $ tree -a --charset=ascii .\n.\n|-- .pg_format\n|   `-- config\n|-- output\n|   |-- queries\n|   |   |-- artists_long_songs-genre-limit.sql\n|   |   |-- artists_long_songs-limit.sql\n|   |   `-- artists_long_songs.sql\n|   `-- tests\n|-- tapestry.toml\n`-- templates\n    |-- queries\n    |   `-- artists_long_songs.sql.j2\n    `-- tests\n

    Here is what the generated output files look like:

    artists_long_songs.sqlartists_long_songs-limit.sqlartists_long_songs-genre-limit.sql 
    SELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC;\n
    SELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC\nLIMIT $1;\n
    SELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\n    INNER JOIN genre g USING (genre_id)\nWHERE\n    g.name = $1\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC\nLIMIT $2;\n

    As you can see, the output SQL is formatted by pg_format.

    "},{"location":"user-guide/getting-started/#adding-a-test_template","title":"Adding a test_template","text":"

    Now that we've defined and rendered queries, let's add test_template. Again there are two changes required - an entry in the manifest file and the Jinja template itself.

    Add the following lines to the manifest file.

    [[test_templates]]\nquery = \"artists_long_songs@genre*limit\"\npath = \"artists_long_songs-genre-limit_test.sql.j2\"\n

    Here we're referencing the query artists_long_songs@genre*limit hence this test is meant for that query. The path key points to a test template file that we need to create. So let's create the file templates/tests/artists_long_songs-genre-limit_test.sql.j2 with the following contents:

    PREPARE artists_long_songs(varchar, int) AS\n{{ prepared_statement }};\n\nBEGIN;\nSELECT\n    plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 10)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n        (59, 'Santana'::varchar, '00:17:50.027'::interval),\n        (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n        (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n        (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n        (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n        (128, 'Rush'::varchar, '00:11:07.428'::interval),\n        (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n        (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n    $$,\n    'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n    *\nFROM\n    finish ();\nROLLBACK;\n

    The test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. But for this tutorial, it's sufficient to understand that the {{ prepared_statement }} Jinja variable is made available to this template, and when it's rendered it will expand to the actual query.

    Let's run the render command again.

    tapestry render\n

    And now you should see the pgTAP test file created at output/tests/artists_long_songs-genre-limit_test.sql.

    Note

    Here the file stem of the test template path itself was used as the output file name. But it's also possible to explicitly specify it in the manifest file (see output in test_templates docs).

    This is how the rendered test file looks like,

    PREPARE artists_long_songs (varchar, int) AS\nSELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\n    INNER JOIN genre g USING (genre_id)\nWHERE\n    g.name = $1\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC\nLIMIT $2;\n\nBEGIN;\nSELECT\n    plan (1);\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 10)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n        (59, 'Santana'::varchar, '00:17:50.027'::interval),\n        (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n        (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n        (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n        (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n        (128, 'Rush'::varchar, '00:11:07.428'::interval),\n        (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n        (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n    $$,\n    'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\nSELECT\n    *\nFROM\n    finish ();\nROLLBACK;\n
    "},{"location":"user-guide/getting-started/#run-tests","title":"Run tests","text":"

    Assuming that all the above mentioned prerequisites are installed, you can run the tests as follows,

    sudo -u postgres pg_prove -d chinook --verbose output/tests/*.sql\n

    If all goes well, the tests should pass and you should see output similar to,

    1..1\nok 1 - Verify return value\nok\nAll tests successful.\nFiles=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.01 cusr  0.00 csys =  0.05 CPU)\nResult: PASS\n
    "},{"location":"user-guide/getting-started/#thats-all","title":"That's all!","text":"

    If you've reached this far, you should now have a basic understanding of what tapestry is and how to use it. Next, it'd be a good idea to understand the manifest file in more detail.

    Note

    The chinook example discussed in this tutorial can also be found in the github repo under the examples/chinook directory (there are a few more tests included for reference).

    "},{"location":"user-guide/install/","title":"Installation","text":"

    Until tapestry is published to crates.io, you can install it directly from github,

    cargo install --git https://github.com/naiquevin/tapestry.git\n
    "},{"location":"user-guide/install/#additional-dependencies","title":"Additional dependencies","text":"

    Tapestry depends on pg_format for formatting the generated SQL files. It's not a hard requirement but recommended.

    On MacOS, it can be installed using homebrew,

    brew install pgformatter\n

    Note that you need to install pg_format on the machine where you'd be rendering the SQL files using tapestry e.g. on your workstation and/or the build server.

    "},{"location":"user-guide/install/#dependencies-for-running-tests","title":"Dependencies for running tests","text":"

    If you are using tapestry to render tests (which you should, because that's what the tool is meant for!) then you'd also need the pgTAP extension and the pg_prove command line tool.

    pgTAP can be easily built from source. Refer to the instructions here.

    You can install pg_prove from a CPAN distribution as follows:

    sudo cpan TAP::Parser::SourceHandler::pgTAP\n

    Refer to the pgTAP installation guide for more details.

    As tapestry is a postgres specific tool, it goes without saying that you'd need a working installation of postgres to be able to run the tests. Please refer to the official documentation for that.

    "},{"location":"user-guide/manifest/","title":"Manifest","text":"

    Every tapestry \"project\" has a tapestry.toml file which is called the manifest. It is in TOML format and serves the dual purpose of configuration as well as a registry of the following entities:

    1. query_templates
    2. queries
    3. test_templates

    The various sections or top level TOML keys are described in detail below. When going through this doc, you may find it helpful to refer to the chinook example in the github repo. If you haven't checked the Getting started section, it's recommended to read it first.

    "},{"location":"user-guide/manifest/#placeholder","title":"placeholder","text":"

    The placeholder key is for configuring the style of the placeholder syntax for parameters i.e. the values values that are substituted into the statement when it is executed.

    Two options are supported:

    "},{"location":"user-guide/manifest/#posargs","title":"posargs","text":"

    posargs is short for positional arguments. The placeholders refer to the parameters by positions e.g. $1, $2 etc. This is the same syntax that's used for defining prepared statements or SQL functions in postgres.

    This option is suitable when your db driver or SQL library accepts queries in prepared statements syntax. E.g. sqlx (Rust).

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    placeholder = posargs\n
    "},{"location":"user-guide/manifest/#variables","title":"variables","text":"

    When placeholder=variables placeholders are added in the rendered query using the variable substitution syntax of postgres. The variable name in the query is preceded with colon e.g. :email, :department

    This option is suitable when your db driver or SQL library accepts queries with variables. E.g. yesql, hugsql (Clojure), aiosql (Python)

    Examples

    Templateplaceholder = posargsplaceholder = variables 
    SELECT\n    *\nFROM\n    employees\nWHERE\n    email = {{ placeholder('email') }}\n    AND department = {{ placeholder('department') }};\n
    SELECT\n    *\nFROM\n    employees\nWHERE\n    email = $1\n    AND department = $2;\n
    SELECT\n    *\nFROM\n    employees\nWHERE\n    email = :email\n    AND department = :department;\n

    Note

    Note that the prepared_statement Jinja variable available in test templates will always have posargs based placeholders even if the placeholder config in manifest file is set to variables. That's the reason the Jinja var is named prepared_statement.

    "},{"location":"user-guide/manifest/#query_templates_dir","title":"query_templates_dir","text":"

    Path where the query templates are located. The path is always relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    query_templates_dir = \"templates/queries\"\n
    "},{"location":"user-guide/manifest/#test_templates_dir","title":"test_templates_dir","text":"

    Path where the query templates are located. The path is always relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    test_templates_dir = \"templates/tests\"\n
    "},{"location":"user-guide/manifest/#queries_output_dir","title":"queries_output_dir","text":"

    Path to the output dir for the rendered queries. This path also needs to be defined relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    queries_output_dir = \"output/queries\"\n

    A common use case to modify this config would be to store SQL files in a directory outside of the tapestry \"project\" dir, so that only the SQL files in that directory can be packaged into the build artifact. There's no need to include the query/test template and the pgTAP test files in the build artifact. E.g.

    queries_output_dir = \"../sql_queries\"\n
    "},{"location":"user-guide/manifest/#tests_output_dir","title":"tests_output_dir","text":"

    Path to the output dir for rendered pgTAP tests. The path is always relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    tests_output_dir = \"output/tests\"\n
    "},{"location":"user-guide/manifest/#formatterpgformatter","title":"formatter.pgFormatter","text":"

    This section is for configuring the pg_format tool that tapestry uses for formatting the rendered SQL files.

    There two config params under this section:

    "},{"location":"user-guide/manifest/#exec_path","title":"exec_path","text":"

    Location of the pg_format executable.

    "},{"location":"user-guide/manifest/#conf_path","title":"conf_path","text":"

    Path to the pg_format config file. It can be used for configuring the behavior of pg_format when it gets executed on rendered SQL. As with all paths that we've seen so far, this one is also relative to the manifest file.

    Example

    [formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\n

    As mentioned in the installation guide, pg_format is not a mandatory requirement but it's recommended.

    Upon running the tapestry init command, this section will be included in the auto-generated manifest file only if the executable pg_format is found on PATH. In that case, a default pg_format config file will also be created.

    To read more about configuring pg_format in the context of tapestry, refer to the pg_format section of the docs.

    "},{"location":"user-guide/manifest/#query_templates","title":"query_templates","text":"

    query_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.

    For every query template, there are two keys to be defined:

    "},{"location":"user-guide/manifest/#path","title":"path","text":"

    It's where the Jinja template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.

    Use .j2 extension as the convention for the query template file.

    "},{"location":"user-guide/manifest/#all_conds","title":"all_conds","text":"

    It's a set of values that will be converted to cond__ Jinja variables that can be referenced inside the template. Note that they are defined in the manifest without the cond__ suffix.

    For documentation on how to write a query_template, refer to Writing query templates

    Example: 

    [[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\n\n[[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\n
    "},{"location":"user-guide/manifest/#queries","title":"queries","text":"

    queries is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.

    A query can be defined using the following keys,

    "},{"location":"user-guide/manifest/#id","title":"id","text":"

    id is an identifier for the query.

    "},{"location":"user-guide/manifest/#template","title":"template","text":"

    template is a reference to a query_template defined previously in the manifest.

    "},{"location":"user-guide/manifest/#conds","title":"conds","text":"

    conds is a subset of the all_conds key that's defined for the linked query template.

    "},{"location":"user-guide/manifest/#output","title":"output","text":"

    output is the path to the output file where the SQL query will be rendered. It must be relative to the queries_output_dir config.

    It's optional to specify the output. If not specified, the filename of the output file will be derived by slugifying the id. This property allows us to use certain Naming conventions for giving suitable and consistent names to the queries.

    Example:

    [[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n
    "},{"location":"user-guide/manifest/#test_templates","title":"test_templates","text":"

    test_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.

    A test_template can be defined using the following keys,

    "},{"location":"user-guide/manifest/#query","title":"query","text":"

    query is a reference to query defined in the manifest.

    "},{"location":"user-guide/manifest/#path_1","title":"path","text":"

    path is the path to the jinja template for the pgTAP test. It must be relative to the test_templates_dir.

    Use .j2 extension as the convention for the test template file.

    "},{"location":"user-guide/manifest/#output_1","title":"output","text":"

    output is the path where the pgTAP test file will be rendered. It must be relative to the tests_output_dir.

    Specifying output for test_templates is optional. If not specified, it will be derived from the file stem of path i.e. by removing the .j2 extension.

    For detailed documentation on how to write a test_template, refer to Writing test templates

    "},{"location":"user-guide/naming-conventions/","title":"Query naming conventions","text":"

    One of the problems that I have encountered when using SQL loading libraries such as yesql and aiosql is that the queries defined in SQL files need to be given unique names. Often, one ends up writing a group of queries that are mostly similar to each other and differ only slightly. Giving unique and consistent names to each query can become tricky.

    A tool like tapestry cannot automatically give a name to a query. However, since the queries are listed in the manifest file, we can partly address the problem with the use of naming conventions.

    These naming conventions involve clever use of special characters such as @, +, & and *. Let's look at some examples from examples/chinook dir.

    [[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n\n[[queries]]\nid = \"songs_formats@artist&file_format+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\", \"file_format\" ]\n

    In the above queries, the id is defined using an alphanumeric prefix (artists_long_songs and songs_formats) followed by suffix that's an encoding of the conditional Jinja variables relevant to the query.

    The convention is as follows,

    • @ precedes \"cond\" vars used for conditionally including a filter i.e. a WHERE clause.
    • + precedes \"cond\" vars used for conditionally returning a column
    • * precedes \"cond\" vars used for conditionally including any other part of the query e.g. LIMIT, ORDER BY etc.
    • & is used as a delimiter between two \"cond\" vars of same type e.g. @artist&file_format.

    The name of the output file for the SQL query will be generated by slugifying the id i.e. by replacing the above special characters with hyphen (-). In case of the above two queries, the output file names will be artists_long_songs-genre-limit.sql and songs_formats-artist-file_format-album.sql respectively.

    Note that these naming conventions are only recommended by tapestry and are not mandatory.

    "},{"location":"user-guide/pg-format/","title":"Configuring pg_format","text":"

    tapestry relies on pg_format for formatting the rendered SQL files. This makes sure that,

    • the rendered SQL files have consistent indentation
    • you don't need to worry about SQL indentation when writing Jinja templates

    However, pg_format is not a hard requirement for tapestry. If pg_format is not installed on your system at the time of running tapestry init command, the formatter.pgFormatter section will not be added in the auto-generated manifest file.

    The behavior of pg_format tool in the context of tapestry can be configured by adding a config file. The sample config file in the pg_format github repo can be used for reference.

    The default config file generated by tapestry init command is located at .pg_format/config (relative to the manifest file) and looks like this,

    # Lines between markers 'start(noformat)' and 'end(noformat)' will not\n# be formatted. If you want to customize the markers, you may do so by\n# modifying this parameter.\nplaceholder=start\\(noformat\\).+end\\(noformat\\)\n\n# Add a list of function to be formatted as PG internal\n# functions. Paths relative to the 'tapestry.toml' file will also work\n#extra-function=./.pg_format/functions.lst\n\n# Add a list of keywords to be formatted as PG internal keywords.\n# Paths relative to the 'tapestry.toml' file will also work\n#extra-keyword=./.pg_format/keywords.lst\n\n# -- DANGER ZONE --\n#\n# Please donot change the following config parameters. Tapestry may\n# not work otherwise.\nmultiline=1\nformat=text\noutput=\n

    As you can see, the generated file itself is well documented.

    "},{"location":"user-guide/pg-format/#disallowed-configuration","title":"Disallowed configuration","text":"

    In the context of tapestry, some pg_format config params are disallowed (or they need to configured only in a certain way) for proper functioning of tapestry. These are explicitly defined with the intended value in the config file and annotated with DANGER ZONE warning in the comments. These must not be changed.

    "},{"location":"user-guide/pg-format/#selectively-opting-out-of-sql-formatting","title":"Selectively opting out of SQL formatting","text":"

    A commonly faced problem with formatting pgTAP tests using pg_format is that hard coded expected values get formatted in a way that could make the test case unreadable for humans.

    Example: Consider the following pgTAP test case written in a test template file,

    SELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 2)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n    $$,\n    'Verify return value'\n);\n

    By default pg_format would format the above SQL snippet as follows,

    SELECT\n    results_eq ('EXECUTE artists_long_songs(''Rock'', 2)', $$\n    VALUES (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval), (58, 'Deep Purple'::varchar, '00:19:56.094'::interval) $$, 'Verify return value');\n

    To retain the readability, we need to preserve the user's custom indentation. This is where the placeholder config param of pg_format is useful

    Note

    pg_format's placeholder config is not to be confused with placeholder config key in tapestry's manifest.

    This can be done by adding noformat markers before and after the snippet.

    -- start(noformat)\nSELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 2)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n    $$,\n    'Verify return value'\n);\n-- end(noformat)\n

    If you want to customize the markers for whatever reason, you can modify the placeholder param in the pg_format config file.

    "},{"location":"user-guide/query-templates/","title":"Writing query templates","text":"

    Query templates are Jinja template files. One query template can be used for generating multiple SQL queries.

    Often, an application needs to issue mostly similiar (or slightly different) queries to the db based on user input. Some examples:

    • two queries that are exactly similar, except that one returns all columns i.e. * whereas the other returns only selected rows
    • two queries that are exactly the same, except that one has a limit
    • multiple similar queries but different combination of WHERE clauses

    Using Jinja templates, it's possible to write a single query template that can render multiple SQL queries. This is possible with a combination of Jinja variables and {% if .. %}...{% endif %} blocks. This is pretty much the main idea behind query templates.

    "},{"location":"user-guide/query-templates/#cond-variables","title":"\"cond\" variables","text":"

    Query templates need to be defined in the manifest where we specify all_conds which is a set of \"cond\" vars that the template supports.

    Let's look at a query template from the chinook example distributed with the github repo.

    SELECT\n    track.name as title,\n    artist.name as artist_name,\n    {% if cond__album_name %}\n      album.title as album_name,\n    {% endif %}\n    media_type.name as file_format\nFROM\n    album\n    JOIN artist USING (artist_id)\n    LEFT JOIN track USING (album_id)\n    JOIN media_type USING (media_type_id)\n\n{% if cond__artist or cond__file_format %}\n  WHERE\n  {% set num_conds = 0 %}\n  {% if cond__artist %}\n    artist.name = {{ placeholder('artist') }}\n    {% set num_conds = num_conds + 1 %}\n  {% endif %}\n\n  {% if cond__file_format %}\n    {% if num_conds > 0 %}\n      AND\n    {% endif %}\n    media_type.name = {{ placeholder('file_format') }}\n    {% set num_conds = num_conds + 1 %}\n  {% endif %}\n{% endif %}\n;\n

    The entry in the manifest file for the above query_template is,

    [[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\n

    Because of the 3 all_conds defined in the manifest file, we have the following Jinja variables available inside the Jinja template.

    1. cond__artist
    2. cond__file_format
    3. cond__album_name

    The cond__artist and cond__file_format vars are used for conditionally including WHERE clauses. Because we want to add the WHERE clause only if either of the two vars are true, and because we want to add the AND operator only if both are true, nested if blocks are used and a temp \"counter\" variable num_conds is defined i.e. it's assigned to 0 and then incremented by 1 if the cond__artist var is true.

    The third variable cond__album_name is used for conditionally including a column in the returned result.

    "},{"location":"user-guide/query-templates/#query","title":"Query","text":"

    Now let's look at how a query associated with this template is defined in the manifest.

    [[queries]]\nid = \"songs_formats@artist+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\" ]\noutput = \"songs_formats__artist__album.sql\"\n

    In this query, only 2 of the 3 \"cond\" variables will be true.

    As a total of 3 all_conds values are supported by the query template, 8 different queries can be generated from it using different subsets of all_conds.

    [ ]\n[ \"artists\" ]\n[ \"artists\", \"file_format\" ]\n[ \"artists\", \"album_name\" ]\n[ \"file_format\" ]\n[ \"file_format\", \"album_name\" ]\n[ \"album_name\" ]\n[ \"artist\", \"file_format\", \"album_name\" ]\n
    "},{"location":"user-guide/test-templates/","title":"Test templates","text":"

    Just like query_templates, test_templates are also Jinja template files. But while one query template could be used to generate several queries, one test template can be used to generate only one pgTAP test file.

    However, many test templates can be associated with a single query. In other words, if multiple pgTAP test suites are to be written for the same query, that's possible.

    The test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. Important thing to note is that the Jinja variable {{ prepared_statement }} is made available to every test template, and at the time of rendering, it will expand to the actual query.

    Let's look at a templates from the chinook example.

    Refer to the test template songs_formats-afa_test.sql.j2. The first few lines are:

    PREPARE song_formats (varchar, varchar) AS\n{{ prepared_statement }};\n

    Here we're using the prepared_statement Jinja variable to create a prepared statement for the user session. The name of the prepared statement is song_formats and it takes two positional args, both of type varchar.

    Later in the same file, the prepared statement is executed as part of a pgTAP test case,

    SELECT results_eq(\n    'EXECUTE song_formats(''Iron Maiden'', ''Protected AAC audio file'')',\n    $$VALUES\n      ...\n      ...\n    $$,\n    'Verify return value'\n);\n

    Check the songs_formats-afa_test.sql output file to see how the actual test file looks like.

    Note

    Note that the SQL query that prepared_statement Jinja var expands to will always have posargs based placeholders, even if the placeholder config in manifest file is set to variables. That's the reason why the Jinja var is named prepared_statement

    "},{"location":"user-guide/test-templates/#function-instead-of-ps","title":"Function instead of PS","text":"

    Sometimes it's tedious to test for result sets returned by the query. In such cases, it helps to manipulate the result returned by the query and compare a derived property. E.g. If a query results too many rows, it's easier to compare the count than the actual values in the rows.

    One limitation of prepared statements and the EXECUTE syntax for executing them is that it's not sub-query friendly i.e. it's not possible to execute a prepared statement as part of another query.

    The following is NOT valid SQL

    SELECT\n    count(*)\nFROM (EXECUTE song_formats ('Iron Maiden', 'Protected AAC audio file'));\n

    In such cases, we can define a SQL function using the same prepared_statement Jinja variable.

    An example of this can be found in the chinook example - all_artists_long_songs_test.sql.j2

    CREATE OR REPLACE FUNCTION all_artists_long_songs ()\nRETURNS SETOF record\nAS $$\n{{ prepared_statement }}\n$$ LANGUAGE sql;\n\nBEGIN;\nSELECT\n    plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT is(count(*), 204::bigint) from all_artists_long_songs() AS (artist_id int, name text, duration interval);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n    *\nFROM\n    finish ();\nROLLBACK;\n
    "},{"location":"user-guide/test-templates/#test-fixtures","title":"Test fixtures","text":"

    When it comes to automated tests, It's a very common requirement to setup some test data to be able to write test cases. pgTAP tests are not any different. In case of pgTAP one needs to create test data in the database.

    Since pgTAP tests are just SQL files, test data creation can be done using SQL itself in the same file. Reusable setup code can also be extracted into SQL functions that can be created as part of importing the database schema.

    The chinook directory doesn't include an example of this. But here's an example from one of my real projects that uses tapestry.

    In my project, there are two entities categories and items (having tables of the same names) with one-to-many relationship i.e. one category can have multiple items.

    In several pgTAP tests, a few categories and items need to be created. To do this, a function is defined as follows,

    CREATE OR REPLACE FUNCTION tapestry.setup_category_n_items (cat_id varchar, item_idx_start integer, item_idx_end integer)\n    RETURNS void\n    AS $$\n    INSERT INTO categories (id, name)\n        VALUES (cat_id, initcap(replace(cat_id, '-', ' ')));\n    INSERT INTO items (id, name, category_id)\n    SELECT\n        'item-' || t AS id,\n        'Item ' || t AS name,\n        cat_id AS category_id\n    FROM\n        generate_series(item_idx_start, item_idx_end) t;\n$$\nLANGUAGE sql;\n

    And then it's used in pgTAP tests like this, 

    ...\n\nBEGIN;\nSELECT plan(1);\n\n-- Fixtures\n-- create 2 categories, 'cat-a' and 'cat-b' each having 5 items\nSELECT\n    tapestry.setup_category_n_items ('cat-a', 1, 5);\nSELECT\n    tapestry.setup_category_n_items ('cat-b', 6, 10);\n\n-- Test cases\n\n...\n
    "}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Tapestry","text":"

    Tapestry is a framework for writing (postgres)SQL queries and (pgTAP) tests using Jinja templates.

    Tapestry is written in Rust but it can be used with applications written in any programming language. It's purely a command line tool that renders Jinja templates into SQL files. How to load the resulting SQL code into memory and use it at runtime is entirely up to the application.

    This approach of loading SQL from files is not new. There are existing libraries such as yesql, hugsql (Clojure), aiosql (Python) etc. that provide excellent abstractions for it. In absence of such a lib for the language of your choice, it shouldn't take more than a few lines of code to implement a simple file loader. In Rust apps, I simply use the include_str! macro.

    One limitation is that tapestry can only be used with PostgreSQL, because of the tight coupling with pgTAP.

    You may find this tool useful if,

    1. you prefer direct SQL queries over ORMs or query builders to interact with RDBMS from application code

    2. you are not averse to the idea of having (reasonable amount of) business logic inside SQL queries

    In fact, if you have had concerns about point 2 i.e. having business logic in SQL queries, perhaps tapestry addresses some of those concerns. Learn more about the rationale behind this tool.

    If you prefer a hands-on introduction, check the Getting started page.

    "},{"location":"rationale/","title":"Rationale","text":""},{"location":"rationale/#problems-with-using-raw-sql-in-application-code","title":"Problems with using raw SQL in application code","text":"

    For many years, I've believed that,

    1. it's a good idea to write raw SQL queries (safely) for interacting with an RDBMS from application code using libs such as yesql, aiosql etc.

    2. it's ok to add reasonable amount of business logic in the SQL queries, rather than using SQL merely for data access.

    Still, I've had concerns about using these ideas in practice, specially in serious projects.

    "},{"location":"rationale/#unit-testing-sql-queries","title":"Unit testing SQL queries","text":"

    Typically, unit tests are written against application code. As more and more business logic gets moved out of the application and into SQL queries, the queries become longer and more complex. In contrast, the application code is reduced to just making db calls using the driver/client library. At this point, it makes more sense to test the queries than the application code.

    Fortunately for PostgreSQL, we have the excellent pgTAP extension that makes it easy to write unit tests for raw queries. Just like the raw queries themselves, pgTAP tests are typically defined in SQL files. But since the query and the tests are in separate files, it's possible that one modifies the SQL query, but forgets to update the tests, and the tests could still pass!

    How to ensure that the tests actually run the exact same query that's being run by the application?

    "},{"location":"rationale/#maintenance-overhead-of-multiple-slightly-differing-queries","title":"Maintenance overhead of multiple, slightly differing queries","text":"

    An application often needs to issue similar queries but returning different set of columns or with different WHERE clauses based on user input. In such cases, a unique query needs to be written and maintained for every combination of the input parameters. This could result in multiple queries that differ only slightly. If some core part of the query needs a change, one needs to remember to update multiple SQL files.

    Moreover, higher level abstractions (e.g. yesql etc.) usually cache queries in memory, so they require the queries to be given a name or an identifier. Since the queries differ only slightly, trying to give them unique names can be tricky.

    "},{"location":"rationale/#how-tapestry-solves-it","title":"How tapestry solves it?","text":"

    Tapestry was built to specifically address the above problems and concerns. It does so by generating actual queries as well as pgTAP test files from Jinja templates, instead of having the user write raw SQL.

    "},{"location":"rationale/#query-templates","title":"Query templates","text":"
    • You write query templates instead of raw queries
    • Multiple queries can be mapped to the same query template. Mapping is defined in the tapestry.toml manifest file.
    • User defined Jinja variables can be used for conditionally adding or omitting parts of the query e.g. a WHERE condition or column to return. These Jinja vars are also defined in the manifest file.
    • Thus, it's easy to generate and maintain multiple queries that are similar enough to be defined using a single query template.
    "},{"location":"rationale/#test-templates","title":"Test templates","text":"
    • pgTAP tests are also written as Jinja templates
    • Test templates are mapped to queries, again in the manifest file. One query can be mapped to multiple test templates.
    • When tapestry renders the final test file from a test template, a special Jinja variable {{ prepared_statement }} gets expanded to the actual query that the test template is mapped to.
    • This way, the generated test SQL file is guaranteed to have the exact same query which is used by the application code.
    "},{"location":"rationale/#naming-conventions","title":"Naming conventions","text":"

    Tapestry suggests some conventions for naming queries consistently but they are not mandatory.

    "},{"location":"user-guide/","title":"Overview","text":"

    Tapestry is built to address the peculiar concerns that I've had about using libraries such as yesql, aiosql and the likes. While I agree with the philosophy behind such libs\u2014that SQL code is better written as SQL directly rather than building it through ORMs, query builders or worse, by string interpolation or concatenation\u2014I've had some concerns about using the approach in practice.

    To understand more about the problems and how tapestry addresses them, please check the Rationale page.

    The general idea behind this tool is, instead of users writing raw SQL queries, have them write Jinja templates from which SQL queries as well as pgTAP tests can be generated.

    Here is a high level overview of how you'd use tapestry in your project:

    1. Create a directory inside your project where the templates will be located. The tapestry init command does this for you.

    2. Add some information in the tapestry.toml manifest file:

      1. Lists of query templates, queries and test templates along with the mappings between them
      2. Location of query templates and test templates (input files)
      3. Location of where the output files are to be created
      4. etc...

    3. Run tapestry render command to generate the SQL files, both for queries as well as tests.

    4. Use a lib such as yesql, aiosql etc. to load the queries rendered by the previous step into the application runtime.

    5. Use pg_prove to run the pgTAP tests, preferably as part of CD/CI. You'd need to implement some kind of automation for this. The github repo also includes a docker image that may help with this.

    "},{"location":"user-guide/commands/","title":"Commands","text":"

    This page documents all commands in the tapestry CLI.

    Note that for all commands except init, this tool will try to read the tapestry.toml manifest file in the current directory and will fail if it's not found. This implies that all tapestry commands except init must be executed from within the \"tapestry project\" root dir.

    "},{"location":"user-guide/commands/#init","title":"init","text":"

    The init command can be used for scaffolding a new tapestry \"project\". It will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. Example:

    Running the following command,

    tapestry init myproj\n

    .. will create the following directory structure

    $ cd myproj\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n|   `-- config\n|-- tapestry.toml\n`-- templates\n    |-- queries\n    `-- tests\n
    "},{"location":"user-guide/commands/#validate","title":"validate","text":"

    The validate command checks and ensures that the manifest file is valid. Additionally it also verifies that the paths referenced in the manifest actually exist and are readable.

    "},{"location":"user-guide/commands/#render","title":"render","text":"

    The render command renders all the template files into SQL files.

    "},{"location":"user-guide/commands/#status","title":"status","text":"

    The status command can be used to preview the effect of running tapestry render command. It will list which output files will be added, modified or remain unchanged if the render command is run. This command will not actually write the output files.

    Output of running tapestry status inside the examples/chinook directory:

    $ tapestry status\nQuery: unchanged: output/queries/artists_long_songs.sql\n  Test: unchanged: output/tests/all_artists_long_songs_count_test.sql\nQuery: unchanged: output/queries/artists_long_songs-limit.sql\nQuery: unchanged: output/queries/artists_long_songs-genre-limit.sql\n  Test: unchanged: output/tests/artists_long_songs-genre-limit_test.sql\nQuery: unchanged: output/queries/songs_formats-artist-album.sql\nQuery: unchanged: output/queries/songs_formats-artist-file_format-album.sql\n  Test: unchanged: output/tests/songs_formats-afa_test.sql\n

    In a way, it's sort of a dry run for the render command.

    "},{"location":"user-guide/commands/#-assert-no-changes","title":"--assert-no-changes","text":"

    A more effective use of this command though is with the --assert-no-changes flag which will cause it to exit with non-zero code if it finds any output files that would get added or modified upon rendering. It's recommended to be run as part of CD/CI, to prevent the user from mistakenly releasing code without rendering the templates.

    "},{"location":"user-guide/commands/#summary","title":"summary","text":"

    The summary command prints a tabular summary of all queries along with their associated (query) templates and tests.

    "},{"location":"user-guide/commands/#-all","title":"--all","text":"

    When --all option is specified with this command, the summary will include query and test files inside queries_output_dir and tests_output_dir respectively that are not added to the manifest.

    Note

    There are legit use cases for having files in the query and test output directories that are not added to the manifest Examples:

    1. queries that don't need any tests but need to be stored in the same directory as other queries, so that yesql, aiosql libs can load all of them together.

    2. Existing queries which are not yet migrated to tapestry (gradual migration strategy).

    3. pgTAP tests written for stored procedures, views, schema etc. that need to be stored in the same directory as other tests, so that all tests can be run together.

    "},{"location":"user-guide/commands/#coverage","title":"coverage","text":"

    The coverage command prints a list of queries along with the no. of tests (i.e. pgTAP test files) for them. It also prints a coverage score which is calculated as the percentage of queries that have at least 1 test.

    Example: Following is the output of running tapestry coverage inside the examples/chinook dir.

    $ tapestry coverage\n+----------------------------------------+------------------------------------+\n| Query                                  | Has tests?                         |\n+=============================================================================+\n| artists_long_songs                     | Yes (1)                            |\n|----------------------------------------+------------------------------------|\n| artists_long_songs*limit               | No                                 |\n|----------------------------------------+------------------------------------|\n| artists_long_songs@genre*limit         | Yes (1)                            |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist+album             | No                                 |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist&file_format+album | Yes (1)                            |\n|----------------------------------------+------------------------------------|\n| Total                                  | 60.00%                             |\n|                                        | (3/5 queries have at least 1 test) |\n+----------------------------------------+------------------------------------+\n
    "},{"location":"user-guide/commands/#-fail-under","title":"--fail-under","text":"

    By specifying the --fail-under option, the coverage command can be made to exit with non-zero return code if the percentage coverage is below a threshold.

    $ tapestry coverage --fail-under=90 > /dev/null\n$ echo $?\n1\n

    The value of --fail-under option must be an integer between 0 and 100.

    The above command can be run as part of CD/CI to ensure that the test coverage doesn't fall below a certain threshold.

    "},{"location":"user-guide/docker/","title":"Docker","text":""},{"location":"user-guide/docker/#docker-based-workflow-for-running-pgtap-tests","title":"Docker based workflow for running pgTAP tests","text":"

    tapestry only generates SQL files for queries and pgTAP tests. To be able to run the tests you need to install and setup:

    1. PostgreSQL server
    2. pgTAP, which is a postgres extension
    3. pg_prove, which is a command line test runner/harness for pgTAP tests

    While these can be setup manually, the tapestry github repo provides a docker based workflow for easily running tests generated by tapestry against a temporary pg database.

    The relevant files can be found inside the docker directory under project root.

    Note

    I use podman instead of docker for managing containers. Hence all the docker commands in this doc have been actually tested using podman only. As podman claims CLI compatibility with docker, I am assuming that replacing podman with docker in the below mentioned commands should just work. If that's not the case, please create an issue on github.

    "},{"location":"user-guide/docker/#build-the-docker-image","title":"Build the docker image","text":"
    cd docker\npodman build -t tapestry-testbed -f ./Dockerfile\n
    "},{"location":"user-guide/docker/#start-container-for-postgres-process","title":"Start container for postgres process","text":"
    podman run --name taptestbed \\\n    --env POSTGRES_PASSWORD=secret \\\n    -d \\\n    -p 5432:5432 \\\n    tapestry-testbed:latest\n

    Verify that the 5432 port is reachable from the host machine.

    nc -vz localhost 5432\n

    The above podman run command will create a container and start it. After that you can manage the container using the podman container commands

    podman container stop taptestbed\npodman container start taptestbed\n
    "},{"location":"user-guide/docker/#running-tests","title":"Running tests","text":"

    The pg_prove executable is part of the image that we have built. But to be able to run tests inside the container, we need to make the database schema and the test SQL files accessible to it. For this we bind mount a volume into the container when running it, using the --volume option.

    The container image has a bash script run-tests installed into it which picks up the schema and the test SQL files from the mounted dir.

    The run-tests scripts makes certain assumptions about organization of files inside the mounted dir. Inside the container, the dir must be mounted at /tmp/tapestry-data/ and there must be be two sub directories under it:

    1. schema: All SQL files inside this dir will be executed against the database server in lexicographical order to setup a temporary test database.

    2. tests: All SQL files inside this dir will be considered as tests and specified as arguments to the pg_prove command.

    Once such a local directory is created, you can run the tests as follows,

    podman run -it \\\n    --rm \\\n    --network podman \\\n    -v ~/tapestry-data/:/tmp/tapestry-data/ \\\n    --env PGPASSWORD=secret \\\n    --env PGHOST=$(podman container inspect -f '{{.NetworkSettings.IPAddress}}' taptestbed) \\\n    tapestry-testbed:latest \\\n    run-tests -c -d temptestdb\n

    In the above command, temptestdb is the name of the db that will be created by the run-tests script. If your schema files themselves take care of creating the db, then you can specify that as the name and omit the -c flag.

    To know more about the usage of run-tests script, run,

    podman run -it --rm tapestry-testbed:latest run-tests --help\n
    "},{"location":"user-guide/getting-started/","title":"Getting started","text":"

    This tutorial is to help you get started with tapestry. It's assumed that the following software is installed on your system:

    • tapestry
    • pg_format
    • a working installation of PostgreSQL (official docs)
    • pgTAP and pg_prove
    "},{"location":"user-guide/getting-started/#sample-database","title":"Sample database","text":"

    For this tutorial, we'll use the chinook sample database. Download and import it as follows,

    wget -P /tmp/ https://github.com/lerocha/chinook-database/releases/download/v1.4.5/Chinook_PostgreSql_SerialPKs.sql\ncreatedb chinook\npsql -d chinook -f /tmp/Chinook_PostgreSql_SerialPKs.sql\n
    "},{"location":"user-guide/getting-started/#init","title":"Init","text":"

    We'll start by running the tapestry init command, which will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. But for this tutorial, you can run it from any suitable location e.g. the home dir ~/

    cd ~/\ntapestry init chinook\n

    This will create a directory named chinook with following structure,

    $ cd chinook\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n|   `-- config\n|-- tapestry.toml\n`-- templates\n    |-- queries\n    `-- tests\n

    Let's look at the tapestry.toml manifest file that has been created (I've stripped out some comments for conciseness)

    $ cat tapestry.toml\nplaceholder = \"posargs\"\n\nquery_templates_dir = \"templates/queries\"\ntest_templates_dir = \"templates/tests\"\n\nqueries_output_dir = \"output/queries\"\ntests_output_dir = \"output/tests\"\n\n[formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\n\n[name_tagger]\nstyle = \"kebab-case\"\n\n# [[query_templates]]\n\n# [[queries]]\n\n# [[test_templates]]\n

    placeholder defines the style of generated queries. Default is posargs (positional arguments) which will generate queries with $1, $2 etc as the placeholders. These are suitable for defining prepared statements.

    Then there are four toml keys for defining directories,

    1. query_templates_dir is where the query templates will be located

    2. test_templates_dir is where the test templates will be located

    3. queries_output_dir is where the SQL files for queries will be generated

    4. tests_output_dir is where the SQL files for pgTAP tests will be generated.

    All directory paths are relative to the manifest file.

    You may have noticed that the init command created only the templates dirs. output dirs will be created when tapestry render is called for the first time.

    The init command has also created a pg_format config file for us. This is because it found the pg_format executable on PATH. Refer to the pg_format section for more details.

    Finally, name_tagger has been configured with kebab-case as the style.

    "},{"location":"user-guide/getting-started/#adding-a-query_template-to-generate-queries","title":"Adding a query_template to generate queries","text":"

    Now we'll define a query template. But before that, you might want to get yourself familiar with the chinook database's schema.

    Suppose we have an imaginary application built on top of the chinook database in which the following queries need to be run,

    1. list all artists with their longest songs

    2. list top 10 artists having longest songs

    3. list top 5 artists having longest songs, and of a specific genre

    As you can see, we'd need different queries for each of the 3 requirements, but all have a common logic of finding longest songs per artist. Using Jinja syntax, we can write a query template that covers all 3 cases as follows,

    SELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\n{% if cond__genre %}\n    INNER JOIN genre g USING (genre_id)\n  WHERE\n  g.name = {{ placeholder('genre') }}\n{% endif %}\nGROUP BY\n    ar.artist_id\nORDER BY\n-- Descending order because we want the top artists\n    duration DESC\n{% if cond__limit %}\n  LIMIT {{ placeholder('limit') }}\n{% endif %}\n;\n

    We've used some custom Jinja variables for selectively including parts of SQL in the query. These need to be prefixed with cond__ and have to be defined in the manifest file (we'll come to that a bit later).

    We have also used the custom Jinja function placeholder which takes one arg and expands to a placeholder in the actual query. This will be clear once we render the queries.

    Let's save the above query template to the file templates/queries/artists_long_songs.sql.j2.

    And now we'll proceed to defining the query_template and the queries that it can generate in the manifest file. Edit the tapestry.toml file by appending the following lines to it.

    [[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\n

    To define a query_template we need to specify 2 keys:

    1. path i.e. where the template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.

    2. all_conds is a set of values that will be converted to cond__ Jinja variables. In this case it means there are two cond__ Jinja templates supported by the template - cond__genre and cond__limit. Note that they are defined in the manifest without the cond__ suffix.

    We can now define three different queries that map to the same query_template

    [[queries]]\nid = \"artists_long_songs\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = []\n\n[[queries]]\nid = \"artists_long_songs*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"limit\" ]\n\n[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n

    To define a query, we need to specify 3 keys,

    1. id is an identifier for the query. Notice that we're following a naming convention by using special chars @ and *. Read more about Query naming conventions.

    2. template is reference to the query template that we defined earlier.

    3. conds is a subset of the all_conds key that's defined for the linked query template. In the context of this query, only the corresponding cond__ Jinja variables will have the value true, and the rest of them will be false.

    We've defined three queries that use the same template. In the first query, both the conds that the template supports i.e. \"genre\" and \"limit\" are false. In the second query, \"limit\" is true but \"genre\" is false. In the third query, both \"genre\" and \"limit\" are true. Queries will be rendered based on these variables and the {% if cond__.. %} expressions in the template.

    Don't worry if all this doesn't make much sense at this point. Things will be clear when we'll run tapestry render shortly.

    "},{"location":"user-guide/getting-started/#rendering","title":"Rendering","text":"

    Now let's run the tapestry render command.

    tapestry render\n

    And you'll notice some files created in our directory.

    $ tree -a --charset=ascii .\n.\n|-- .pg_format\n|   `-- config\n|-- output\n|   |-- queries\n|   |   |-- artists_long_songs-genre-limit.sql\n|   |   |-- artists_long_songs-limit.sql\n|   |   `-- artists_long_songs.sql\n|   `-- tests\n|-- tapestry.toml\n`-- templates\n    |-- queries\n    |   `-- artists_long_songs.sql.j2\n    `-- tests\n

    Here is what the generated output files look like:

    artists_long_songs.sqlartists_long_songs-limit.sqlartists_long_songs-genre-limit.sql 
    -- name: artists-long-songs\nSELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC;\n
    -- name: artists-long-songs-limit\nSELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC\nLIMIT $1;\n
    -- name: artists-long-songs-genre-limit\nSELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\n    INNER JOIN genre g USING (genre_id)\nWHERE\n    g.name = $1\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC\nLIMIT $2;\n

    The SQL comments before the SQL with name of the query are generated by name_tagger added to the manifest. Learn more about Name tagging.

    Also notice that the output SQL is formatted by pg_format.

    "},{"location":"user-guide/getting-started/#adding-a-test_template","title":"Adding a test_template","text":"

    Now that we've defined and rendered queries, let's add test_template. Again there are two changes required - an entry in the manifest file and the Jinja template itself.

    Add the following lines to the manifest file.

    [[test_templates]]\nquery = \"artists_long_songs@genre*limit\"\npath = \"artists_long_songs-genre-limit_test.sql.j2\"\n

    Here we're referencing the query artists_long_songs@genre*limit hence this test is meant for that query. The path key points to a test template file that we need to create. So let's create the file templates/tests/artists_long_songs-genre-limit_test.sql.j2 with the following contents:

    PREPARE artists_long_songs(varchar, int) AS\n{{ prepared_statement }};\n\nBEGIN;\nSELECT\n    plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 10)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n        (59, 'Santana'::varchar, '00:17:50.027'::interval),\n        (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n        (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n        (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n        (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n        (128, 'Rush'::varchar, '00:11:07.428'::interval),\n        (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n        (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n    $$,\n    'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n    *\nFROM\n    finish ();\nROLLBACK;\n

    The test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. But for this tutorial, it's sufficient to understand that the {{ prepared_statement }} Jinja variable is made available to this template, and when it's rendered it will expand to the actual query.

    Let's run the render command again.

    tapestry render\n

    And now you should see the pgTAP test file created at output/tests/artists_long_songs-genre-limit_test.sql.

    Note

    Here the file stem of the test template path itself was used as the output file name. But it's also possible to explicitly specify it in the manifest file (see output in test_templates docs).

    This is how the rendered test file looks like,

    PREPARE artists_long_songs (varchar, int) AS\nSELECT\n    ar.artist_id,\n    ar.name,\n    max(milliseconds) * interval '1 ms' AS duration\nFROM\n    track t\n    INNER JOIN album al USING (album_id)\n    INNER JOIN artist ar USING (artist_id)\n    INNER JOIN genre g USING (genre_id)\nWHERE\n    g.name = $1\nGROUP BY\n    ar.artist_id\nORDER BY\n    -- Descending order because we want the top artists\n    duration DESC\nLIMIT $2;\n\nBEGIN;\nSELECT\n    plan (1);\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 10)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n        (59, 'Santana'::varchar, '00:17:50.027'::interval),\n        (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n        (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n        (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n        (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n        (128, 'Rush'::varchar, '00:11:07.428'::interval),\n        (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n        (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n    $$,\n    'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\nSELECT\n    *\nFROM\n    finish ();\nROLLBACK;\n
    "},{"location":"user-guide/getting-started/#run-tests","title":"Run tests","text":"

    Assuming that all the above mentioned prerequisites are installed, you can run the tests as follows,

    sudo -u postgres pg_prove -d chinook --verbose output/tests/*.sql\n

    If all goes well, the tests should pass and you should see output similar to,

    1..1\nok 1 - Verify return value\nok\nAll tests successful.\nFiles=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.01 cusr  0.00 csys =  0.05 CPU)\nResult: PASS\n
    "},{"location":"user-guide/getting-started/#thats-all","title":"That's all!","text":"

    If you've reached this far, you should now have a basic understanding of what tapestry is and how to use it. Next, it'd be a good idea to understand the manifest file in more detail.

    Note

    The chinook example discussed in this tutorial can also be found in the github repo under the examples/chinook directory (there are a few more tests included for reference).

    "},{"location":"user-guide/install/","title":"Installation","text":"

    Until tapestry is published to crates.io, you can install it directly from github,

    cargo install --git https://github.com/naiquevin/tapestry.git\n
    "},{"location":"user-guide/install/#additional-dependencies","title":"Additional dependencies","text":"

    Tapestry depends on pg_format for formatting the generated SQL files. It's not a hard requirement but recommended.

    On MacOS, it can be installed using homebrew,

    brew install pgformatter\n

    Note that you need to install pg_format on the machine where you'd be rendering the SQL files using tapestry e.g. on your workstation and/or the build server.

    "},{"location":"user-guide/install/#dependencies-for-running-tests","title":"Dependencies for running tests","text":"

    If you are using tapestry to render tests (which you should, because that's what the tool is meant for!) then you'd also need the pgTAP extension and the pg_prove command line tool.

    pgTAP can be easily built from source. Refer to the instructions here.

    You can install pg_prove from a CPAN distribution as follows:

    sudo cpan TAP::Parser::SourceHandler::pgTAP\n

    Refer to the pgTAP installation guide for more details.

    As tapestry is a postgres specific tool, it goes without saying that you'd need a working installation of postgres to be able to run the tests. Please refer to the official documentation for that.

    "},{"location":"user-guide/layouts/","title":"Layouts","text":"

    Tapestry lets you control the layout of the query files i.e. how the generated SQL is organized in files. It supports two ways at present:

    1. one-file-one-query: Each SQL query will be written to a separate file
    2. one-file-all-queries: All SQL queries will be written to a single file

    To configure this, you need to specify the query_output_layout key in the manifest. The default option if not specified is one-file-one-query.

    "},{"location":"user-guide/layouts/#layout-and-queriesoutput-field","title":"Layout and queries[].output field","text":"

    Users may specify output field for every query, which is the path where the generated SQL output will be written. If output is not specified, it's value is derived from the query id. This works well for the one-file-one-query layout.

    When the layout is one-file-all-queries, it's expected that the output field of all queries must be the same. Otherwise the manifest fails to validate. To avoid duplication, a related setting query_output_file is provided.

    If layout = one-file-all-queries, it's recommended to set query_output_file and omit the output field for individual queries.

    If layout = one-file-one-query, then you must not set query_output_file. Whether or not to set the output field for individual queries is up to you.

    "},{"location":"user-guide/manifest/","title":"Manifest","text":"

    Every tapestry \"project\" has a tapestry.toml file which is called the manifest. It is in TOML format and serves the dual purpose of configuration as well as a registry of the following entities:

    1. query_templates
    2. queries
    3. test_templates

    The various sections or top level TOML keys are described in detail below. When going through this doc, you may find it helpful to refer to the chinook example in the github repo. If you haven't checked the Getting started section, it's recommended to read it first.

    "},{"location":"user-guide/manifest/#placeholder","title":"placeholder","text":"

    The placeholder key is for configuring the style of the placeholder syntax for parameters i.e. the values values that are substituted into the statement when it is executed.

    Two options are supported:

    "},{"location":"user-guide/manifest/#posargs","title":"posargs","text":"

    posargs is short for positional arguments. The placeholders refer to the parameters by positions e.g. $1, $2 etc. This is the same syntax that's used for defining prepared statements or SQL functions in postgres.

    This option is suitable when your db driver or SQL library accepts queries in prepared statements syntax. E.g. sqlx (Rust).

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    placeholder = posargs\n
    "},{"location":"user-guide/manifest/#variables","title":"variables","text":"

    When placeholder=variables placeholders are added in the rendered query using the variable substitution syntax of postgres. The variable name in the query is preceded with colon e.g. :email, :department

    This option is suitable when your db driver or SQL library accepts queries with variables. E.g. yesql, hugsql (Clojure), aiosql (Python)

    Examples

    Templateplaceholder = posargsplaceholder = variables 
    SELECT\n    *\nFROM\n    employees\nWHERE\n    email = {{ placeholder('email') }}\n    AND department = {{ placeholder('department') }};\n
    SELECT\n    *\nFROM\n    employees\nWHERE\n    email = $1\n    AND department = $2;\n
    SELECT\n    *\nFROM\n    employees\nWHERE\n    email = :email\n    AND department = :department;\n

    Note

    Note that the prepared_statement Jinja variable available in test templates will always have posargs based placeholders even if the placeholder config in manifest file is set to variables. That's the reason the Jinja var is named prepared_statement.

    "},{"location":"user-guide/manifest/#query_templates_dir","title":"query_templates_dir","text":"

    Path where the query templates are located. The path is always relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    query_templates_dir = \"templates/queries\"\n
    "},{"location":"user-guide/manifest/#test_templates_dir","title":"test_templates_dir","text":"

    Path where the query templates are located. The path is always relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    test_templates_dir = \"templates/tests\"\n
    "},{"location":"user-guide/manifest/#queries_output_dir","title":"queries_output_dir","text":"

    Path to the output dir for the rendered queries. This path also needs to be defined relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    queries_output_dir = \"output/queries\"\n

    A common use case to modify this config would be to store SQL files in a directory outside of the tapestry \"project\" dir, so that only the SQL files in that directory can be packaged into the build artifact. There's no need to include the query/test template and the pgTAP test files in the build artifact. E.g.

    queries_output_dir = \"../sql_queries\"\n
    "},{"location":"user-guide/manifest/#tests_output_dir","title":"tests_output_dir","text":"

    Path to the output dir for rendered pgTAP tests. The path is always relative to the manifest file.

    Default: The manifest file auto-generated upon running the tapestry init command will have,

    tests_output_dir = \"output/tests\"\n
    "},{"location":"user-guide/manifest/#query_output_layout","title":"query_output_layout","text":"

    Layout to be used for the generated query files. The two options are:

    1. one-file-one-query: Each SQL query will be written to a separate file

    2. one-file-all-queries: All SQL queries will be written to a single file

    It's optional. The default value is one-file-one-query.

    "},{"location":"user-guide/manifest/#query_output_file","title":"query_output_file","text":"

    query_output_file is optional but it's use is valid only when the layout is one-file-all-queries. It basically saves the user from having to define the same output for all queries.

    Refer to the Layouts section of the user guide for more info.

    "},{"location":"user-guide/manifest/#formatterpgformatter","title":"formatter.pgFormatter","text":"

    This section is for configuring the pg_format tool that tapestry uses for formatting the rendered SQL files.

    There two config params under this section:

    "},{"location":"user-guide/manifest/#exec_path","title":"exec_path","text":"

    Location of the pg_format executable.

    "},{"location":"user-guide/manifest/#conf_path","title":"conf_path","text":"

    Path to the pg_format config file. It can be used for configuring the behavior of pg_format when it gets executed on rendered SQL. As with all paths that we've seen so far, this one is also relative to the manifest file.

    Example

    [formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\n

    As mentioned in the installation guide, pg_format is not a mandatory requirement but it's recommended.

    Upon running the tapestry init command, this section will be included in the auto-generated manifest file only if the executable pg_format is found on PATH. In that case, a default pg_format config file will also be created.

    To read more about configuring pg_format in the context of tapestry, refer to the pg_format section of the docs.

    "},{"location":"user-guide/manifest/#name_tagger","title":"name_tagger","text":"

    name_tagger is a TOML table, which if present in the manifest will cause the generated SQL queries to be name tagged.

    "},{"location":"user-guide/manifest/#style","title":"style","text":"

    name_tagger.style can be used to control how name tags will be derived from query id. The two options are:

    1. kebab-case
    2. snake_case
    3. exact

    Any special characters in the query id will be replaced with an appropriate character based on the above option \u2014 hyphen in case of kebab-case and underscore in case of snake_case. The third option exact is different in the sense that the query id will be used as it is as the name tag.

    Example:

    [name_tagger]\nstyle = \"kebab-case\"\n

    Note

    Note the autological naming of options kebab-case (with a hyphen) v/s snake_case (with an underscore).

    "},{"location":"user-guide/manifest/#query_templates","title":"query_templates","text":"

    query_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.

    For every query template, there are two keys to be defined:

    "},{"location":"user-guide/manifest/#path","title":"path","text":"

    It's where the Jinja template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.

    Use .j2 extension as the convention for the query template file.

    "},{"location":"user-guide/manifest/#all_conds","title":"all_conds","text":"

    It's a set of values that will be converted to cond__ Jinja variables that can be referenced inside the template. Note that they are defined in the manifest without the cond__ suffix.

    This field is optional. If not specified, an empty set is considered as the default.

    For documentation on how to write a query_template, refer to Writing query templates

    Example: 

    [[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\n\n[[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\n

    Note

    When all_conds is not specified, it essentially means that the query is a valid SQL statement and not a Jinja template. Then why define it as a template? The answer to that is \u2014 so that it can be embedded in tests.

    "},{"location":"user-guide/manifest/#queries","title":"queries","text":"

    queries is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.

    A query can be defined using the following keys,

    "},{"location":"user-guide/manifest/#id","title":"id","text":"

    id is an identifier for the query.

    "},{"location":"user-guide/manifest/#template","title":"template","text":"

    template is a reference to a query_template defined previously in the manifest.

    "},{"location":"user-guide/manifest/#conds","title":"conds","text":"

    conds is a subset of the all_conds key that's defined for the linked query template. It's an optional and if not specified, an empty set will be considered by default.

    "},{"location":"user-guide/manifest/#output","title":"output","text":"

    output is the path to the output file where the SQL query will be rendered. It must be relative to the queries_output_dir config.

    It's optional to specify the output. If not specified, the filename of the output file will be derived by slugifying the id. This property allows us to use certain Naming conventions for giving suitable and consistent names to the queries.

    Example:

    [[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n

    The derived value of output for the above will be artists_long_songs-genre-limit.sql.

    "},{"location":"user-guide/manifest/#name_tag","title":"name_tag","text":"

    name_tag can be optionally set to specify a custom name tag for the query. Name tags are prefixed to the SQL queries as comments and they are used by SQL loading libraries such as yesql, aiosql etc. Read more about in Name tagging queries.

    Note

    A query will be tagged with the specified name_tag only if name_tagger is set.

    "},{"location":"user-guide/manifest/#test_templates","title":"test_templates","text":"

    test_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.

    A test_template can be defined using the following keys,

    "},{"location":"user-guide/manifest/#query","title":"query","text":"

    query is a reference to query defined in the manifest.

    "},{"location":"user-guide/manifest/#path_1","title":"path","text":"

    path is the path to the jinja template for the pgTAP test. It must be relative to the test_templates_dir.

    Use .j2 extension as the convention for the test template file.

    "},{"location":"user-guide/manifest/#output_1","title":"output","text":"

    output is the path where the pgTAP test file will be rendered. It must be relative to the tests_output_dir.

    Specifying output for test_templates is optional. If not specified, it will be derived from the file stem of path i.e. by removing the .j2 extension.

    For detailed documentation on how to write a test_template, refer to Writing test templates

    "},{"location":"user-guide/naming-conventions/","title":"Query naming conventions","text":"

    One of the problems that I have encountered when using SQL loading libraries such as yesql and aiosql is that the queries defined in SQL files need to be given unique names. Often, one ends up writing a group of queries that are mostly similar to each other and differ only slightly. Giving unique and consistent names to each query can become tricky.

    A tool like tapestry cannot automatically give a name to a query. However, since the queries are listed in the manifest file, we can partly address the problem with the use of naming conventions.

    These naming conventions involve clever use of special characters such as @, +, & and *. Let's look at some examples from examples/chinook dir.

    [[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n\n[[queries]]\nid = \"songs_formats@artist&file_format+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\", \"file_format\" ]\n

    In the above queries, the id is defined using an alphanumeric prefix (artists_long_songs and songs_formats) followed by suffix that's an encoding of the conditional Jinja variables relevant to the query.

    The convention is as follows,

    • @ precedes \"cond\" vars used for conditionally including a filter i.e. a WHERE clause.
    • + precedes \"cond\" vars used for conditionally returning a column
    • * precedes \"cond\" vars used for conditionally including any other part of the query e.g. LIMIT, ORDER BY etc.
    • & is used as a delimiter between two \"cond\" vars of same type e.g. @artist&file_format.

    The name of the output file for the SQL query will be generated by slugifying the id i.e. by replacing the above special characters with hyphen (-). In case of the above two queries, the output file names will be artists_long_songs-genre-limit.sql and songs_formats-artist-file_format-album.sql respectively.

    Note that these naming conventions are only recommended by tapestry and are not mandatory.

    "},{"location":"user-guide/pg-format/","title":"Configuring pg_format","text":"

    tapestry relies on pg_format for formatting the rendered SQL files. This makes sure that,

    • the rendered SQL files have consistent indentation
    • you don't need to worry about SQL indentation when writing Jinja templates

    However, pg_format is not a hard requirement for tapestry. If pg_format is not installed on your system at the time of running tapestry init command, the formatter.pgFormatter section will not be added in the auto-generated manifest file.

    The behavior of pg_format tool in the context of tapestry can be configured by adding a config file. The sample config file in the pg_format github repo can be used for reference.

    The default config file generated by tapestry init command is located at .pg_format/config (relative to the manifest file) and looks like this,

    # Lines between markers 'start(noformat)' and 'end(noformat)' will not\n# be formatted. If you want to customize the markers, you may do so by\n# modifying this parameter.\nplaceholder=start\\(noformat\\).+end\\(noformat\\)\n\n# Add a list of function to be formatted as PG internal\n# functions. Paths relative to the 'tapestry.toml' file will also work\n#extra-function=./.pg_format/functions.lst\n\n# Add a list of keywords to be formatted as PG internal keywords.\n# Paths relative to the 'tapestry.toml' file will also work\n#extra-keyword=./.pg_format/keywords.lst\n\n# -- DANGER ZONE --\n#\n# Please donot change the following config parameters. Tapestry may\n# not work otherwise.\nmultiline=1\nformat=text\noutput=\n

    As you can see, the generated file itself is well documented.

    "},{"location":"user-guide/pg-format/#disallowed-configuration","title":"Disallowed configuration","text":"

    In the context of tapestry, some pg_format config params are disallowed (or they need to configured only in a certain way) for proper functioning of tapestry. These are explicitly defined with the intended value in the config file and annotated with DANGER ZONE warning in the comments. These must not be changed.

    "},{"location":"user-guide/pg-format/#selectively-opting-out-of-sql-formatting","title":"Selectively opting out of SQL formatting","text":"

    A commonly faced problem with formatting pgTAP tests using pg_format is that hard coded expected values get formatted in a way that could make the test case unreadable for humans.

    Example: Consider the following pgTAP test case written in a test template file,

    SELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 2)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n    $$,\n    'Verify return value'\n);\n

    By default pg_format would format the above SQL snippet as follows,

    SELECT\n    results_eq ('EXECUTE artists_long_songs(''Rock'', 2)', $$\n    VALUES (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval), (58, 'Deep Purple'::varchar, '00:19:56.094'::interval) $$, 'Verify return value');\n

    To retain the readability, we need to preserve the user's custom indentation. This is where the placeholder config param of pg_format is useful

    Note

    pg_format's placeholder config is not to be confused with placeholder config key in tapestry's manifest.

    This can be done by adding noformat markers before and after the snippet.

    -- start(noformat)\nSELECT results_eq(\n    'EXECUTE artists_long_songs(''Rock'', 2)',\n    $$VALUES\n        (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n        (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n    $$,\n    'Verify return value'\n);\n-- end(noformat)\n

    If you want to customize the markers for whatever reason, you can modify the placeholder param in the pg_format config file.

    "},{"location":"user-guide/query-tags/","title":"Query tags","text":""},{"location":"user-guide/query-tags/#name-tagging-queries","title":"Name tagging queries","text":"

    Typically, the output query files rendered by tapestry are intended to be used by libraries such as yesql, aiosql etc. These libraries require the queries to be \"name-tagged\". Tagging is done by simply adding a comment before the query as follows,

    -- name: my-query\n-- A simple query\nSELECT 1;\n

    This way, these libraries can map the queries with the functions that it generates in code. These functions wraps around the database client/driver code and provides an easy interface for the user.

    The following example is taken from yesql's README:

    -- name: users-by-country\nSELECT *\nFROM users\nWHERE country_code = :country_code\n

    ...and then read that file to turn it into a regular Clojure function:

    (defqueries \"some/where/users_by_country.sql\"\n   {:connection db-spec})\n\n;;; A function with the name `users-by-country` has been created.\n;;; Let's use it:\n(users-by-country {:country_code \"GB\"})\n;=> ({:name \"Kris\" :country_code \"GB\" ...} ...)\n
    "},{"location":"user-guide/query-tags/#deriving-name-tags-from-id","title":"Deriving name tags from id","text":"

    Tapestry does support name tagging of queries, but it's disabled by default. To enable it, just add the following lines in the manifest,

    [name_tagger]\nstyle = \"kebab-case\"\n

    This will result in name tags added to queries. The name tags are derived from the query ids. The style setting allows us to control how the id should be slugified to derive the name tag. For e.g. kebab-case will cause all non-alphanumeric characters in the id to be replaced by hyphens.

    The other options for style are snake_case and exact.

    "},{"location":"user-guide/query-tags/#custom-name-tags","title":"Custom name tags","text":"

    The above method derives name tags from query ids. But yesql and aiosql sometimes require the query names to be suffixed with specific characters to indicate specific operations. Example: In yesql, the name tags for INSERT/UPDATE/DELETE statements need to be suffixed with !.

    -- name: save-person!\nUPDATE person\n    SET name = :name\n    WHERE id = :id\n

    There are two ways to achieve this:

    1. Specify exact as the name_tagger.style. Then the query id itself to be used as the name tag (as it is).

    2. Specify the optional queries[].name_tag field when defining the queries.

    While it may seem like the first approach involves less effort, the downside is that we'd be giving up on the Naming conventions that tapestry recommends.

    Libraries such as yesql and aiosql usually don't allow special characters in the name tags as they use them to generate functions in code. So yesql recommends the name tags to be in kebab-case as Clojure functions follow that convention, whereas aiosql needs the name tags to be in snake_case as that's the requirement and also the convention in Python.

    "},{"location":"user-guide/query-templates/","title":"Writing query templates","text":"

    Query templates are Jinja template files. One query template can be used for generating multiple SQL queries.

    Often, an application needs to issue mostly similiar (or slightly different) queries to the db based on user input. Some examples:

    • two queries that are exactly similar, except that one returns all columns i.e. * whereas the other returns only selected rows
    • two queries that are exactly the same, except that one has a limit
    • multiple similar queries but different combination of WHERE clauses

    Using Jinja templates, it's possible to write a single query template that can render multiple SQL queries. This is possible with a combination of Jinja variables and {% if .. %}...{% endif %} blocks. This is pretty much the main idea behind query templates.

    "},{"location":"user-guide/query-templates/#cond-variables","title":"\"cond\" variables","text":"

    Query templates need to be defined in the manifest where we specify all_conds which is a set of \"cond\" vars that the template supports.

    Let's look at a query template from the chinook example distributed with the github repo.

    SELECT\n    track.name as title,\n    artist.name as artist_name,\n    {% if cond__album_name %}\n      album.title as album_name,\n    {% endif %}\n    media_type.name as file_format\nFROM\n    album\n    JOIN artist USING (artist_id)\n    LEFT JOIN track USING (album_id)\n    JOIN media_type USING (media_type_id)\n\n{% if cond__artist or cond__file_format %}\n  WHERE\n  {% set num_conds = 0 %}\n  {% if cond__artist %}\n    artist.name = {{ placeholder('artist') }}\n    {% set num_conds = num_conds + 1 %}\n  {% endif %}\n\n  {% if cond__file_format %}\n    {% if num_conds > 0 %}\n      AND\n    {% endif %}\n    media_type.name = {{ placeholder('file_format') }}\n    {% set num_conds = num_conds + 1 %}\n  {% endif %}\n{% endif %}\n;\n

    The entry in the manifest file for the above query_template is,

    [[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\n

    Because of the 3 all_conds defined in the manifest file, we have the following Jinja variables available inside the Jinja template.

    1. cond__artist
    2. cond__file_format
    3. cond__album_name

    The cond__artist and cond__file_format vars are used for conditionally including WHERE clauses. Because we want to add the WHERE clause only if either of the two vars are true, and because we want to add the AND operator only if both are true, nested if blocks are used and a temp \"counter\" variable num_conds is defined i.e. it's assigned to 0 and then incremented by 1 if the cond__artist var is true.

    The third variable cond__album_name is used for conditionally including a column in the returned result.

    "},{"location":"user-guide/query-templates/#query","title":"Query","text":"

    Now let's look at how a query associated with this template is defined in the manifest.

    [[queries]]\nid = \"songs_formats@artist+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\" ]\noutput = \"songs_formats__artist__album.sql\"\n

    In this query, only 2 of the 3 \"cond\" variables will be true.

    As a total of 3 all_conds values are supported by the query template, 8 different queries can be generated from it using different subsets of all_conds.

    [ ]\n[ \"artists\" ]\n[ \"artists\", \"file_format\" ]\n[ \"artists\", \"album_name\" ]\n[ \"file_format\" ]\n[ \"file_format\", \"album_name\" ]\n[ \"album_name\" ]\n[ \"artist\", \"file_format\", \"album_name\" ]\n
    "},{"location":"user-guide/test-templates/","title":"Test templates","text":"

    Just like query_templates, test_templates are also Jinja template files. But while one query template could be used to generate several queries, one test template can be used to generate only one pgTAP test file.

    However, many test templates can be associated with a single query. In other words, if multiple pgTAP test suites are to be written for the same query, that's possible.

    The test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. Important thing to note is that the Jinja variable {{ prepared_statement }} is made available to every test template, and at the time of rendering, it will expand to the actual query.

    Let's look at a templates from the chinook example.

    Refer to the test template songs_formats-afa_test.sql.j2. The first few lines are:

    PREPARE song_formats (varchar, varchar) AS\n{{ prepared_statement }};\n

    Here we're using the prepared_statement Jinja variable to create a prepared statement for the user session. The name of the prepared statement is song_formats and it takes two positional args, both of type varchar.

    Later in the same file, the prepared statement is executed as part of a pgTAP test case,

    SELECT results_eq(\n    'EXECUTE song_formats(''Iron Maiden'', ''Protected AAC audio file'')',\n    $$VALUES\n      ...\n      ...\n    $$,\n    'Verify return value'\n);\n

    Check the songs_formats-afa_test.sql output file to see how the actual test file looks like.

    Note

    Note that the SQL query that prepared_statement Jinja var expands to will always have posargs based placeholders, even if the placeholder config in manifest file is set to variables. That's the reason why the Jinja var is named prepared_statement

    "},{"location":"user-guide/test-templates/#function-instead-of-ps","title":"Function instead of PS","text":"

    Sometimes it's tedious to test for result sets returned by the query. In such cases, it helps to manipulate the result returned by the query and compare a derived property. E.g. If a query results too many rows, it's easier to compare the count than the actual values in the rows.

    One limitation of prepared statements and the EXECUTE syntax for executing them is that it's not sub-query friendly i.e. it's not possible to execute a prepared statement as part of another query.

    The following is NOT valid SQL

    SELECT\n    count(*)\nFROM (EXECUTE song_formats ('Iron Maiden', 'Protected AAC audio file'));\n

    In such cases, we can define a SQL function using the same prepared_statement Jinja variable.

    An example of this can be found in the chinook example - all_artists_long_songs_test.sql.j2

    CREATE OR REPLACE FUNCTION all_artists_long_songs ()\nRETURNS SETOF record\nAS $$\n{{ prepared_statement }}\n$$ LANGUAGE sql;\n\nBEGIN;\nSELECT\n    plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT is(count(*), 204::bigint) from all_artists_long_songs() AS (artist_id int, name text, duration interval);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n    *\nFROM\n    finish ();\nROLLBACK;\n
    "},{"location":"user-guide/test-templates/#test-fixtures","title":"Test fixtures","text":"

    When it comes to automated tests, It's a very common requirement to setup some test data to be able to write test cases. pgTAP tests are not any different. In case of pgTAP one needs to create test data in the database.

    Since pgTAP tests are just SQL files, test data creation can be done using SQL itself in the same file. Reusable setup code can also be extracted into SQL functions that can be created as part of importing the database schema.

    The chinook directory doesn't include an example of this. But here's an example from one of my real projects that uses tapestry.

    In my project, there are two entities categories and items (having tables of the same names) with one-to-many relationship i.e. one category can have multiple items.

    In several pgTAP tests, a few categories and items need to be created. To do this, a function is defined as follows,

    CREATE OR REPLACE FUNCTION tapestry.setup_category_n_items (cat_id varchar, item_idx_start integer, item_idx_end integer)\n    RETURNS void\n    AS $$\n    INSERT INTO categories (id, name)\n        VALUES (cat_id, initcap(replace(cat_id, '-', ' ')));\n    INSERT INTO items (id, name, category_id)\n    SELECT\n        'item-' || t AS id,\n        'Item ' || t AS name,\n        cat_id AS category_id\n    FROM\n        generate_series(item_idx_start, item_idx_end) t;\n$$\nLANGUAGE sql;\n

    And then it's used in pgTAP tests like this, 

    ...\n\nBEGIN;\nSELECT plan(1);\n\n-- Fixtures\n-- create 2 categories, 'cat-a' and 'cat-b' each having 5 items\nSELECT\n    tapestry.setup_category_n_items ('cat-a', 1, 5);\nSELECT\n    tapestry.setup_category_n_items ('cat-b', 6, 10);\n\n-- Test cases\n\n...\n
    "}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 60b5e3846b0a98d899be016d97a6067379d1b237..4d950e28eff243d7564a2b2b77130eec0dcf0848 100644 GIT binary patch delta 13 Ucmb=gXP58h;8>wtJdwQu02}rM4gdfE delta 13 Ucmb=gXP58h;8;-~JdwQu03AXEN&o-= diff --git a/user-guide/commands/index.html b/user-guide/commands/index.html index a1e0a24..a47d794 100644 --- a/user-guide/commands/index.html +++ b/user-guide/commands/index.html @@ -468,6 +468,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • @@ -593,6 +635,21 @@ + +
  • @@ -781,6 +838,21 @@ + +
  • @@ -888,6 +960,31 @@

    --assert-no-changes

    summary

    The summary command prints a tabular summary of all queries along with their associated (query) templates and tests.

    +

    --all

    +

    When --all option is specified with this command, the summary will +include query and test files inside queries_output_dir and +tests_output_dir respectively that are not added to the manifest.

    +
    +

    Note

    +

    There are legit use cases for having files in the query and test +output directories that are not added to the manifest Examples:

    +
      +
    1. +

      queries that don't need any tests but need to be stored in the same +directory as other queries, so that yesql, aiosql libs can load all of +them together.

      +
      2. +
    2. +

      Existing queries which are not yet migrated to tapestry (gradual +migration strategy).

      +
      3. +
    3. +

      pgTAP tests written for stored procedures, views, schema etc. that +need to be stored in the same directory as other tests, so that all +tests can be run together.

      +
      4. +
    +

    coverage

    The coverage command prints a list of queries along with the no. of tests (i.e. pgTAP test files) for them. It also prints a coverage diff --git a/user-guide/docker/index.html b/user-guide/docker/index.html index 150e7a9..b34d85d 100644 --- a/user-guide/docker/index.html +++ b/user-guide/docker/index.html @@ -468,6 +468,48 @@ +

  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/user-guide/getting-started/index.html b/user-guide/getting-started/index.html index 5e2095a..b976905 100644 --- a/user-guide/getting-started/index.html +++ b/user-guide/getting-started/index.html @@ -571,6 +571,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • @@ -837,6 +879,9 @@

    Init

    exec_path = "pg_format" conf_path = "./.pg_format/config" +[name_tagger] +style = "kebab-case" + # [[query_templates]] # [[queries]] @@ -872,6 +917,8 @@

    Init

    us. This is because it found the pg_format executable on PATH. Refer to the     pg_format section for more details.

    +

    Finally, name_tagger has been configured +with kebab-case as the style.

    Adding a query_template to generate queries

    Now we'll define a query template. But before that, you might want to get yourself familiar with the chinook database's @@ -1015,7 +1062,8 @@

    Rendering

    -
    SELECT
+
    -- name: artists-long-songs
+SELECT
     ar.artist_id,
     ar.name,
     max(milliseconds) * interval '1 ms' AS duration
@@ -1031,7 +1079,8 @@ 
    Rendering
    -
    SELECT
+
    -- name: artists-long-songs-limit
+SELECT
     ar.artist_id,
     ar.name,
     max(milliseconds) * interval '1 ms' AS duration
@@ -1048,7 +1097,8 @@ 
    Rendering
    -
    SELECT
+
    -- name: artists-long-songs-genre-limit
+SELECT
     ar.artist_id,
     ar.name,
     max(milliseconds) * interval '1 ms' AS duration
@@ -1069,7 +1119,10 @@ 
    Rendering
    -

    As you can see, the output SQL is formatted by pg_format.

    +

    The SQL comments before the SQL with name of the query are generated +by name_tagger added to the manifest. Learn more about Name +tagging.

    +

    Also notice that the output SQL is formatted by pg_format.

    Adding a test_template

    Now that we've defined and rendered queries, let's add test_template. Again there are two changes required - an entry in diff --git a/user-guide/index.html b/user-guide/index.html index 7fb5ca1..9a4ea26 100644 --- a/user-guide/index.html +++ b/user-guide/index.html @@ -478,6 +478,48 @@ +

  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/user-guide/install/index.html b/user-guide/install/index.html index 913476a..c59a101 100644 --- a/user-guide/install/index.html +++ b/user-guide/install/index.html @@ -526,6 +526,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/user-guide/layouts.md~ b/user-guide/layouts.md~ new file mode 100644 index 0000000..ce80eef --- /dev/null +++ b/user-guide/layouts.md~ @@ -0,0 +1,20 @@ +# Layouts + +Tapestry lets you control the layout of the generated query files. You +can organize the queries in two ways: + +1. `one-file-one-query`: Each SQL query will be written to a separate file +2. `one-file-all-queries`: All SQL queries will be written to a single file + +This can be specified using `query_output_layout` key in the manifest +file. If not specified, `one-file-one-query` will be considered as +default. + +When layout is set to `one-file-all-queries`, it's expected that the +`output` field for all queries is the same i.e. path to the file that +all SQL will be written to. To avoid this duplication, you may +configure an associated setting `query_output_file`. + + + + diff --git a/user-guide/layouts/index.html b/user-guide/layouts/index.html new file mode 100644 index 0000000..f07fe72 --- /dev/null +++ b/user-guide/layouts/index.html @@ -0,0 +1,802 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Layouts - tapestry + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + Skip to content + + +
    +
    + +
    + + + + +
    + + +
    + +
    + + + + + + + + + +
    +
    + + + +
    +
    +
    + + + + + + + +
    +
    +
    + + + +
    +
    +
    + + + +
    +
    +
    + + + +
    +
    + + + + + + + +

    Layouts

    +

    Tapestry lets you control the layout of the query files i.e. how the +generated SQL is organized in files. It supports two ways at present:

    +
      +
    1. one-file-one-query: Each SQL query will be written to a separate file
      2. +
    2. one-file-all-queries: All SQL queries will be written to a single file
      3. +
    +

    To configure this, you need to specify the query_output_layout key +in the manifest. The default option if not specified is +one-file-one-query.

    +

    Layout and queries[].output field

    +

    Users may specify output field +for every query, which is the path where the generated SQL output will +be written. If output is not specified, it's value is derived from +the query id. This works well for the one-file-one-query layout.

    +

    When the layout is one-file-all-queries, it's expected that the +output field of all queries must be the same. Otherwise the manifest +fails to validate. To avoid duplication, a related setting +query_output_file is provided.

    +

    If layout = one-file-all-queries, it's recommended to set +query_output_file and omit the output field for individual +queries.

    +

    If layout = one-file-one-query, then you must not set +query_output_file. Whether or not to set the output field for +individual queries is up to you.

    + + + + + + + + + + + + + +
    +
    + + + +
    + +
    + + + +
    +
    +
    +
    + + + + + + + + + + \ No newline at end of file diff --git a/user-guide/manifest/index.html b/user-guide/manifest/index.html index 04deeb5..d74286a 100644 --- a/user-guide/manifest/index.html +++ b/user-guide/manifest/index.html @@ -518,6 +518,24 @@ +
    • + +
  • + + + query_output_layout + + + +
    • + +
  • + + + query_output_file + + +
  • @@ -551,6 +569,30 @@ +
    • + +
  • + + + name_tagger + + + + +
  • @@ -630,6 +672,15 @@ +
    • + +
  • + + + name_tag + + +
    • @@ -736,6 +787,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • @@ -934,6 +1027,24 @@ +
    • + +
  • + + + query_output_layout + + + +
    • + +
  • + + + query_output_file + + +
  • @@ -967,6 +1078,30 @@ +
    • + +
  • + + + name_tagger + + + + +
  • @@ -1046,6 +1181,15 @@ +
    • + +
  • + + + name_tag + + +
    • @@ -1236,6 +1380,25 @@

    tests_output_dir

    init     command will have,

    tests_output_dir = "output/tests"
    +

    query_output_layout

    +

    Layout to be used for the generated query files. The two +options are:

    +
      +
    1. +

      one-file-one-query: Each SQL query will be written to a separate file

      +
      2. +
    2. +

      one-file-all-queries: All SQL queries will be written to a single file

      +
      3. +
    +

    It's optional. The default value is one-file-one-query.

    +

    query_output_file

    +

    query_output_file is optional but it's use is valid only when the +layout is one-file-all-queries. It basically +saves the user from having to define the same output for +all queries.

    +

    Refer to the Layouts section of the user guide for more +info.

    formatter.pgFormatter

    This section is for configuring the pg_format tool that tapestry uses for formatting the rendered SQL files.

    @@ -1261,6 +1424,32 @@

    conf_path

    To read more about configuring pg_format in the context of tapestry, refer to the pg_format section of the docs.

    +

    name_tagger

    +

    name_tagger is a TOML table, which if present in the manifest will +cause the generated SQL queries to be name +tagged.

    +

    style

    +

    name_tagger.style can be used to control how name tags will be +derived from query id. The two options are:

    +
      +
    1. kebab-case
      2. +
    2. snake_case
      3. +
    3. exact
      4. +
    +

    Any special characters in the query id will be replaced with an +appropriate character based on the above option — hyphen in case +of kebab-case and underscore in case of snake_case. The third +option exact is different in the sense that the query id will be +used as it is as the name tag.

    +

    Example:

    +
    [name_tagger]
+style = "kebab-case"
+
    +
    +

    Note

    +

    Note the autological naming of options kebab-case (with a hyphen) +v/s snake_case (with an underscore).

    +

    query_templates

    query_templates is an array of tables in TOML @@ -1277,6 +1466,8 @@

    all_conds

    It's a set of values that will be converted to cond__ Jinja variables that can be referenced inside the template. Note that they are defined in the manifest without the cond__ suffix.

    +

    This field is optional. If not specified, an empty set is considered +as the default.

    For documentation on how to write a query_template, refer to Writing query templates

    Example:

    @@ -1288,6 +1479,13 @@

    all_conds

    path = "songs_formats.sql.j2" all_conds = [ "artist", "file_format", "album_name" ]     +
    +

    Note

    +

    When all_conds is not specified, it essentially means that the query +is a valid SQL statement and not a Jinja template. Then why define it +as a template? The answer to that is — so that it can be +embedded in tests.

    +

    queries

    queries is an array of tables in TOML @@ -1301,7 +1499,8 @@

    template

    defined previously in the manifest.

    conds

    conds is a subset of the all_conds key that's defined for the -linked query template.

    +linked query template. It's an optional and if not specified, an empty +set will be considered by default.

    output

    output is the path to the output file where the SQL query will be rendered. It must be relative to the queries_output_dir config.

    @@ -1316,6 +1515,18 @@

    output

    template = "artists_long_songs.sql.j2" conds = [ "genre", "limit" ] +

    The derived value of output for the above will be +artists_long_songs-genre-limit.sql.

    +

    name_tag

    +

    name_tag can be optionally set to specify a custom name tag for the +query. Name tags are prefixed to the SQL queries as comments and they +are used by SQL loading libraries such as yesql, aiosql etc. Read more +about in Name tagging queries.

    +
    +

    Note

    +

    A query will be tagged with the specified name_tag only if +name_tagger is set.

    +

    test_templates

    test_templates is an array of tables in TOML diff --git a/user-guide/naming-conventions/index.html b/user-guide/naming-conventions/index.html index df52171..6ea8ce5 100644 --- a/user-guide/naming-conventions/index.html +++ b/user-guide/naming-conventions/index.html @@ -11,7 +11,7 @@ - + @@ -466,6 +466,48 @@ + + +

  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + diff --git a/user-guide/pg-format/index.html b/user-guide/pg-format/index.html index 1a73d9f..41dc9f7 100644 --- a/user-guide/pg-format/index.html +++ b/user-guide/pg-format/index.html @@ -468,6 +468,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/user-guide/query-tags.md~ b/user-guide/query-tags.md~ new file mode 100644 index 0000000..e69de29 diff --git a/user-guide/query-tags/index.html b/user-guide/query-tags/index.html new file mode 100644 index 0000000..5f5b75e --- /dev/null +++ b/user-guide/query-tags/index.html @@ -0,0 +1,898 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Query Tags - tapestry + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + Skip to content + + +
    +
    + +
    + + + + +
    + + +
    + +
    + + + + + + + + + +
    +
    + + + +
    +
    +
    + + + + + + + +
    +
    +
    + + + +
    +
    +
    + + + +
    +
    +
    + + + +
    +
    + + + + + + + +

    Query tags

    +

    Name tagging queries

    +

    Typically, the output query files rendered by tapestry are intended to +be used by libraries such as yesql, aiosql etc. These libraries +require the queries to be "name-tagged". Tagging is done by simply +adding a comment before the query as follows,

    +
    -- name: my-query
+-- A simple query
+SELECT 1;
+
    +

    This way, these libraries can map the queries with the functions that +it generates in code. These functions wraps around the database +client/driver code and provides an easy interface for the user.

    +

    The following example is taken from yesql's +README:

    +
    -- name: users-by-country
+SELECT *
+FROM users
+WHERE country_code = :country_code
+
    +

    ...and then read that file to turn it into a regular Clojure function:

    +
    (defqueries "some/where/users_by_country.sql"
+   {:connection db-spec})
+
+;;; A function with the name `users-by-country` has been created.
+;;; Let's use it:
+(users-by-country {:country_code "GB"})
+;=> ({:name "Kris" :country_code "GB" ...} ...)
+
    +

    Deriving name tags from id

    +

    Tapestry does support name tagging of queries, but it's disabled by +default. To enable it, just add the following lines in the manifest,

    +
    [name_tagger]
+style = "kebab-case"
+
    +

    This will result in name tags added to queries. The name tags are +derived from the query ids. The style setting +allows us to control how the id should be slugified to derive the name +tag. For e.g. kebab-case will cause all non-alphanumeric characters +in the id to be replaced by hyphens.

    +

    The other options for style are snake_case and exact.

    +

    Custom name tags

    +

    The above method derives name tags from query ids. But yesql and +aiosql sometimes require the query names to be suffixed with specific +characters to indicate specific operations. Example: In yesql, the +name tags for INSERT/UPDATE/DELETE statements need to be suffixed with +!.

    +
    -- name: save-person!
+UPDATE person
+    SET name = :name
+    WHERE id = :id
+
    +

    There are two ways to achieve this:

    +
      +
    1. +

      Specify exact as the name_tagger.style. Then the query + id itself to be used as the name tag (as it + is).

      +
      2. +
    2. +

      Specify the optional queries[].name_tag + field when defining the queries.

      +
      3. +
    +

    While it may seem like the first approach involves less effort, the +downside is that we'd be giving up on the Naming +conventions that tapestry recommends.

    +

    Libraries such as yesql and aiosql usually don't allow special +characters in the name tags as they use them to generate functions in +code. So yesql recommends the name tags to be in kebab-case as +Clojure functions follow that convention, whereas aiosql needs the +name tags to be in snake_case as that's the requirement and also the +convention in Python.

    + + + + + + + + + + + + + +
    +
    + + + +
    + +
    + + + +
    +
    +
    +
    + + + + + + + + + + \ No newline at end of file diff --git a/user-guide/query-templates/index.html b/user-guide/query-templates/index.html index 9a60458..dbb24c5 100644 --- a/user-guide/query-templates/index.html +++ b/user-guide/query-templates/index.html @@ -526,6 +526,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +
  • diff --git a/user-guide/test-templates/index.html b/user-guide/test-templates/index.html index 7d9c1e6..7dce986 100644 --- a/user-guide/test-templates/index.html +++ b/user-guide/test-templates/index.html @@ -14,7 +14,7 @@ - + @@ -526,6 +526,48 @@ +
  • + + + + + Layouts + + + + +
    • + + + + + + + + + + +
  • + + + + + Query Tags + + + + +
    • + + + + + + + + + +