PostgreSQL + GraphQL (#345)

* fix: create chain data directory if it doesn't exist (#331)

* Upgrade chainsauce (#315)

* chore: upgrade chainsauce package

* chore: vendor in chainsauce cache

* feat: update indexer to use new chainsauce interface

* chore: reorganize abis as a flat map

* chore: restore roundcreted event rename

* chore: configure new json storage

* fix: unused import

* fix: await live indexer

* fix: indexer watch after catchup

* chore: update event handler builder

* fix: set event poll interval to 20 seconds

* fix: index project registry v2

* feat: update applications in parallel

* feat: fix up disk cache

* fix: use strict null check in cache

* fix: do not silence disk write errors

* feat: allow disabling chainsauce cache

* chore: rename diskCache to deprecatedDiskCache

* fix: error handling for live indexing

* feat: use event types from chainsauce

* fix: snapshot differences

* feat: change to eventemitter interface

* chore: update new chainsauce rpc client interface

* fix: calculator vote assignment

* fix: @typescript-eslint/unbound-method linter error for subscribeToContract

* fix: handle event handler errors

* fix: proportional match test

* enh: use object for clearer logging

* fix: use hex type

* fix: shorten log call

* chore: remove leftover conflict marker

* feat: implement resource monitor (#330)

* feat: implement resource monitor

* fix: monitor configured storage directories

* fix: check specific directory

* chore: move polyfilled types to types.d.ts

* fix: boolean coercion to enable res monitor

* fix: resourcemonitor test

* feat: check in types.d.ts

* feat: separate tests

* fix:r esourcelog import

* feat: sqlite backed indexing

* fix: update polygon usdc to native usdc (#334)

feat: update polygon usdc to native usdc

* Add dai support to pgn mainnet (#335)

feat: dai support to pgn mainnet

* postgres poc

* postgres poc

* Add gist valueless token on optimism (#341)

feat: gist valueless token on optimism

* fix: gist address (#343)

* fix: gist address

* chore: update voteAmountCap for gist

---------

Co-authored-by: eagle <[email protected]>

* fix: update polygon usdc to native usdc (#334)

feat: update polygon usdc to native usdc

* Add dai support to pgn mainnet (#335)

feat: dai support to pgn mainnet

* Add gist valueless token on optimism (#341)

feat: gist valueless token on optimism

* fix: gist address (#343)

* fix: gist address

* chore: update voteAmountCap for gist

---------

Co-authored-by: eagle <[email protected]>

* handle invalid dates

* feat: switch to pg, use schemas to namespace versions

* feat: data driven database repository

* chore: remove unused migrations from drizzle

* chore: delete drizzle

* sorry i have no idea

* chore: update chansauce for postgres support

* feat: add rpc logger

* fix: lint

* feat: add docker compose

* fix: allow json fields in postgraphile

* feat: add simplify names plugin

* feat: simplify grapql names

* feat: snake case postgres columns

* feat: update chainsauce for batched indexing

* feat: index prices in postgres

* feat: add postgraphile filters

* fix: tests

* feat: handle event tests

* fix: handle event tests

* chore: remove snapshot scripts

* buffered event inserts, cached round match amount, delete price updater

* update chainsauce

* fix price fetching

* feat: multi chain prices

* round and app stats

* async donation handling

* get contract name for event

* fix tests

* rename mutation to changeset

* rename mutation to changeset

* rename mutation to changeset

* reduce log noise

* fix retry for coingecko and rpc

* reduce log noise

* add tags to rounds and projects

* donor address index

* add changeset

* update chainsauce

* WIP

* fix registry abi

* index allo v2 profiles creation

* temporarily set projectNumber to 0 for allo v2 profiles

* fix linter

* bump version and index all chains in staging

* plain methods for querying

* split allov1 and allov2

* stop logging rpc calls

* refactor to use new query methods

* fix tests

* update readme

* unnest storage dir one level

* parse address in calculator

* fix: applications unique donors count

* fix: remove duplicate pluralistic

* fix tests

* fix linter

* increase data version and enable staging deployments

---------

Co-authored-by: eagle <[email protected]>
Co-authored-by: Atris <[email protected]>
Co-authored-by: Andrea Franz <[email protected]>

.env.example

@@ -1,14 +1,18 @@
 INDEXED_CHAINS=pgn-testnet
 PASSPORT_SCORER_ID=335
-STORAGE_DIR=/tmp/grants-stack-indexer/storage/data
-CACHE_DIR=/tmp/grants-stack-indexer/storage/cache
+STORAGE_DIR=./var
+CACHE_DIR=/.var/cache
 DEPLOYMENT_ENVIRONMENT=local
 PORT=8080
 LOG_LEVEL=debug
 # The BUILD_TAG will be set at deployment time to the latest commit hash
 BUILD_TAG=local
+ENABLE_RESOURCE_MONITOR=false
+
+DATABASE_URL=postgres://postgres:postgres@localhost:5432/grants_stack_indexer
 
 ESTIMATES_LINEARQF_WORKER_POOL_SIZE=10
+
 COINGECKO_API_KEY=
 
 #LOG_LEVEL=
@@ -25,4 +29,4 @@ COINGECKO_API_KEY=
 #POLYGON_RPC_URL=
 #POLYGON_MUMBAI_RPC_URL=
 #COINGECKO_API_KEY=
-#IPFS_GATEWAY=
+#IPFS_GATEWAY=

.github/workflows/deploy-branch.yml

@@ -2,7 +2,7 @@ name: Deploy Branch
 
 on:
   push:
-    branches: [release]
+    branches: [main, release]
 
 jobs:
   build-and-deploy:

README.md

@@ -2,38 +2,26 @@
 
 The Grants Stack Indexer is a tool that indexes blockchain events generated by [Allo contracts](https://github.com/Allo-Protocol/contracts) and serves the data over HTTP in JSON format. The data is organized in a specific structure that enables easy access to different parts of the protocol. The indexer is built using [Chainsauce](https://github.com/boudra/chainsauce) and is designed to work with any EVM-compatible chain.
 
-The indexer data is used by [Grants Stack](https://github.com/gitcoinco/grants-stack) to show stats and allocate matching funds.
+The indexer data is used by [Grants Stack](https://github.com/gitcoinco/grants-stack) as a data layer.
 
 ## API
 
-Access indexed data through this URL: https://grants-stack-indexer.gitcoin.co/data/
+Access indexed data through the GraphQL endpoint at: https://grants-stack-indexer.gitcoin.co/graphql/
 
-The indexed data follows this structure:
-
-```
-/{chainId}/rounds.json
-/{chainId}/prices.json
-/{chainId}/projects.json
-/{chainId}/rounds/{roundId}/projects.json
-/{chainId}/rounds/{roundId}/projects/{projectId}/votes.json
-/{chainId}/rounds/{roundId}/projects/{projectId}/contributors.json
-/{chainId}/rounds/{roundId}/applications.json
-/{chainId}/rounds/{roundId}/applications/{applicationIndex}/votes.json
-/{chainId}/rounds/{roundId}/votes.json
-/{chainId}/rounds/{roundId}/contributors.json
-```
+Use the GraphiQL Playground to see what queries are available: https://grants-stack-indexer.gitcoin.co/graphiql/
 
 Indexed chains are defined in [config.ts](src/config.ts).
 
 ## Setup
 
 Copy `.env.example` to `.env`, review and optionally customize it.
 
-To pick chains to index, set `INDEXED_CHAINS` to a comma-separated list of chain identifiers. Chain identifiers can be found in `src/config.ts`.
+To pick chains to index, set `INDEXED_CHAINS` to a comma-separated list of chain identifiers. Available chain identifiers can be found in `src/config.ts`.
 
 ## Running in development
 
 ```bash
+docker-compose up -d
 npm install
 npm run dev
 ```

docker-compose.yml

@@ -0,0 +1,11 @@
+version: "3"
+
+services:
+  db:
+    image: postgres:13
+    ports:
+      - "5432:5432"
+    environment:
+      - POSTGRES_USER=postgres
+      - POSTGRES_PASSWORD=postgres
+      - POSTGRES_DB=grants_stack_indexer

fly.development.toml

@@ -12,6 +12,7 @@ kill_timeout = 5
   LOG_LEVEL = "debug"
   STORAGE_DIR = "/mnt/indexer"
   INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-testnet,pgn-mainnet,polygon-mumbai,polygon"
+  ENABLE_RESOURCE_MONITOR = "true"
   # BUILD_TAG = # set in deploy github action
 
 [processes]

fly.production.toml

@@ -11,7 +11,8 @@ kill_timeout = 5
   DEPLOYMENT_ENVIRONMENT = "production"
   LOG_LEVEL = "debug"
   STORAGE_DIR = "/mnt/indexer"
-  INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-mainnet,arbitrum,arbitrum-goerli,polygon-mumbai,polygon"
+  INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-testnet,pgn-mainnet,arbitrum,arbitrum-goerli,polygon-mumbai,polygon"
+  ENABLE_RESOURCE_MONITOR = "true"
   NODE_OPTIONS="--max-old-space-size=7168"
   # BUILD_TAG = # set in deploy github action
 

fly.staging.toml

@@ -12,6 +12,7 @@ kill_timeout = 5
   LOG_LEVEL = "debug"
   STORAGE_DIR = "/mnt/indexer"
   INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-testnet,pgn-mainnet,arbitrum,arbitrum-goerli,polygon-mumbai,polygon"
+  ENABLE_RESOURCE_MONITOR = "true"
   # BUILD_TAG = # set in deploy github action
 
 [processes]