Skip to content

Commit

Permalink
PostgreSQL + GraphQL (#345)
Browse files Browse the repository at this point in the history
* fix: create chain data directory if it doesn't exist (#331)

* Upgrade chainsauce (#315)

* chore: upgrade chainsauce package

* chore: vendor in chainsauce cache

* feat: update indexer to use new chainsauce interface

* chore: reorganize abis as a flat map

* chore: restore roundcreted event rename

* chore: configure new json storage

* fix: unused import

* fix: await live indexer

* fix: indexer watch after catchup

* chore: update event handler builder

* fix: set event poll interval to 20 seconds

* fix: index project registry v2

* feat: update applications in parallel

* feat: fix up disk cache

* fix: use strict null check in cache

* fix: do not silence disk write errors

* feat: allow disabling chainsauce cache

* chore: rename diskCache to deprecatedDiskCache

* fix: error handling for live indexing

* feat: use event types from chainsauce

* fix: snapshot differences

* feat: change to eventemitter interface

* chore: update new chainsauce rpc client interface

* fix: calculator vote assignment

* fix: @typescript-eslint/unbound-method linter error for subscribeToContract

* fix: handle event handler errors

* fix: proportional match test

* enh: use object for clearer logging

* fix: use hex type

* fix: shorten log call

* chore: remove leftover conflict marker

* feat: implement resource monitor (#330)

* feat: implement resource monitor

* fix: monitor configured storage directories

* fix: check specific directory

* chore: move polyfilled types to types.d.ts

* fix: boolean coercion to enable res monitor

* fix: resourcemonitor test

* feat: check in types.d.ts

* feat: separate tests

* fix:r esourcelog import

* feat: sqlite backed indexing

* fix: update polygon usdc to native usdc (#334)

feat: update polygon usdc to native usdc

* Add dai support to pgn mainnet (#335)

feat: dai support to pgn mainnet

* postgres poc

* postgres poc

* Add gist valueless token on optimism (#341)

feat: gist valueless token on optimism

* fix: gist address (#343)

* fix: gist address

* chore: update voteAmountCap for gist

---------

Co-authored-by: eagle <[email protected]>

* fix: update polygon usdc to native usdc (#334)

feat: update polygon usdc to native usdc

* Add dai support to pgn mainnet (#335)

feat: dai support to pgn mainnet

* Add gist valueless token on optimism (#341)

feat: gist valueless token on optimism

* fix: gist address (#343)

* fix: gist address

* chore: update voteAmountCap for gist

---------

Co-authored-by: eagle <[email protected]>

* handle invalid dates

* feat: switch to pg, use schemas to namespace versions

* feat: data driven database repository

* chore: remove unused migrations from drizzle

* chore: delete drizzle

* sorry i have no idea

* chore: update chansauce for postgres support

* feat: add rpc logger

* fix: lint

* feat: add docker compose

* fix: allow json fields in postgraphile

* feat: add simplify names plugin

* feat: simplify grapql names

* feat: snake case postgres columns

* feat: update chainsauce for batched indexing

* feat: index prices in postgres

* feat: add postgraphile filters

* fix: tests

* feat: handle event tests

* fix: handle event tests

* chore: remove snapshot scripts

* buffered event inserts, cached round match amount, delete price updater

* update chainsauce

* fix price fetching

* feat: multi chain prices

* round and app stats

* async donation handling

* get contract name for event

* fix tests

* rename mutation to changeset

* rename mutation to changeset

* rename mutation to changeset

* reduce log noise

* fix retry for coingecko and rpc

* reduce log noise

* add tags to rounds and projects

* donor address index

* add changeset

* update chainsauce

* WIP

* fix registry abi

* index allo v2 profiles creation

* temporarily set projectNumber to 0 for allo v2 profiles

* fix linter

* bump version and index all chains in staging

* plain methods for querying

* split allov1 and allov2

* stop logging rpc calls

* refactor to use new query methods

* fix tests

* update readme

* unnest storage dir one level

* parse address in calculator

* fix: applications unique donors count

* fix: remove duplicate pluralistic

* fix tests

* fix linter

* increase data version and enable staging deployments

---------

Co-authored-by: eagle <[email protected]>
Co-authored-by: Atris <[email protected]>
Co-authored-by: Andrea Franz <[email protected]>
  • Loading branch information
4 people authored Jan 8, 2024
1 parent e6ad887 commit 6560b41
Show file tree
Hide file tree
Showing 95 changed files with 15,081 additions and 7,370 deletions.
10 changes: 7 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
INDEXED_CHAINS=pgn-testnet
PASSPORT_SCORER_ID=335
STORAGE_DIR=/tmp/grants-stack-indexer/storage/data
CACHE_DIR=/tmp/grants-stack-indexer/storage/cache
STORAGE_DIR=./var
CACHE_DIR=/.var/cache
DEPLOYMENT_ENVIRONMENT=local
PORT=8080
LOG_LEVEL=debug
# The BUILD_TAG will be set at deployment time to the latest commit hash
BUILD_TAG=local
ENABLE_RESOURCE_MONITOR=false

DATABASE_URL=postgres://postgres:postgres@localhost:5432/grants_stack_indexer

ESTIMATES_LINEARQF_WORKER_POOL_SIZE=10

COINGECKO_API_KEY=

#LOG_LEVEL=
Expand All @@ -25,4 +29,4 @@ COINGECKO_API_KEY=
#POLYGON_RPC_URL=
#POLYGON_MUMBAI_RPC_URL=
#COINGECKO_API_KEY=
#IPFS_GATEWAY=
#IPFS_GATEWAY=
2 changes: 1 addition & 1 deletion .github/workflows/deploy-branch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Deploy Branch

on:
push:
branches: [release]
branches: [main, release]

jobs:
build-and-deploy:
Expand Down
22 changes: 5 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,26 @@

The Grants Stack Indexer is a tool that indexes blockchain events generated by [Allo contracts](https://github.com/Allo-Protocol/contracts) and serves the data over HTTP in JSON format. The data is organized in a specific structure that enables easy access to different parts of the protocol. The indexer is built using [Chainsauce](https://github.com/boudra/chainsauce) and is designed to work with any EVM-compatible chain.

The indexer data is used by [Grants Stack](https://github.com/gitcoinco/grants-stack) to show stats and allocate matching funds.
The indexer data is used by [Grants Stack](https://github.com/gitcoinco/grants-stack) as a data layer.

## API

Access indexed data through this URL: https://grants-stack-indexer.gitcoin.co/data/
Access indexed data through the GraphQL endpoint at: https://grants-stack-indexer.gitcoin.co/graphql/

The indexed data follows this structure:

```
/{chainId}/rounds.json
/{chainId}/prices.json
/{chainId}/projects.json
/{chainId}/rounds/{roundId}/projects.json
/{chainId}/rounds/{roundId}/projects/{projectId}/votes.json
/{chainId}/rounds/{roundId}/projects/{projectId}/contributors.json
/{chainId}/rounds/{roundId}/applications.json
/{chainId}/rounds/{roundId}/applications/{applicationIndex}/votes.json
/{chainId}/rounds/{roundId}/votes.json
/{chainId}/rounds/{roundId}/contributors.json
```
Use the GraphiQL Playground to see what queries are available: https://grants-stack-indexer.gitcoin.co/graphiql/

Indexed chains are defined in [config.ts](src/config.ts).

## Setup

Copy `.env.example` to `.env`, review and optionally customize it.

To pick chains to index, set `INDEXED_CHAINS` to a comma-separated list of chain identifiers. Chain identifiers can be found in `src/config.ts`.
To pick chains to index, set `INDEXED_CHAINS` to a comma-separated list of chain identifiers. Available chain identifiers can be found in `src/config.ts`.

## Running in development

```bash
docker-compose up -d
npm install
npm run dev
```
Expand Down
11 changes: 11 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
version: "3"

services:
db:
image: postgres:13
ports:
- "5432:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=grants_stack_indexer
1 change: 1 addition & 0 deletions fly.development.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ kill_timeout = 5
LOG_LEVEL = "debug"
STORAGE_DIR = "/mnt/indexer"
INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-testnet,pgn-mainnet,polygon-mumbai,polygon"
ENABLE_RESOURCE_MONITOR = "true"
# BUILD_TAG = # set in deploy github action

[processes]
Expand Down
3 changes: 2 additions & 1 deletion fly.production.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ kill_timeout = 5
DEPLOYMENT_ENVIRONMENT = "production"
LOG_LEVEL = "debug"
STORAGE_DIR = "/mnt/indexer"
INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-mainnet,arbitrum,arbitrum-goerli,polygon-mumbai,polygon"
INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-testnet,pgn-mainnet,arbitrum,arbitrum-goerli,polygon-mumbai,polygon"
ENABLE_RESOURCE_MONITOR = "true"
NODE_OPTIONS="--max-old-space-size=7168"
# BUILD_TAG = # set in deploy github action

Expand Down
1 change: 1 addition & 0 deletions fly.staging.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ kill_timeout = 5
LOG_LEVEL = "debug"
STORAGE_DIR = "/mnt/indexer"
INDEXED_CHAINS = "mainnet,optimism,fantom,pgn-testnet,pgn-mainnet,arbitrum,arbitrum-goerli,polygon-mumbai,polygon"
ENABLE_RESOURCE_MONITOR = "true"
# BUILD_TAG = # set in deploy github action

[processes]
Expand Down
Loading

0 comments on commit 6560b41

Please sign in to comment.