-
Notifications
You must be signed in to change notification settings - Fork 7.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(nodestore): add Postgres for the doc and index store (#1706)
* Adding Postgres for the doc and index store * Adding documentation. Rename postgres database local->simple. Postgres storage dependencies * Update documentation for postgres storage * Renaming feature to nodestore * update docstore -> nodestore in doc * missed some docstore changes in doc * Updated poetry.lock * Formatting updates to pass ruff/black checks * Correction to unreachable code! * Format adjustment to pass black test * Adjust extra inclusion name for vector pg * extra dep change for pg vector * storage-postgres -> storage-nodestore-postgres * Hash change on poetry lock
- Loading branch information
Showing
9 changed files
with
226 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
## NodeStores | ||
PrivateGPT supports **Simple** and [Postgres](https://www.postgresql.org/) providers. Simple being the default. | ||
|
||
In order to select one or the other, set the `nodestore.database` property in the `settings.yaml` file to `simple` or `postgres`. | ||
|
||
```yaml | ||
nodestore: | ||
database: simple | ||
``` | ||
### Simple Document Store | ||
Setting up simple document store: Persist data with in-memory and disk storage. | ||
Enabling the simple document store is an excellent choice for small projects or proofs of concept where you need to persist data while maintaining minimal setup complexity. To get started, set the nodestore.database property in your settings.yaml file as follows: | ||
```yaml | ||
nodestore: | ||
database: simple | ||
``` | ||
The beauty of the simple document store is its flexibility and ease of implementation. It provides a solid foundation for managing and retrieving data without the need for complex setup or configuration. The combination of in-memory processing and disk persistence ensures that you can efficiently handle small to medium-sized datasets while maintaining data consistency across runs. | ||
### Postgres Document Store | ||
To enable Postgres, set the `nodestore.database` property in the `settings.yaml` file to `postgres` and install the `storage-nodestore-postgres` extra. Note: Vector Embeddings Storage in Postgres is configured separately | ||
|
||
```bash | ||
poetry install --extras storage-nodestore-postgres | ||
``` | ||
|
||
The available configuration options are: | ||
| Field | Description | | ||
|---------------|-----------------------------------------------------------| | ||
| **host** | The server hosting the Postgres database. Default is `localhost` | | ||
| **port** | The port on which the Postgres database is accessible. Default is `5432` | | ||
| **database** | The specific database to connect to. Default is `postgres` | | ||
| **user** | The username for database access. Default is `postgres` | | ||
| **password** | The password for database access. (Required) | | ||
| **schema_name** | The database schema to use. Default is `private_gpt` | | ||
|
||
For example: | ||
```yaml | ||
nodestore: | ||
database: postgres | ||
postgres: | ||
host: localhost | ||
port: 5432 | ||
database: postgres | ||
user: postgres | ||
password: <PASSWORD> | ||
schema_name: private_gpt | ||
``` | ||
|
||
Given the above configuration, Two PostgreSQL tables will be created upon successful connection: one for storing metadata related to the index and another for document data itself. | ||
|
||
``` | ||
postgres=# \dt private_gpt.* | ||
List of relations | ||
Schema | Name | Type | Owner | ||
-------------+-----------------+-------+-------------- | ||
private_gpt | data_docstore | table | postgres | ||
private_gpt | data_indexstore | table | postgres | ||
postgres=# | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
# Using ollama and postgres for the vector, doc and index store. Ollama is also used for embeddings. | ||
# To use install these extras: | ||
# poetry install --extras "llms-ollama ui vector-stores-postgres embeddings-ollama storage-nodestore-postgres" | ||
server: | ||
env_name: ${APP_ENV:ollama} | ||
|
||
llm: | ||
mode: ollama | ||
max_new_tokens: 512 | ||
context_window: 3900 | ||
|
||
embedding: | ||
mode: ollama | ||
|
||
ollama: | ||
llm_model: mistral | ||
embedding_model: nomic-embed-text | ||
api_base: http://localhost:11434 | ||
|
||
nodestore: | ||
database: postgres | ||
|
||
vectorstore: | ||
database: pgvector | ||
|
||
pgvector: | ||
host: localhost | ||
port: 5432 | ||
database: postgres | ||
user: postgres | ||
password: admin | ||
embed_dim: 768 | ||
schema_name: private_gpt | ||
table_name: embeddings | ||
|
||
postgres: | ||
host: localhost | ||
port: 5432 | ||
database: postgres | ||
user: postgres | ||
password: admin | ||
schema_name: private_gpt | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters