[CAI-118] Presidio

.changeset/long-camels-sell.md

@@ -0,0 +1,5 @@
+---
+"chatbot": minor
+---
+
+"Add Presidio to detect and mask PII entities"

apps/chatbot/.env.example

@@ -4,12 +4,16 @@ PYTHONPATH=app-path
 LOG_LEVEL=DEBUG
 CHB_AWS_ACCESS_KEY_ID=...
 CHB_AWS_SECRET_ACCESS_KEY=...
-CHB_AWS_DEFAULT_REGION=eu-west-3
+CHB_AWS_DEFAULT_REGION=eu-south-1
+CHB_AWS_BEDROCK_REGION=eu-west-3
 CHB_AWS_S3_BUCKET=...
 CHB_AWS_GUARDRAIL_ID=...
 CHB_AWS_GUARDRAIL_VERSION=...
 CHB_REDIS_URL=...
 CHB_WEBSITE_URL=...
+CHB_REDIS_INDEX_NAME=...
+CHB_LLAMAINDEX_INDEX_ID=...
+CHB_DOCUMENTATION_DIR=...
 CHB_GOOGLE_API_KEY=...
 CHB_PROVIDER=...
 CHB_MODEL_ID=...

apps/chatbot/README.md

@@ -1,10 +1,15 @@
 # PagoPA Chatbot
 
-This folder contains all the details to build a RAG using the documentation provided in [`PagoPA Developer Portal`](https://developer.pagopa.it/). The retriver chosen is the `Auto Merging Retriver` one and it was implemented using [`llama-index`](https://docs.llamaindex.ai/en/stable/). Check out `src/modules/retriever.py`.
+This folder contains all the details to build a RAG using the documentation provided in [`PagoPA Developer Portal`](https://developer.pagopa.it/).
 
-This chatbot uses [`AWS Bedrock`](https://aws.amazon.com/bedrock/) as provider, so be sure to have installed [`aws-cli`](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and stored your credential in `~/.aws/credentials`.
+This chatbot uses [Google](https://ai.google.dev/) or [AWS Bedrock](https://aws.amazon.com/bedrock/) as provider.
+Even though the provider is the Google one, we stored its API key in AWS. So, be sure to have installed [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and stored your credential in `~/.aws/credentials`.
 
-All the parameters and prompts used to build the Retrieval-Augmented Generation (RAG) are available in `config`.
+The Retrieval-Augmented Generation (RAG) was implemented using [llama-index](https://docs.llamaindex.ai/en/stable/). All the parameters and prompts used are stored in `config`.
+
+## Environment Variables
+
+Create a `.env` file inside this folder and store the environment variables listed in `.env.example`.
 
 ## Virtual environment
 
@@ -27,40 +32,17 @@ The working directory is `/developer-portal/apps/chatbot`. So, to set the `PYTHO
 
 In this way, `PYTHONPATH` points to where the Python packages and modules are, not where your checkouts are.
 
-## File for Environment Variables
-
-Create a `.env` file inside the folder and write to the file the following environment variables:
-
- CHB_AWS_ACCESS_KEY_ID=...
- CHB_AWS_SECRET_ACCESS_KEY=...
- CHB_AWS_DEFAULT_REGION=...
- CHB_AWS_S3_BUCKET=...
- CHB_AWS_GUARDRAIL_ID=...
- CHB_AWS_GUARDRAIL_VERSION=...
- CHB_REDIS_URL=...
- CHB_REDIS_INDEX_NAME=...
- CHB_WEBSITE_URL=...
- CHB_GOOGLE_API_KEY=...
- CHB_PROVIDER=...
- CHB_MODEL_ID=...
- CHB_MODEL_TEMPERATURE=...
- CHB_MODEL_MAXTOKENS=...
- CHB_EMBED_MODEL_ID=...
- CHB_ENGINE_SIMILARITY_TOPK=...
- CHB_ENGINE_SIMILARITY_CUTOFF=...
- CHB_ENGINE_USE_ASYNC=...
- CHB_ENGINE_USE_STREAMING=...
-
-## Knowledge vector database
+## Knowledge index vector database
 
 To reach the remote redis instance, it is necessary to open a tunnel:
+
 ```
  ./scripts/redis-tunnel.sh
 ```
 
 Verify that the HTML files that compose the Developer Portal documentation exist in a directory. Otherwise create the documentation. Once you have the documentation directory ready, put its path in `params` and, in the end, create the vector index doing:
 
-``` 
+```
  python src/modules/create_vector_index.py --params config/params.yaml
 ```
 

apps/chatbot/config/params.yaml

@@ -5,9 +5,57 @@ vector_index:
  path: index
  chunk_sizes: [2816, 704, 176]
  chunk_overlap: 20
- use_redis: True
- use_s3: False
 
 engine:
  response_mode: compact
  verbose: False
+
+config_presidio:
+ nlp_engine_name: spacy
+ models:
+ -
+ lang_code: en
+ model_name: en_core_web_md
+ -
+ lang_code: it
+ model_name: it_core_news_md
+ # -
+ # lang_code: de
+ # model_name: de_core_news_md
+ # -
+ # lang_code: es
+ # model_name: es_core_news_md
+ # -
+ # lang_code: fr
+ # model_name: fr_core_news_md
+ ner_model_configuration:
+ labels_to_ignore:
+ - ORDINAL
+ - QUANTITY
+ - ORGANIZATION
+ - ORG
+ - LANGUAGE
+ - PRODUCT
+ - MONEY
+ - PERCENT
+ - O
+ - CARDINAL
+ - EVENT
+ - WORK_OF_ART
+ - LAW
+ - MISC
+ model_to_presidio_entity_mapping:
+ PER: PERSON
+ PERSON: PERSON
+ LOC: LOCATION
+ LOCATION: LOCATION
+ GPE: LOCATION
+ ORG: ORGANIZATION
+ DATE: DATE_TIME
+ TIME: DATE_TIME
+ NORP: NRP
+ low_confidence_score_multiplier: 0.4
+ low_score_entity_names:
+ - ORGANIZATION
+ - ORG
+ default_score: 0.8

apps/chatbot/config/prompts.yaml

@@ -1,6 +1,6 @@
 qa_prompt_str: |
  You are an customer services chatbot.
- Your name is Discovery and your duty is to assist the user with the PagoPA DevPortal documentation!
+ Your name is Discovery and your duty is to assist the user with the PagoPA DevPortal documentation, homepage: https://dev.developer.pagopa.it!
  --------------------
  Context information:
  {context_str}

apps/chatbot/docker/app.Dockerfile

@@ -14,4 +14,6 @@ RUN poetry install
 
 COPY . ${LAMBDA_TASK_ROOT}
 RUN python ./scripts/nltk_download.py
+RUN python ./scripts/spacy_download.py
+
 CMD ["src.app.main.handler"]