Add Custom File Path to Watcher which is outside the path of the main script

[akramsystems]

PR open: #9655

Checklist

• I have searched the existing issues for similar feature requests.
• I added a descriptive title and summary to this issue.

Summary

Add a feature to allow watching a custom path in Streamlit, enabling the application to monitor all files within that directory and its subdirectories. Currently, Streamlit supports a blacklist for excluding paths but lacks an option to specify a custom path to be watched in full.

Example Scenario:

/home/user/
├── streamlit_project/
│   ├── main_app.py
│   ├── data/
│   │   └── local_data.csv
└── shared_data/
    └── external_data.csv

Why?

The current system only supports watching the directory of the main streamlit script or excluding certain files via a blacklist. In cases where projects have structured directories or additional assets in specific folders, it is important to have the flexibility to define custom paths for Streamlit to watch, ensuring that all relevant files are reloaded as needed. This would improve developer experience by allowing greater control over what Streamlit monitors for changes, preventing unnecessary reloads and improving workflow.

How?

The implementation introduces a new configuration option, server.customWatchPath, which allows users to specify a custom path for Streamlit to monitor. This path will be recursively watched for changes to any files within it. The main changes in this implementation are:

1. Configurable Option: A new option, server.customWatchPath, is added in the config.py file. This allows users to define a custom directory for file watching through Streamlit's configuration system.

2. Runtime and Path Watching: The runtime has been updated to store the custom_watch_path in the RuntimeConfig. The path watcher (EventBasedPathWatcher) has been extended to accept this custom path and watch it recursively for any file changes. If a custom watch path is provided, the watcher registers it and logs relevant debug information about the additional path being watched.

3. Recursive Directory Watching: In local_sources_watcher.py, the _register_watcher method has been updated to handle directory watching, ensuring that all files within the custom path are monitored for changes. If the custom path is valid, it will be registered with the path watcher, and any file changes within the directory will trigger a file change event.

These changes allow for a flexible approach where users can monitor any directory, ensuring that all necessary files are captured without relying solely on the default Streamlit watch settings.

Additional Context

You can find other users expressing a similar needs in the streamlit discussion threads below:

• https://discuss.streamlit.io/t/how-to-monitor-the-filesystem-and-have-streamlit-updated-when-some-files-are-modified/822/21
• https://discuss.streamlit.io/t/watching-custom-folders/1507
• https://discuss.streamlit.io/t/how-to-extend-streamlit-watchers-to-monitor-custom-user-files/36694/2

Note:

Streamlit offers the ability to allow users to watch modules in the PYTHONPATH as seen in #845 , but this method requires modifying the PYTHONPATH, which some users may find undesirable. Allowing a custom path to be specified provides a more flexible solution without requiring any changes to the PYTHONPATH, ensuring a smoother experience for users who may have structured directories outside of the default Python paths.

[github-actions]

To help Streamlit prioritize this feature, react with a 👍 (thumbs up emoji) to the initial post.

Your vote helps us identify which enhancements matter most to our users.

[AliTranscrypts]

Awesome feature addition! Exactly what our company is looking for as we prototype.

[akramsystems]

@jrieke i have a PR open here #9656 let me know if there is anything i can do to help with, in terms of either prioritizing this ticket or working on the PR, I appreciate you adding the tag keep up the amazing work!