Add --ignored Flag to Exclude Specific Files and Directories During Ingestion

[smbrine]

This pull request introduces the --ignored flag to the ingestion script. The motivation for this change stems from encountering UnicodeDecodeError when the script processes Python-generated __pycache__ folders and MacOS-specific .DS_Store files. These files are not relevant to our data processing and can cause errors due to their format.

The --ignored flag allows users to specify a list of files or directories to exclude from the ingestion process, enhancing the script's flexibility and reliability, especially in Python and Mac environments.

[imartinez]

Great contribution!

[smbrine]

It's interesting to note that the test pipeline's failure was due to not running 'black .' for code formatting, rather than actual code errors. To prevent such issues, i might integrate 'black' as a step in your GitHub Actions. This way, code is automatically formatted before it's even merged, ensuring consistency and preventing similar failures. This would streamline our development process by catching formatting issues early and automatically, without relying on manual execution. Would you merge if I make these changes?

[imartinez]

It's interesting to note that the test pipeline's failure was due to not running 'black .' for code formatting, rather than actual code errors. To prevent such issues, i might integrate 'black' as a step in your GitHub Actions. This way, code is automatically formatted before it's even merged, ensuring consistency and preventing similar failures. This would streamline our development process by catching formatting issues early and automatically, without relying on manual execution. Would you merge if I make these changes?

I'd rather add black as a pre-commit action than a github action. I'm no big fan of Github changing code without the developer noticing. Thanks for your feedback!

[imartinez]

black still fails. Run make check and commit the format fixes please. Thanks!

[smbrine]

Ran make check twice just in case. Also added some .gitignore entries and a make list command to list all available commands.

[github-actions]

Stale pull request

[tylervick]

Hey @imartinez @smbrine, it looks like the feedback in this PR was addressed. Is there anything else preventing it from being merged?

I've been using this branch for some weeks now, I'd love this feature in main 🙂

[imartinez]

Hey @imartinez @smbrine, it looks like the feedback in this PR was addressed. Is there anything else preventing it from being merged?

I've been using this branch for some weeks now, I'd love this feature in main 🙂

Thanks for the heads up! I've disabled the GitHub action that closed the PR. Merging it now!