Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: release auto-update script source code #1

Open
henrirosten opened this issue Jun 9, 2023 · 8 comments
Open

Question: release auto-update script source code #1

henrirosten opened this issue Jun 9, 2023 · 8 comments
Labels
enhancement New feature or request

Comments

@henrirosten
Copy link

Would it be possible to share the script you use to query NVD API and auto-update the data on this repository?

@rhelmke
Copy link
Collaborator

rhelmke commented Jun 9, 2023

At some point we can certainly release the code auto-updating this repo. However we would like to make it a little bit more robust first :-). Please give us some time for that.

@rhelmke rhelmke closed this as completed Jun 9, 2023
@rhelmke
Copy link
Collaborator

rhelmke commented Jun 9, 2023

I'm glad to re-open this issue once we're ready!

@henrirosten
Copy link
Author

Thanks, and also thanks for the detailed explanation in #2.

The reason I'm asking for the script source is to be able to re-generate the data locally, or perhaps mirror the data in another repository in case you would happen to stop, for one reason or another, to cache the NVD data on this repo.

Anyway, thanks for the great work you are doing here!

@yann-morin-1998
Copy link

@rhelmke Sorry to chime in this old issue: has there been any progress in making the mirroring
scripts available?

Additionally, we would also be very much interested in the scripts that aggregates the
individual CVEs into the daily feeds. Indeed, those feeds are short-lived; they are replaced
daily. As such, there is no possibility to do reproducible builds.

For example, in our project, Buildroot, we are tracking a regression
in our tooling, that occurred around 2024-02-07. Unfortunately, we can't validate when the
issue actually happened, because the CVE feed from that day is no longer available. Since this
is a git tree, we could easily reconstruct the feed from the individual entries, if the scripts
were available.

@rhelmke
Copy link
Collaborator

rhelmke commented Feb 27, 2024

Hello @yann-morin-1998,

unfortunately there is still no release timeline for the software stack driving this repo. We are currently occupied with a lot of other projects and wouldn't be able to allocate the required resources at this time - I'm sorry.

Either ways the packaging code wouldn't help you guys to reconstruct any daily packages from this repo's history. This is because the code also uses our OpenSearch backend and is no standalone script.

However, we certainly see and understand the issues you guys are faced with in terms of reproducibility. In fact, the idea to provide companion scripts that are able to reliably reconstruct historical packages has been around for a while. I assume that we could provide such a script and verify its correctness in manageable time. Give us maybe a week and we'll see what we can do :-).

@rhelmke rhelmke reopened this Feb 27, 2024
@rhelmke rhelmke added the enhancement New feature or request label Feb 27, 2024
@rhelmke
Copy link
Collaborator

rhelmke commented Feb 27, 2024

On another note, we also thought about not wiping historical release packages, but refrained from the idea because it would certainly create a lot of duplicate data to host. And it is truly unnecessary considering that a companion script could use the git history for reconstruction.

@yann-morin-1998
Copy link

@rhelmke Thanks for the feedback, and thanks for considering our request. That's very much appreciated.

How open are you to contributions? I have been playing on a little python script here, that walks the individual CVE directories in the repository, and generates reproducible yearly archives. It's working now, and just needs a little eye-candy. Shall I open a PR?

@rhelmke
Copy link
Collaborator

rhelmke commented Feb 28, 2024

@yann-morin-1998 thank you very much! We're of course open to PRs and would really appreciate it. But I'm not quite sure if this is the right repository for it. I thought about a tool that would take an ISO date as input, automatically clone the repo, check out the correct commit, and then recreate the packages. It might be better to move the script to another repo such that it does not have to sit in the working file tree.

Let me talk to a colleague of mine, he might be able to quickly throw together a python package for that. I'll (or he'll) let you know how he'd like to proceed :-).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants