fields.yml not being completely regenerated

[pzl]

So an interesting conundrum occurred. I don't understand the nature of it, so this is just going to state the facts of what occurred, sense and acceptance criteria can come out of it:

As CI was being spun up, one of the steps is to run make, and then check for a dirty work-tree (any uncommitted difference between what was made and what was already checked in).

CI was coming up with failures, stating some differences.

  17:36:05  [Pipeline] sh (update git index)
  17:36:05  + git update-index -q --really-refresh
  17:36:06  [Pipeline] sh (check for uncommitted build artifacts)
  17:36:06  + git diff-index --exit-code HEAD --
  17:36:06  :100644 100644 c80838ae1bd3d9bdddfeed9384f6a01ff81a376b 0000000000000000000000000000000000000000 M	package/endpoint/data_stream/alerts/fields/fields.yml
  17:36:06  :100644 100644 f6d2d6d59e9426b7b0a4a5336c00cfd1e5ebf27b 0000000000000000000000000000000000000000 M	package/endpoint/data_stream/process/fields/fields.yml
  17:36:06  :100644 100644 e4691015fc8cde818667dc4d42ee0db58610a81a 0000000000000000000000000000000000000000 M	package/endpoint/docs/README.md
  17:36:06  [Pipeline] sh (print diff)

Full log output

Running make locally did not produce this diff. Creating a fresh clone elsewhere on my machine and running make did not produce the difference. I am running python3.10 and CI is running python3.8, perhaps that was the difference? I spun up docker with docker.io/ubuntu:20.04, installed python which gave me python3.8, then the rest of the necessary build tools python3-pip, golang, git, etc.

I cloned this repo in that docker, and ran make. No change.

Now back on my local outer host machine, I deleted package/endpoint/data_stream/process/fields/fields.yml and ran make and the diff for that file appeared. So I deleted all the fields.yml with

find package/ -name fields.yml -delete

and on the next make, the exact diff that CI saw, including to the README appeared.

I don't quite know what's happening here, but this can likely be sorted out in the Makefile. Perhaps something is not being copied over correctly, or perhaps modtime funny business. Deleting fields.yml as a normal step could work to keep them always fresh.

[mitodrummer]

I'm noticing this problem for myself locally. I can't seem to generate the proper artifacts after making changes to custom_schemas and custom_subsets. I've been told most devs are running macOS to build locally. I have been using ubuntu. Perhaps that is the difference between CI and local?