Unexpected EOF log and vulnerability report no data

[chary1112004]

What steps did you take and what happened:

Hi,

We are facing issue the vulnerability report no data and log file contains Unexpected EOF

{"level":"error","ts":"2024-01-22T02:24:27Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-59cb6b98d4","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-59cb6b98d4","reconcileID":"054a1f46-d3f3-43a0-a5a5-c8abb278627b","error":"unexpected EOF","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

value file:

operator:
  # scanJobsConcurrentLimit the maximum number of scan jobs create by the operator
  scanJobsConcurrentLimit: 1

  # -- scanJobTimeout the length of time to wait before giving up on a scan job
  scanJobTimeout: 20m
  
  # -- infraAssessmentScannerEnabled the flag to enable infra assessment scanner
  infraAssessmentScannerEnabled: false

  # -- controllerCacheSyncTimeout the duration to wait for controller resources cache sync (default: 5m).
  controllerCacheSyncTimeout: "10m"

serviceMonitor:
  enabled: true
  labels:
    release: prometheus

resources:
  # -- We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  requests:
    cpu: 15m
    memory: 250Mi

trivyOperator:
  # -- scanJobPodTemplateContainerSecurityContext SecurityContext the user wants the scanner and node collector containers (and their
  # initContainers) to be amended with.
  scanJobPodTemplateContainerSecurityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    privileged: false
    readOnlyRootFilesystem: true
    # -- For filesystem scanning, Trivy needs to run as the root user
    runAsUser: 0

trivy:
  # -- slow this flag is to use less CPU/memory for scanning though it takes more time than normal scanning. It fits small-footprint
  slow: false

  # -- timeout is the duration to wait for scan completion.
  timeout: "20m0s"

  # -- command. One of `image`, `filesystem` or `rootfs` scanning, depending on the target type required for the scan.
  # For 'filesystem' and `rootfs` scanning, ensure that the `trivyOperator.scanJobPodTemplateContainerSecurityContext` is configured
  # to run as the root user (runAsUser = 0).
  command: filesystem
  
  # -- resources resource requests and limits for scan job containers
  resources:
    requests:
      cpu: 100m
      memory: 1000M
      # ephemeralStorage: "2Gi"
    limits:
      cpu: 500m
      memory: 2000M
      # ephemeralStorage: "2Gi"

Any suggestion to resolve this issue would be very much appreciated!

Thanks!

Environment:

• Trivy-Operator version (use trivy-operator version): 0.20.1
• Kubernetes version (use kubectl version): 1.25
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc):

[chen-keinan]

@chary1112004 is the pod still running can you get it output
kubectl logs <pod name> -n trivy-system

[chary1112004]

@chary1112004 is the pod still running can you get it output kubectl logs <pod name> -n trivy-system

You mean about scan-job pod or trivy-operator pod?
If you mean about trivy-operator pod then yes, it is still running

[chen-keinan]

@chary1112004 is the pod still running can you get it output kubectl logs <pod name> -n trivy-system

You mean about scan-job pod or trivy-operator pod? If you mean about trivy-operator pod then yes, it is still running

I mean for the scan-job

[chary1112004]

@chary1112004 is the pod still running can you get it output kubectl logs <pod name> -n trivy-system

You mean about scan-job pod or trivy-operator pod? If you mean about trivy-operator pod then yes, it is still running

I mean for the scan-job

yes, scan-job is in status Completed 0/1. However I do not store log of this pod.

[daanschipper]

I'm facing the same issue, possibly related to #591. It seems the pod is terminated before the stdout buffer is flushed, resulting in the unexpected EOF. I tried both setting scanJobCompressLogs to false and true, doesn't seem to make a difference as the output still needs to be flushed before the container is stopped. Running the same command locally does output the full json.

kubectl logs -n trivy-system pod/scan-vulnerabilityreport-85fd8c6947-rfr66 -c <container>

  ... omitted
  {
    "VulnerabilityID": "DLA-3412-1",
    "VendorIDs": [
      "DLA-3412-1"
    ],
    "PkgID": "tzdata@2021a-0+deb10u6",
    "PkgName": "tzdata",
    "PkgIdentifier": {
      "PURL": "pkg:deb/debian/tzdata@2021a-0%2Bdeb10u6?arch=all\u0026distro=debian-%

The other issue suggests adding a sleep to allow the full report to be outputted, but it seems this cannot be configured. Probably a better solution would be to use sync.

[hubertbarc-rho]

Having the same issue here although my error is a bit different

{"level":"error","ts":"2024-09-05T02:56:27Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-7955c58bd","namespace":"security-tools"},"namespace":"security-tools","name":"scan-vulnerabilityreport-7955c58bd","reconcileID":"416f2e4e-a244-408c-a2e6-fbdc6fb644b4","error":"unexpected end of JSON input","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}