Verify image signatures

NetFoundry signs its FIPS container images with cosign and one NetFoundry key held in AWS KMS. Pin the public key on this page and verify images by tag or digest before you run them.

Image	Repository
Ziti CLI (FIPS)	quay.io/netfoundry/ziti-cli-fips
Ziti controller (FIPS)	quay.io/netfoundry/ziti-controller-fips
Ziti router (FIPS)	quay.io/netfoundry/ziti-router-fips
Ziti tunneler (FIPS)	quay.io/netfoundry/ziti-tunnel-fips

Every release from 1.6.12 on is signed, and so is latest.

Public key

Save this as nf-selfhosted-signing.pub:

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEdwgObNB2NaTPkGmpdOequwhrs5P+
SBlMKE7NsLXqAwD2SW220fvD/6aCQia4zN1A1s8vibEVPS6AKDA/8QaNAA==
-----END PUBLIC KEY-----

Check its SHA-256 fingerprint:

openssl pkey -pubin -in nf-selfhosted-signing.pub -outform DER | openssl dgst -sha256

The output must be:

a4abb8edea583e334cc799a2a2001f65742a81cdd217e5b576a7e655555d4e29

If it isn't, don't use the key, and contact NetFoundry support.

What's signed

• The digest each tag points at. For multi-arch tags that's the image index digest, and each per-platform image is signed too.
• Every digest is signed in two formats, both stored in the image's own quay repository, so the pull credentials you already have can read them:
  • A classic cosign signature in the sha256-<digest>.sig tag. Every cosign version and older admission controllers read this.
  • A Sigstore bundle (the cosign 3 default), stored as an OCI referrer of the digest. quay serves it through the OCI referrers API, so a registry proxy in front of quay must pass that API through for cosign 3 to see it.
• Signatures are recorded in the public Sigstore transparency log (Rekor).
• Each image's SBOM and build provenance are signed with the same key, in the same two formats. See SBOM and provenance.

A signature on a tag published before signing began asserts that NetFoundry published that digest. It doesn't assert that the image passed checks added to the release pipeline later.

Verify a signature

Log in to quay first (docker login quay.io or cosign login quay.io). cosign reads the same credentials.

cosign verify --key nf-selfhosted-signing.pub quay.io/netfoundry/ziti-controller-fips:1.6.21

cosign resolves the tag to its digest, checks the signature against the key, and checks the Rekor entry. Pull by the digest it prints, so the image you run is the one you verified.

cosign 3 checks the Sigstore bundle. To check the classic .sig tag instead, add --new-bundle-format=false. cosign 2 checks the .sig tag. Any of these passing means the image is signed by the NetFoundry key.

Air-gapped clusters

Without access to Rekor, skip the transparency log check. The key check still applies:

cosign verify --key nf-selfhosted-signing.pub --insecure-ignore-tlog=true \
  registry.example.internal/netfoundry/ziti-controller-fips:1.6.21

The mirror needs both signature formats, and a plain tag copy carries only one. The .sig tags are ordinary tags, but quay serves the Sigstore bundles through the OCI referrers API, not as tags. Copy each image with its referrers, then copy its .sig tags:

src=quay.io/netfoundry/ziti-controller-fips:1.6.21
dst=registry.example.internal/netfoundry/ziti-controller-fips:1.6.21
oras cp -r "$src" "$dst"                 # the image, every manifest under it, and their bundles (oras 1.3)
cosign copy --only=sig,att "$src" "$dst" # the .sig and .att tags of the index and every manifest

cosign copy alone doesn't carry the bundles: it copies only the .sig, .att, and .sbom tags. oras marks -r as a preview flag.

SBOM and provenance

Every release from 1.6.12 on, and latest, carries a signed SBOM and signed build provenance.

What's attested

Every image is built with Docker buildx, which attaches two documents to each platform image:

• An SBOM in SPDX JSON (predicate type https://spdx.dev/Document): the Go modules, Debian packages, and other software in the image.
• SLSA build provenance (predicate type https://slsa.dev/provenance/v0.2 or https://slsa.dev/provenance/v1, depending on the BuildKit version that built the image): how the image was built, and from what source.

buildx stores them unsigned, in attestation manifests inside the image index. NetFoundry signs the same documents, unchanged, as in-toto attestations with the NetFoundry key:

• On each platform image digest: that platform's SBOM and provenance.
• On the index digest (what a tag resolves to, and what admission controllers check): the SBOM and provenance of every platform. So the index carries one SBOM and one provenance per platform.

Like the signatures, each attestation is stored in two formats in the image's own quay repository:

• A classic cosign attestation in the sha256-<digest>.att tag.
• A Sigstore bundle, stored as an OCI referrer of the digest.

Why the SBOM is handled differently

The SBOMs are large. buildx's SPDX SBOM for one platform of these images is about 5 MB (most of it is a list of every file in the image), and about 7 MB once signed, since the signed envelope base64-encodes it.

The public Sigstore transparency log has two versions, and they treat that size differently:

• Rekor v1 (rekor.sigstore.dev, the log classic cosign attestations use) receives the whole signed attestation, and the public instance accepts attestations only up to 100 KB: "The attestation size limit for uploads to the public instance is 100KB" (Rekor README). The SBOM is about 50 times that, so it can't be logged there. The provenance is much smaller, and is logged.
• Rekor v2 is "compatible with large DSSE envelopes, as the whole envelope is no longer sent over to the network to be processed by the log" (Sigstore blog). It keeps only hashes and the signature (Rekor DSSE type). Sigstore bundles are logged there, with a signed timestamp from Sigstore's timestamp authority.

So:

Format	SBOM	Provenance
Classic .att	Signed, not logged. Verify with --insecure-ignore-tlog (cosign) or ignoreTlog: true (Kyverno).	Signed, logged in Rekor v1.
Sigstore bundle	Signed, logged in Rekor v2.	Signed, logged in Rekor v2.

The signature is what proves the SBOM came from NetFoundry, and the key check applies in every case. The log adds a public, timestamped record that the signature was made. That's why the classic SBOM skips the log rather than being cut down to fit: a shortened SBOM would no longer be the one buildx produced, and even without its file list it's several times the limit.

Both formats are published because verifiers read different ones. cosign 2 and Kyverno's default Cosign verifier read only .att tags. cosign 3 and Kyverno's SigstoreBundle verifier read bundles.

Verify with cosign

k=nf-selfhosted-signing.pub
i=quay.io/netfoundry/ziti-controller-fips:1.6.21
# Sigstore bundle, checked against Rekor v2 (cosign 3):
cosign verify-attestation --key $k --type spdxjson $i > /dev/null
# Classic .att (any cosign). The SBOM isn't in the log, so skip the log check:
cosign verify-attestation --key $k --type spdxjson --new-bundle-format=false --insecure-ignore-tlog=true $i > /dev/null

A passing check prints this on stderr and exits 0:

Verification for quay.io/netfoundry/ziti-controller-fips:1.6.21 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

For provenance, use --type https://slsa.dev/provenance/v0.2 or --type https://slsa.dev/provenance/v1. The classic provenance is logged, so it needs no --insecure-ignore-tlog. In an air-gapped cluster, add --insecure-ignore-tlog=true to all of them.

To read the verified SBOMs (one per platform on an index):

cosign verify-attestation --key $k --type spdxjson $i 2> /dev/null \
  | jq '(.payload // .dsseEnvelope.payload) | @base64d | fromjson | .predicate'

For a quick summary, one line per platform:

cosign verify-attestation --key $k --type spdxjson $i 2> /dev/null \
  | jq -r '(.payload // .dsseEnvelope.payload) | @base64d | fromjson | .predicate
           | "\(.name) \(.packages | length) packages"'

sbom 416 packages
sbom 416 packages

Read the SBOM from the verified output, not from the unsigned copy buildx attached. Only the verified one is known to come from NetFoundry.

Enforce signatures in Kubernetes with Kyverno

This sample ClusterPolicy admits NetFoundry FIPS images only when they're signed by the NetFoundry key, and rewrites them to the verified digest. Give Kyverno pull credentials for quay (imagePullSecrets on the policy rule, or Kyverno's --imagePullSecrets flag). The policy reads the public key from a secret in the Kyverno namespace:

kubectl -n kyverno create secret generic netfoundry-cosign-key --from-file=cosign.pub=nf-selfhosted-signing.pub

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-netfoundry-images
spec:
  webhookTimeoutSeconds: 30
  rules:
    - name: netfoundry-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "quay.io/netfoundry/ziti-*-fips*"
          failureAction: Enforce  # Kyverno < 1.13: spec.validationFailureAction: Enforce
          mutateDigest: true
          verifyDigest: true
          required: true
          attestors:
            - entries:
                - keys:
                    publicKeys: "k8s://kyverno/netfoundry-cosign-key"
                    rekor:
                      url: https://rekor.sigstore.dev
                      # air-gapped: replace url with ignoreTlog: true

Tested with Kyverno 1.19.2.

Require a signed SBOM

A Kyverno rule checks either signatures or attestations, not both (Kyverno docs), so add a second rule to the policy above. This one reads the classic .att tag. The classic SBOM isn't in the transparency log (why), so it needs ignoreTlog: true (Kyverno docs):

    - name: netfoundry-sbom
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "quay.io/netfoundry/ziti-*-fips*"
          failureAction: Enforce
          attestations:
            - type: https://spdx.dev/Document
              attestors:
                - entries:
                    - keys:
                        publicKeys: "k8s://kyverno/netfoundry-cosign-key"
                        rekor:
                          ignoreTlog: true

To read the Sigstore bundles instead, add type: SigstoreBundle beside imageReferences (Kyverno docs), and remove the rekor block. The bundles are in Rekor v2, so Kyverno checks the SBOM against the transparency log and its signed timestamp, with no ignoreTlog:

      verifyImages:
        - imageReferences:
            - "quay.io/netfoundry/ziti-*-fips*"
          type: SigstoreBundle
          failureAction: Enforce
          attestations:
            - type: https://spdx.dev/Document
              attestors:
                - entries:
                    - keys:
                        publicKeys: "k8s://kyverno/netfoundry-cosign-key"

Don't add ctlog: {ignoreSCT: true}. A key signature has no certificate, so it has no SCT, and Kyverno doesn't ask for one. With ignoreSCT: true, Kyverno 1.19.2 also turns off the timestamp check, and verification then fails with "you must specify at least one of WithObserverTimestamps() ...".

Both rules were tested with Kyverno 1.19.2 against quay: the NetFoundry key passes, and a wrong key fails.

Kyverno tips

• Pulling through a registry proxy or mirror: Kyverno matches and verifies the image reference the pod uses. If your pods pull through a proxy (for example, an Artifactory remote for quay), set imageReferences to the proxy's path, such as "artifactory.example.com/quay-remote/netfoundry/ziti-*-fips*", and give Kyverno credentials for the proxy. The proxy must serve the .sig tags. Kyverno 1.19.2 also verified an image that carried only the .sig tag. The classic SBOM rule also needs the .att tags. The SigstoreBundle rule needs the OCI referrers API passed through.

• Roll out in audit mode first: set failureAction: Audit, check the policy reports (kubectl get policyreports -A) for failures, then switch to Enforce.

• Air-gapped: replace the rekor.url line with ignoreTlog: true, and point imageReferences at your mirror. For the SigstoreBundle SBOM rule, add both rekor: {ignoreTlog: true} and ctlog: {ignoreSCT: true} under keys. Kyverno then checks the key alone, and doesn't fetch Sigstore's trust root.

• Digest rewrite: mutateDigest: true rewrites the image to tag@digest. Kyverno also applies Pod rules to Deployments, StatefulSets, and other controllers, so their pod templates get rewritten too. GitOps tools such as Argo CD can report that as drift. Either ignore the image field in the GitOps tool, or set mutateDigest: false and pin digests in your Helm values yourself.

• Test before you apply: the Kyverno CLI checks the policy against a pod spec without a cluster. Put the key inline in publicKeys (the CLI can't read the k8s:// secret), reference the image as tag@digest (in a cluster mutateDigest adds the digest before verifyDigest checks it, but the CLI checks the pod as written), and pass --registry so it uses your docker login:

  kyverno apply policy.yaml --resource pod.yaml --registry --detailed-results

Key changes

NetFoundry announces a new key on this page at least one release cycle before the old one stops signing. During that overlap, images carry signatures from both keys, so either pinned key verifies. After a suspected compromise, NetFoundry revokes the key on this page and notifies customers directly.